Apache Hadoop Changelog

Release 3.1.4 - 2020-07-21

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-9761	Allow overriding application submissions based on server side configs	Major	.	Jonathan Hung	pralabhkumar
YARN-9760	Support configuring application priorities on a workflow level	Major	.	Jonathan Hung	Varun Saxena
HDFS-14745	Backport HDFS persistent memory read cache support to branch-3.1	Major	.	Feilong He	Feilong He
HDFS-12943	Consistent Reads from Standby Node	Major	hdfs	Konstantin Shvachko	Konstantin Shvachko

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-8361	Change App Name Placement Rule to use App Name instead of App Id for configuration	Major	yarn	Zian Chen	Zian Chen
YARN-8750	Refactor TestQueueMetrics	Minor	resourcemanager	Szilard Nemeth	Szilard Nemeth
HADOOP-15849	Upgrade netty version to 3.10.6	Major	.	Xiao Chen	Xiao Chen
HDFS-14064	WEBHDFS: Support Enable/Disable EC Policy	Major	erasure-coding, webhdfs	Ayush Saxena	Ayush Saxena
HDFS-14113	EC : Add Configuration to restrict UserDefined Policies	Major	erasure-coding	Ayush Saxena	Ayush Saxena
HDFS-14124	EC : Support EC Commands (set/get/unset EcPolicy) via WebHdfs	Major	erasure-coding, httpfs, webhdfs	Souryakanta Dwivedy	Ayush Saxena
HDFS-14006	Refactor name node to allow different token verification implementations	Major	.	CR Hota	CR Hota
HADOOP-15909	KeyProvider class should implement Closeable	Major	kms	Kuhu Shukla	Kuhu Shukla
HDFS-14187	Make warning message more clear when there are not enough data nodes for EC write	Major	erasure-coding	Kitti Nanasi	Kitti Nanasi
HDFS-14235	Handle ArrayIndexOutOfBoundsException in DataNodeDiskMetrics#slowDiskDetectionDaemon	Major	.	Surendra Singh Lilhore	Ranith Sardar
HADOOP-16126	ipc.Client.stop() may sleep too long to wait for all connections	Major	ipc	Tsz-wo Sze	Tsz-wo Sze
HADOOP-16140	hadoop fs expunge to add -immediate option to purge trash immediately	Major	fs	Stephen O'Donnell	Stephen O'Donnell
HADOOP-15014	KMS should log the IP address of the clients	Major	kms	Zsombor Gegesy	Zsombor Gegesy
HDFS-14460	DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured	Major	.	CR Hota	CR Hota
HDFS-14624	When decommissioning a node, log remaining blocks to replicate periodically	Major	namenode	Stephen O'Donnell	Stephen O'Donnell
HDFS-13693	Remove unnecessary search in INodeDirectory.addChild during image loading	Major	namenode	zhouyingchao	Lisheng Sun
HDFS-14313	Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/du	Major	datanode, performance	Lisheng Sun	Lisheng Sun
HDFS-14678	Allow triggerBlockReport to a specific namenode	Major	datanode	Leon Gao	Leon Gao
HDFS-14523	Remove excess read lock for NetworkToplogy	Major	.	Wu Weiwei	Wu Weiwei
HDFS-14497	Write lock held by metasave impact following RPC processing	Major	namenode	Xiaoqiao He	Xiaoqiao He
YARN-9756	Create metric that sums total memory/vcores preempted per round	Major	capacity scheduler	Eric Payne	Manikandan R
YARN-9810	Add queue capacity/maxcapacity percentage metrics	Major	.	Jonathan Hung	Shubham Gupta
HADOOP-16531	Log more detail for slow RPC	Major	.	Chen Zhang	Chen Zhang
YARN-9763	Print application tags in application summary	Major	.	Jonathan Hung	Manoj Kumar
YARN-9764	Print application submission context label in application summary	Major	.	Jonathan Hung	Manoj Kumar
YARN-9824	Fall back to configured queue ordering policy class name	Major	.	Jonathan Hung	Jonathan Hung
HADOOP-16069	Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOST	Minor	common	luhuachao	luhuachao
YARN-9762	Add submission context label to audit logs	Major	.	Jonathan Hung	Manoj Kumar
HDFS-14850	Optimize FileSystemAccessService#getFileSystemConfiguration	Major	httpfs, performance	Lisheng Sun	Lisheng Sun
HDFS-14192	Track missing DFS operations in Statistics and StorageStatistics	Major	.	Ayush Saxena	Ayush Saxena
HADOOP-16625	Backport HADOOP-14624 to branch-3.1	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
YARN-9356	Add more tests to ratio method in TestResourceCalculator	Major	.	Szilard Nemeth	Zoltan Siegl
HDFS-14915	Move Superuser Check Before Taking Lock For Encryption API	Major	.	Ayush Saxena	Ayush Saxena
HDFS-14921	Remove SuperUser Check in Setting Storage Policy in FileStatus During Listing	Major	.	Ayush Saxena	Ayush Saxena
HDFS-14923	Remove dead code from HealthMonitor	Minor	.	Fei Hui	Fei Hui
YARN-9914	Use separate configs for free disk space checking for full and not-full disks	Minor	yarn	Jim Brennan	Jim Brennan
MAPREDUCE-7208	Tuning TaskRuntimeEstimator	Minor	.	Ahmed Hussein	Ahmed Hussein
HDFS-14942	Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndex	Minor	.	Lisheng Sun	Lisheng Sun
HDFS-14979	[Observer Node] Balancer should submit getBlocks to Observer Node when possible	Major	balancer & mover, hdfs	Erik Krogen	Erik Krogen
HADOOP-16705	MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug log	Major	metrics	Lukas Majercak	Lukas Majercak
HADOOP-16712	Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xml	Trivial	.	Wei-Chiu Chuang	Xieming Li
HADOOP-16703	Backport HADOOP-16152 to branch-3.1	Major	.	Siyao Meng	Siyao Meng
HDFS-14952	Skip safemode if blockTotal is 0 in new NN	Trivial	namenode	Rajesh Balamohan	Xiaoqiao He
YARN-8842	Expose metrics for custom resource types in QueueMetrics	Major	.	Szilard Nemeth	Szilard Nemeth
YARN-9966	Code duplication in UserGroupMappingPlacementRule	Major	.	Szilard Nemeth	Kevin Su
YARN-9937	Add missing queue configs in RMWebService#CapacitySchedulerQueueInfo	Major	capacity scheduler	Prabhu Joseph	Prabhu Joseph
HADOOP-16718	Allow disabling Server Name Indication (SNI) for Jetty	Major	.	Siyao Meng	Aravindan Vijayan
HADOOP-16735	Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKEN	Minor	documentation, fs/s3	Mingliang Liu	Mingliang Liu
YARN-10012	Guaranteed and max capacity queue metrics for custom resources	Major	.	Jonathan Hung	Manikandan R
HDFS-15050	Optimize log information when DFSInputStream meet CannotObtainBlockLengthException	Major	dfsclient	Xiaoqiao He	Xiaoqiao He
YARN-10033	TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resources	Major	capacity scheduler, test	Eric Payne	Eric Payne
YARN-10039	Allow disabling app submission from REST endpoints	Major	.	Jonathan Hung	Jonathan Hung
YARN-9894	CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues.	Major	capacity scheduler, test	Eric Payne	Eric Payne
HADOOP-16771	Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0	Major	build	Andras Bokor	Andras Bokor
YARN-10009	In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is defined	Critical	capacity scheduler	Eric Payne	Eric Payne
HDFS-12999	When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice.	Major	erasure-coding, hdfs-client	lufei	lufei
HDFS-15074	DataNode.DataTransfer thread should catch all the expception and log it.	Major	datanode	Surendra Singh Lilhore	Hemanth Boyina
HDFS-14740	Recover data blocks from persistent memory read cache during datanode restarts	Major	caching, datanode	Feilong He	Feilong He
HDFS-15097	Purge log in KMS and HttpFS	Minor	httpfs, kms	Doris Gu	Doris Gu
HDFS-14968	Add ability to know datanode staleness	Minor	datanode, logging, namenode	Ahmed Hussein	Ahmed Hussein
YARN-7913	Improve error handling when application recovery fails with exception	Major	resourcemanager	Gergo Repas	Wilfred Spiegelenburg
HDFS-15119	Allow expiration of cached locations in DFSInputStream	Minor	dfsclient	Ahmed Hussein	Ahmed Hussein
MAPREDUCE-7262	MRApp helpers block for long intervals (500ms)	Minor	mr-am	Ahmed Hussein	Ahmed Hussein
YARN-10084	Allow inheritance of max app lifetime / default app lifetime	Major	capacity scheduler	Eric Payne	Eric Payne
HDFS-12491	Support wildcard in CLASSPATH for libhdfs	Major	libhdfs	John Zhuge	Muhammad Samir Khan
YARN-10116	Expose diagnostics in RMAppManager summary	Major	.	Jonathan Hung	Jonathan Hung
HDFS-14758	Decrease lease hard limit	Minor	.	Eric Payne	Hemanth Boyina
HDFS-15086	Block scheduled counter never get decremet if the block got deleted before replication.	Major	3.1.1	Surendra Singh Lilhore	Hemanth Boyina
HDFS-15174	Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operations	Major	.	Lisheng Sun	Lisheng Sun
YARN-9018	Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given path	Major	.	Kuhu Shukla	Kuhu Shukla
HDFS-14861	Reset LowRedundancyBlocks Iterator periodically	Major	namenode	Stephen O'Donnell	Stephen O'Donnell
HADOOP-16882	Update jackson-databind to 2.9.10.2 in branch-3.1, branch-2.10	Blocker	.	Wei-Chiu Chuang	Lisheng Sun
HADOOP-16776	backport HADOOP-16775: distcp copies to s3 are randomly corrupted	Blocker	tools/distcp	Amir Shenavandeh	Amir Shenavandeh
HDFS-15197	[SBN read] Change ObserverRetryOnActiveException log to debug	Minor	hdfs	Chen Liang	Chen Liang
YARN-10200	Add number of containers to RMAppManager summary	Major	.	Jonathan Hung	Jonathan Hung
HADOOP-16952	Add .diff to gitignore	Minor	.	Ayush Saxena	Ayush Saxena
YARN-10212	Create separate configuration for max global AM attempts	Major	.	Jonathan Hung	Bilwa S T
YARN-9954	Configurable max application tags and max tag length	Major	.	Jonathan Hung	Bilwa S T
HADOOP-17001	The suffix name of the unified compression class	Major	io	bianqi	bianqi
HDFS-15295	AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance.	Minor	.	Jinglun	Jinglun
HADOOP-17127	Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTime	Minor	common	Jim Brennan	Jim Brennan

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-13806	EC: No error message for unsetting EC policy of the directory inherits the erasure coding policy from an ancestor directory	Minor	erasure-coding	Souryakanta Dwivedy	Ayush Saxena
HDFS-12459	Fix revert: Add new op GETFILEBLOCKLOCATIONS to WebHDFS REST API	Major	webhdfs	Weiwei Yang	Weiwei Yang
HADOOP-15418	Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationException	Major	common	Suma Shivaprasad	Suma Shivaprasad
HDFS-14004	TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunk	Major	.	Ayush Saxena	Ayush Saxena
HDFS-13959	TestUpgradeDomainBlockPlacementPolicy is flaky	Major	.	Ayush Saxena	Ayush Saxena
YARN-8948	PlacementRule interface should be for all YarnSchedulers	Major	.	Bibin Chundatt	Bibin Chundatt
HADOOP-16013	DecayRpcScheduler decay thread should run as a daemon	Major	ipc	Erik Krogen	Erik Krogen
HDFS-14175	EC: Native XOR decoder should reset the output buffer before using it.	Major	ec, hdfs	Surendra Singh Lilhore	Ayush Saxena
HDFS-14202	"dfs.disk.balancer.max.disk.throughputInMBperSec" property is not working as per set value.	Major	diskbalancer	Ranith Sardar	Ranith Sardar
HADOOP-16032	Distcp It should clear sub directory ACL before applying new ACL on it.	Major	tools/distcp	Ranith Sardar	Ranith Sardar
HADOOP-16127	In ipc.Client, put a new connection could happen after stop	Major	ipc	Tsz-wo Sze	Tsz-wo Sze
YARN-4901	QueueMetrics needs to be cleared before MockRM is initialized	Major	scheduler	Daniel Templeton	Peter Bacsko
HADOOP-16161	NetworkTopology#getWeightUsingNetworkLocation return unexpected result	Major	net	Xiaoqiao He	Xiaoqiao He
HDFS-14434	webhdfs that connect secure hdfs should not use user.name parameter	Minor	webhdfs	KWON BYUNGCHANG	KWON BYUNGCHANG
HDFS-14527	Stop all DataNodes may result in NN terminate	Major	namenode	Xiaoqiao He	Xiaoqiao He
HDFS-14494	Move Server logging of StatedId inside receiveRequestState()	Major	.	Konstantin Shvachko	Shweta
HDFS-14618	Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe).	Critical	.	Paul Ward	Paul Ward
HDFS-14610	HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized.	Critical	.	Paul Ward	Paul Ward
HDFS-14499	Misleading REM_QUOTA value with snapshot and trash feature enabled for a directory	Major	snapshots	Shashikant Banerjee	Shashikant Banerjee
HDFS-14647	NPE during secure namenode startup	Major	hdfs	Fengnan Li	Fengnan Li
HADOOP-16461	Regression: FileSystem cache lock parses XML within the lock	Major	fs	Gopal Vijayaraghavan	Gopal Vijayaraghavan
HDFS-14660	[SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProvider	Major	.	Chao Sun	Chao Sun
HDFS-14569	Result of crypto -listZones is not formatted properly	Major	.	Hemanth Boyina	Hemanth Boyina
HADOOP-12282	Connection thread's name should be updated after address changing is detected	Major	ipc	zhouyingchao	Lisheng Sun
HDFS-14686	HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns null	Major	httpfs	Siyao Meng	Siyao Meng
HADOOP-15865	ConcurrentModificationException in Configuration.overlay() method	Major	.	Oleksandr Shevchenko	Oleksandr Shevchenko
HDFS-14759	HDFS cat logs an info message	Major	.	Eric Badger	Eric Badger
HDFS-2470	NN should automatically set permissions on dfs.namenode.*.dir	Major	namenode	Aaron Myers	Siddharth Wagle
YARN-9718	Yarn REST API, services endpoint remote command ejection	Major	.	Eric Yang	Eric Yang
YARN-9813	RM does not start on JDK11 when UIv2 is enabled	Critical	resourcemanager, yarn	Adam Antal	Adam Antal
YARN-9820	RM logs InvalidStateTransitionException when app is submitted	Critical	.	Rohith Sharma K S	Prabhu Joseph
HDFS-14838	RBF: Display RPC (instead of HTTP) Port Number in RBF web UI	Minor	rbf, ui	Xieming Li	Xieming Li
HDFS-14699	Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold	Critical	ec	Zhao Yi Ming	Zhao Yi Ming
YARN-9833	Race condition when DirectoryCollection.checkDirs() runs during container launch	Major	.	Peter Bacsko	Peter Bacsko
YARN-9837	YARN Service fails to fetch status for Stopped apps with bigger spec files	Major	yarn-native-services	Tarun Parimi	Tarun Parimi
YARN-2255	YARN Audit logging not added to log4j.properties	Major	.	Varun Saxena	Aihua Xu
HDFS-14836	FileIoProvider should not increase FileIoErrors metric in datanode volume metric	Minor	.	Aiphago	Aiphago
HADOOP-16581	ValueQueue does not trigger an async refill when number of values falls below watermark	Major	common, kms	Yuval Degani	Yuval Degani
HDFS-14853	NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not present	Major	.	Ranith Sardar	Ranith Sardar
HDFS-13660	DistCp job fails when new data is appended in the file while the distCp copy job is running	Critical	distcp	Mukund Thakur	Mukund Thakur
HDFS-14808	EC: Improper size values for corrupt ec block in LOG	Major	ec	Harshakiran Reddy	Ayush Saxena
HDFS-14849	Erasure Coding: the internal block is replicated many times when datanode is decommissioning	Major	ec, erasure-coding	HuangTao	HuangTao
YARN-9858	Optimize RMContext getExclusiveEnforcedPartitions	Major	.	Jonathan Hung	Jonathan Hung
HDFS-14492	Snapshot memory leak	Major	snapshots	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-14418	Remove redundant super user priveledge checks from namenode.	Major	.	Ayush Saxena	Ayush Saxena
HDFS-14216	NullPointerException happens in NamenodeWebHdfs	Critical	.	lujie	lujie
HDFS-14637	Namenode may not replicate blocks to meet the policy after enabling upgradeDomain	Major	namenode	Stephen O'Donnell	Stephen O'Donnell
HDFS-14879	Header was wrong in Snapshot web UI	Major	.	Hemanth Boyina	Hemanth Boyina
HDFS-14655	[SBN Read] Namenode crashes if one of The JN is down	Critical	.	Harshakiran Reddy	Ayush Saxena
HDFS-14859	Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zero	Major	hdfs	Srinivasu Majeti	Srinivasu Majeti
YARN-6715	Fix documentation about NodeHealthScriptRunner	Major	documentation, nodemanager	Peter Bacsko	Peter Bacsko
YARN-9552	FairScheduler: NODE_UPDATE can cause NoSuchElementException	Major	fairscheduler	Peter Bacsko	Peter Bacsko
HDFS-14754	Erasure Coding : The number of Under-Replicated Blocks never reduced	Critical	ec	Hemanth Boyina	Hemanth Boyina
HDFS-14245	Class cast error in GetGroups with ObserverReadProxyProvider	Major	.	Shen Yinjie	Erik Krogen
HDFS-14373	EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripe	Critical	ec, hdfs-client	Surendra Singh Lilhore	Surendra Singh Lilhore
HDFS-14509	DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.x	Blocker	.	Yuxuan Wang	Yuxuan Wang
HDFS-14886	In NameNode Web UI's Startup Progress page, Loading edits always shows 0 sec	Major	.	Hemanth Boyina	Hemanth Boyina
YARN-8453	Additional Unit tests to verify queue limit and max-limit with multiple resource types	Major	capacity scheduler	Sunil G	Adam Antal
HDFS-14890	Setting permissions on name directory fails on non posix compliant filesystems	Blocker	.	hirik	Siddharth Wagle
HADOOP-16580	Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlException	Major	common	Adam Antal	Adam Antal
HDFS-14909	DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scope	Major	namenode	Surendra Singh Lilhore	Surendra Singh Lilhore
HADOOP-16600	StagingTestBase uses methods not available in Mockito 1.8.5 in branch-3.1	Major	.	Lisheng Sun	Duo Zhang
MAPREDUCE-6441	Improve temporary directory name generation in LocalDistributedCacheManager for concurrent processes	Major	.	William Watson	Haibo Chen
HADOOP-16662	Remove unnecessary InnerNode check in NetworkTopology#add()	Minor	.	Lisheng Sun	Lisheng Sun
HDFS-14847	Erasure Coding: Blocks are over-replicated while EC decommissioning	Critical	ec	Fei Hui	Fei Hui
HDFS-14913	Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType()	Major	.	Ayush Saxena	Ayush Saxena
YARN-9915	Fix FindBug issue in QueueMetrics	Minor	.	Prabhu Joseph	Prabhu Joseph
HDFS-12749	DN may not send block report to NN after NN restart	Major	datanode	TanYuxin	Xiaoqiao He
HDFS-13901	INode access time is ignored because of race between open and rename	Major	.	Jinglun	Jinglun
YARN-9921	Issue in PlacementConstraint when YARN Service AM retries allocation on component failure.	Major	.	Tarun Parimi	Tarun Parimi
HDFS-14910	Rename Snapshot with Pre Descendants Fail With IllegalArgumentException.	Blocker	.	Íñigo Goiri	Wei-Chiu Chuang
HDFS-14308	DFSStripedInputStream curStripeBuf is not freed by unbuffer()	Major	ec	Joe McDonnell	Zhao Yi Ming
HDFS-14931	hdfs crypto commands limit column width	Major	.	Eric Badger	Eric Badger
HADOOP-16669	TestRawLocalFileSystemContract.testPermission fails if no native library	Minor	common, test	Steve Loughran	Steve Loughran
HDFS-14920	Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommission	Major	ec	Fei Hui	Fei Hui
HDFS-13736	BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to false	Major	.	hu xiaodong	hu xiaodong
HDFS-14925	rename operation should check nest snapshot	Major	namenode	Junwang Zhao	Junwang Zhao
YARN-9949	Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfo	Minor	capacity scheduler	Prabhu Joseph	Prabhu Joseph
HDFS-14945	Revise PacketResponder's log.	Minor	datanode	Xudong Cao	Xudong Cao
HDFS-14946	Erasure Coding: Block recovery failed during decommissioning	Major	.	Fei Hui	Fei Hui
HDFS-14384	When lastLocatedBlock token expire, it will take 1~3s second to refetch it.	Major	hdfs-client	Surendra Singh Lilhore	Surendra Singh Lilhore
HDFS-14806	Bootstrap standby may fail if used in-progress tailing	Major	namenode	Chen Liang	Chen Liang
HDFS-14958	TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroup	Minor	hdfs	Jim Brennan	Jim Brennan
HDFS-14720	DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE.	Major	datanode	Surendra Singh Lilhore	Hemanth Boyina
HADOOP-16677	Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.	Minor	common	Xudong Cao	Xudong Cao
HDFS-14884	Add sanity check that zone key equals feinfo key while setting Xattrs	Major	encryption, hdfs	Mukul Kumar Singh	Mukul Kumar Singh
HADOOP-15097	AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading path	Minor	fs, test	zhoutai.zt	Xieming Li
YARN-9984	FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a node	Major	fairscheduler	Wilfred Spiegelenburg	Wilfred Spiegelenburg
YARN-9983	Typo in YARN Service overview documentation	Trivial	documentation	Denes Gerencser	Denes Gerencser
HADOOP-16719	Remove the disallowed element config within maven-checkstyle-plugin	Major	.	Wanqiang Ji	Wanqiang Ji
HADOOP-16700	RpcQueueTime may be negative when the response has to be sent later	Minor	.	xuzq	xuzq
HADOOP-15686	Supress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr	Major	kms	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-14940	HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TB	Minor	balancer & mover	Souryakanta Dwivedy	Hemanth Boyina
YARN-9838	Fix resource inconsistency for queues when moving app with reserved container to another queue	Critical	capacity scheduler	jiulongzhu	jiulongzhu
YARN-9968	Public Localizer is exiting in NodeManager due to NullPointerException	Major	nodemanager	Tarun Parimi	Tarun Parimi
YARN-9011	Race condition during decommissioning	Major	nodemanager	Peter Bacsko	Peter Bacsko
HDFS-14973	Balancer getBlocks RPC dispersal does not function properly	Major	balancer & mover	Erik Krogen	Erik Krogen
MAPREDUCE-7240	Exception ' Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER' cause job error	Critical	.	luhuachao	luhuachao
HDFS-14986	ReplicaCachingGetSpaceUsed throws ConcurrentModificationException	Major	datanode, performance	Ryan Wu	Aiphago
MAPREDUCE-7249	Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failure	Critical	applicationmaster, mrv2	Wilfred Spiegelenburg	Wilfred Spiegelenburg
HDFS-15010	BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static method	Major	datanode	Surendra Singh Lilhore	Surendra Singh Lilhore
HADOOP-16744	Fix building instruction to enable zstd	Minor	documentation	Masatake Iwasaki	Masatake Iwasaki
HDFS-14869	Data loss in case of distcp using snapshot diff. Replication should include rename records if file was skipped in the previous iteration	Major	distcp	Aasha Medhi	Aasha Medhi
HADOOP-16754	Fix docker failed to build yetus/hadoop	Blocker	build	Kevin Su	Kevin Su
HDFS-15032	Balancer crashes when it fails to contact an unavailable NN via ObserverReadProxyProvider	Major	balancer & mover	Erik Krogen	Erik Krogen
HDFS-15036	Active NameNode should not silently fail the image transfer	Major	namenode	Konstantin Shvachko	Chen Liang
HDFS-14519	NameQuota is not update after concat operation, so namequota is wrong	Major	.	Ranith Sardar	Ranith Sardar
YARN-10055	bower install fails	Blocker	build, yarn-ui-v2	Akira Ajisaka	Akira Ajisaka
HDFS-15076	Fix tests that hold FSDirectory lock, without holding FSNamesystem lock.	Major	test	Konstantin Shvachko	Konstantin Shvachko
HDFS-15073	Replace curator-shaded guava import with the standard one	Minor	hdfs-client	Akira Ajisaka	Chandra Sanivarapu
HADOOP-16042	Update the link to HadoopJavaVersion	Minor	documentation	Akira Ajisaka	Chandra Sanivarapu
HDFS-14934	[SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0	Major	.	Takanobu Asanuma	Ayush Saxena
YARN-10053	Placement rules do not use correct group service init	Major	yarn	Wilfred Spiegelenburg	Wilfred Spiegelenburg
HDFS-15068	DataNode could meet deadlock if invoke refreshVolumes when register	Major	datanode	Xiaoqiao He	Aiphago
MAPREDUCE-7255	Fix typo in MapReduce documentaion example	Trivial	documentation	Sergey Pogorelov	Sergey Pogorelov
HDFS-15072	HDFS MiniCluster fails to start when run in directory path with a %	Minor	.	Geoffrey Jacoby	Masatake Iwasaki
HDFS-15077	Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeout	Minor	test	Masatake Iwasaki	Masatake Iwasaki
HDFS-15080	Fix the issue in reading persistent memory cached data with an offset	Major	caching, datanode	Feilong He	Feilong He
YARN-7387	org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittently	Major	.	Miklos Szegedi	Jim Brennan
YARN-8672	TestContainerManager#testLocalingResourceWhileContainerRunning occasionally times out	Major	nodemanager	Jason Darrell Lowe	Chandni Singh
HDFS-14957	INodeReference Space Consumed was not same in QuotaUsage and ContentSummary	Major	namenode	Hemanth Boyina	Hemanth Boyina
MAPREDUCE-7252	Handling 0 progress in SimpleExponential task runtime estimator	Minor	.	Ahmed Hussein	Ahmed Hussein
HADOOP-16749	Configuration parsing of CDATA values are blank	Major	conf	Jonathan Turner Eagles	Daryn Sharp
HDFS-15095	Fix accidental comment in flaky test TestDecommissioningStatus	Major	hdfs	Ahmed Hussein	Ahmed Hussein
HDFS-15099	[SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException on ObserverNode	Major	namenode	Konstantin Shvachko	Chen Liang
HDFS-14578	AvailableSpaceBlockPlacementPolicy always prefers local node	Major	block placement	Wei-Chiu Chuang	Ayush Saxena
HADOOP-16683	Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlException	Major	common	Adam Antal	Adam Antal
MAPREDUCE-7256	Fix javadoc error in SimpleExponentialSmoothing	Minor	documentation	Masatake Iwasaki	Masatake Iwasaki
YARN-8373	RM Received RMFatalEvent of type CRITICAL_THREAD_CRASH	Major	fairscheduler, resourcemanager	Girish Bhat	Wilfred Spiegelenburg
MAPREDUCE-7247	Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to int	Major	documentation	zhaoshengjie
YARN-8148	Update decimal values for queue capacities shown on queue status CLI	Major	client	Prabhu Joseph	Prabhu Joseph
HADOOP-16808	Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire plugin	Minor	build	Akira Ajisaka	Xieming Li
HADOOP-16793	Remove WARN log when ipc connection interrupted in Client#handleSaslConnectionFailure()	Minor	.	Lisheng Sun	Lisheng Sun
YARN-9790	Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zero	Major	.	kyungwan nam	kyungwan nam
HDFS-14993	checkDiskError doesn't work during datanode startup	Major	datanode	Yang Yun	Yang Yun
HDFS-13179	TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittently	Critical	fs	Gabor Bota	Ahmed Hussein
MAPREDUCE-7259	testSpeculateSuccessfulWithUpdateEvents fails Intermittently	Minor	.	Ahmed Hussein	Ahmed Hussein
MAPREDUCE-7079	JobHistory#ServiceStop implementation is incorrect	Major	.	Jason Darrell Lowe	Ahmed Hussein
HDFS-15118	[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster.	Major	hdfs-client	Konstantin Shvachko	Chen Liang
HDFS-7175	Client-side SocketTimeoutException during Fsck	Major	namenode	Carl Steinbach	Stephen O'Donnell
HDFS-15148	dfs.namenode.send.qop.enabled should not apply to primary NN port	Major	.	Chen Liang	Chen Liang
HDFS-15115	Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debug	Major	.	wangzhixiang	wangzhixiang
HDFS-15157	Fix compilation failure for branch-3.1	Major	.	Ayush Saxena	Ayush Saxena
HDFS-15158	The number of failed volumes mismatch with volumeFailures of Datanode metrics	Minor	datanode	Yang Yun	Yang Yun
HADOOP-16849	start-build-env.sh behaves incorrectly when username is numeric only	Minor	build	Jihyun Cho	Jihyun Cho
HDFS-15161	When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close()	Major	.	Lisheng Sun	Lisheng Sun
HDFS-15164	Fix TestDelegationTokensWithHA	Major	.	Ayush Saxena	Ayush Saxena
HADOOP-16868	ipc.Server readAndProcess threw NullPointerException	Major	rpc-server	Tsz-wo Sze	Tsz-wo Sze
HADOOP-16869	Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failure	Major	build	Akira Ajisaka	Akira Ajisaka
HDFS-15052	WebHDFS getTrashRoot leads to OOM due to FileSystem object creation	Major	webhdfs	Wei-Chiu Chuang	Masatake Iwasaki
HDFS-15185	StartupProgress reports edits segments until the entire startup completes	Major	namenode	Konstantin Shvachko	Konstantin Shvachko
HDFS-15166	Remove redundant field fStream in ByteStringLog	Major	.	Konstantin Shvachko	Xieming Li
HADOOP-16841	The description of hadoop.http.authentication.signature.secret.file contains outdated information	Minor	documentation	Akira Ajisaka	Xieming Li
YARN-10156	Fix typo 'complaint' which means quite different in Federation.md	Minor	documentation, federation	Sungpeo Kook	Sungpeo Kook
HDFS-15147	LazyPersistTestCase wait logic is error-prone	Minor	.	Ahmed Hussein	Ahmed Hussein
HDFS-14668	Support Fuse with Users from multiple Security Realms	Critical	fuse-dfs	Sailesh Patel	Istvan Fajth
HDFS-15111	stopStandbyServices() should log which service state it is transitioning from.	Major	hdfs, logging	Konstantin Shvachko	Xieming Li
HDFS-15199	NPE in BlockSender	Major	.	Ayush Saxena	Ayush Saxena
HADOOP-16891	Upgrade jackson-databind to 2.9.10.3	Blocker	.	Siyao Meng	Siyao Meng
HADOOP-16840	AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucket	Major	fs/oss	wujinhu	wujinhu
YARN-9427	TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadically	Major	scheduler, test	Prabhu Joseph	Ahmed Hussein
HDFS-15216	Wrong Use Case of -showprogress in fsck	Major	.	Ravuri Sushma sree	Ravuri Sushma sree
HDFS-15211	EC: File write hangs during close in case of Exception during updatePipeline	Critical	.	Ayush Saxena	Ayush Saxena
HDFS-15208	Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfs	Trivial	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-15219	DFS Client will stuck when ResponseProcessor.run throw Error	Major	hdfs-client	zhengchenyu	zhengchenyu
HADOOP-16949	pylint fails in the build environment	Critical	build	Akira Ajisaka	Akira Ajisaka
HADOOP-14836	Upgrade maven-clean-plugin to 3.1.0	Major	build	Allen Wittenauer	Akira Ajisaka
MAPREDUCE-7272	TaskAttemptListenerImpl excessive log messages	Major	.	Ahmed Hussein	Ahmed Hussein
HDFS-15283	Cache pool MAXTTL is not persisted and restored on cluster restart	Major	namenode	Stephen O'Donnell	Stephen O'Donnell
HADOOP-16944	Use Yetus 0.12.0 in GitHub PR	Major	build	Akira Ajisaka	Akira Ajisaka
HDFS-15276	Concat on INodeRefernce fails with illegal state exception	Critical	.	Hemanth Boyina	Hemanth Boyina
HDFS-15281	ZKFC ignores dfs.namenode.rpc-bind-host and uses dfs.namenode.rpc-address to bind to host address	Major	ha, namenode	Dhiraj Hegde	Dhiraj Hegde
HDFS-15297	TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently in trunk	Major	datanode, test	Mingliang Liu	Ayush Saxena
HADOOP-17014	Upgrade jackson-databind to 2.9.10.4	Blocker	.	Siyao Meng	Siyao Meng
HDFS-15286	Concat on a same files deleting the file	Critical	.	Hemanth Boyina	Hemanth Boyina
HADOOP-15565	ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.	Major	.	Jinglun	Jinglun
HADOOP-17052	NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abort	Major	net	Dhiraj Hegde	Dhiraj Hegde
YARN-10347	Fix double locking in CapacityScheduler#reinitialize in branch-3.1	Critical	capacity scheduler	Masatake Iwasaki	Masatake Iwasaki

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-10072	TestCSAllocateCustomResource failures	Major	yarn	Jim Brennan	Jim Brennan
YARN-10161	TestRouterWebServicesREST is corrupting STDOUT	Minor	yarn	Jim Brennan	Jim Brennan
HADOOP-14206	TestSFTPFileSystem#testFileExists failure: Invalid encoding for signature	Major	fs, test	John Zhuge	Jim Brennan

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-15636	Add ITestDynamoDBMetadataStore	Minor	fs/s3, test	Sean Mackrory	Gabor Bota
HADOOP-15787	[JDK11] TestIPC.testRTEDuringConnectionSetup fails	Major	.	Akira Ajisaka	Zsolt Venczel
HDFS-14262	[SBN read] Unclear Log.WARN message in GlobalStateIdContext	Major	hdfs	Shweta	Shweta
HDFS-14590	[SBN Read] Add the document link to the top page	Major	documentation	Takanobu Asanuma	Takanobu Asanuma
YARN-9791	Queue Mutation API does not allow to remove a config	Major	capacity scheduler	Prabhu Joseph	Prabhu Joseph
HDFS-14822	[SBN read] Revisit GlobalStateIdContext locking when getting server state id	Major	hdfs	Chen Liang	Chen Liang
HDFS-14785	[SBN read] Change client logging to be less aggressive	Major	hdfs	Chen Liang	Chen Liang
YARN-9864	Format CS Configuration present in Configuration Store	Major	capacity scheduler	Prabhu Joseph	Prabhu Joseph
YARN-9801	SchedConfCli does not work with https mode	Major	.	Prabhu Joseph	Prabhu Joseph
HDFS-14858	[SBN read] Allow configurably enable/disable AlignmentContext on NameNode	Major	hdfs	Chen Liang	Chen Liang
HDFS-12979	StandbyNode should upload FsImage to ObserverNode after checkpointing.	Major	hdfs	Konstantin Shvachko	Chen Liang
YARN-9873	Mutation API Config Change need to update Version Number	Major	.	Prabhu Joseph	Prabhu Joseph
HDFS-14162	Balancer should work with ObserverNode	Major	.	Konstantin Shvachko	Erik Krogen
YARN-9773	Add QueueMetrics for Custom Resources	Major	.	Manikandan R	Manikandan R
HADOOP-16598	Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branches	Major	common	Duo Zhang	Duo Zhang
YARN-9950	Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectively	Major	capacity scheduler	Prabhu Joseph	Prabhu Joseph
YARN-9900	Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig Format	Major	capacity scheduler	Prabhu Joseph	Prabhu Joseph
HADOOP-16610	Upgrade to yetus 0.11.1 and use emoji vote on github pre commit	Major	build	Duo Zhang	Duo Zhang
YARN-9909	Offline format of YarnConfigurationStore	Major	capacity scheduler	Prabhu Joseph	Prabhu Joseph
YARN-9836	General usability improvements in showSimulationTrace.html	Minor	scheduler-load-simulator	Adam Antal	Adam Antal
HADOOP-16758	Refine testing.md to tell user better how to use auth-keys.xml	Minor	fs/s3	Mingliang Liu	Mingliang Liu
HADOOP-16609	Add Jenkinsfile for all active branches	Major	build	Duo Zhang	Akira Ajisaka
YARN-10022	Create RM Rest API to validate a CapacityScheduler Configuration	Major	.	Kinga Marton	Kinga Marton
HDFS-15173	RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable'	Minor	documentation, rbf	panlijie	panlijie
YARN-10139	ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GB	Major	.	Prabhu Joseph	Prabhu Joseph
HDFS-14731	[FGL] Remove redundant locking on NameNode.	Major	namenode	Konstantin Shvachko	Konstantin Shvachko
HADOOP-14918	Remove the Local Dynamo DB test option	Major	fs/s3	Steve Loughran	Gabor Bota
HDFS-15305	Extend ViewFS and provide ViewFSOverloadScheme implementation with scheme configurable.	Major	fs, hadoop-client, hdfs-client, viewfs	Uma Maheswara Rao G	Uma Maheswara Rao G

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-8016	Refine PlacementRule interface and add a app-name queue mapping rule as an example	Major	.	Zian Chen	Zian Chen
HADOOP-16491	Upgrade jetty version to 9.3.27	Major	.	Hrishikesh Gadre	Hrishikesh Gadre
HADOOP-16542	Update commons-beanutils version to 1.9.4	Major	.	Wei-Chiu Chuang	Kevin Su
HADOOP-16551	The changelog*.md seems not generated when create-release	Blocker	.	Zhankun Tang
YARN-9730	Support forcing configured partitions to be exclusive based on app node label	Major	.	Jonathan Hung	Jonathan Hung
HDFS-14959	[SBNN read] access time should be turned off	Major	documentation	Wei-Chiu Chuang	Chao Sun
HADOOP-16784	Update the year to 2020	Major	.	Ayush Saxena	Ayush Saxena
HADOOP-16871	Upgrade Netty version to 4.1.45.Final to handle CVE-2019-20444,CVE-2019-16869	Major	.	Aray Chenchu Sukesh	Aray Chenchu Sukesh
HADOOP-16982	Update Netty to 4.1.48.Final	Blocker	.	Wei-Chiu Chuang	Lisheng Sun