Apache Hadoop Changelog

Release 3.2.4 - 2022-07-12

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-16337	Show start time of Datanode on Web	Minor	.	Tao Li	Tao Li

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-15075	Remove process command timing from BPServiceActor	Major	.	Íñigo Goiri	Xiaoqiao He
HDFS-15150	Introduce read write lock to Datanode	Major	datanode	Stephen O'Donnell	Stephen O'Donnell
HDFS-16175	Improve the configurable value of Server #PURGE_INTERVAL_NANOS	Major	ipc	JiangHua Zhu	JiangHua Zhu
HDFS-16173	Improve CopyCommands#Put#executor queue configurability	Major	fs	JiangHua Zhu	JiangHua Zhu
HADOOP-17897	Allow nested blocks in switch case in checkstyle settings	Minor	build	Masatake Iwasaki	Masatake Iwasaki
HADOOP-17857	Check real user ACLs in addition to proxied user ACLs	Major	.	Eric Payne	Eric Payne
HDFS-14997	BPServiceActor processes commands from NameNode asynchronously	Major	datanode	Xiaoqiao He	Xiaoqiao He
HADOOP-17926	Maven-eclipse-plugin is no longer needed since Eclipse can import Maven projects by itself.	Minor	documentation	Rintaro Ikeda	Rintaro Ikeda
YARN-10935	AM Total Queue Limit goes below per-user AM Limit if parent is full.	Major	capacity scheduler, capacityscheduler	Eric Payne	Eric Payne
HDFS-16241	Standby close reconstruction thread	Major	.	zhanghuazong	zhanghuazong
YARN-1115	Provide optional means for a scheduler to check real user ACLs	Major	capacity scheduler, scheduler	Eric Payne
HDFS-16279	Print detail datanode info when process first storage report	Minor	.	Tao Li	Tao Li
HDFS-16294	Remove invalid DataNode#CONFIG_PROPERTY_SIMULATED	Major	datanode	JiangHua Zhu	JiangHua Zhu
HDFS-16299	Fix bug for TestDataNodeVolumeMetrics#verifyDataNodeVolumeMetrics	Minor	.	Tao Li	Tao Li
HDFS-16301	Improve BenchmarkThroughput#SIZE naming standardization	Minor	benchmarks, test	JiangHua Zhu	JiangHua Zhu
YARN-10997	Revisit allocation and reservation logging	Major	.	Andras Gyori	Andras Gyori
HDFS-16315	Add metrics related to Transfer and NativeCopy for DataNode	Major	.	Tao Li	Tao Li
HADOOP-17998	Allow get command to run with multi threads.	Major	fs	Chengwei Wang	Chengwei Wang
HDFS-16345	Fix test cases fail in TestBlockStoragePolicy	Major	build	guophilipse	guophilipse
HADOOP-18035	Skip unit test failures to run all the unit tests	Major	build	Akira Ajisaka	Akira Ajisaka
HADOOP-18040	Use maven.test.failure.ignore instead of ignoreTestFailure	Major	build	Akira Ajisaka	Akira Ajisaka
HDFS-16352	return the real datanode numBlocks in #getDatanodeStorageReport	Major	.	qinyuren	qinyuren
HDFS-16386	Reduce DataNode load when FsDatasetAsyncDiskService is working	Major	datanode	JiangHua Zhu	JiangHua Zhu
HDFS-16391	Avoid evaluation of LOG.debug statement in NameNodeHeartbeatService	Trivial	.	wangzhaohui	wangzhaohui
YARN-8234	Improve RM system metrics publisher's performance by pushing events to timeline server in batch	Critical	resourcemanager, timelineserver	Hu Ziqian	Ashutosh Gupta
HDFS-16430	Validate maximum blocks in EC group when adding an EC policy	Minor	ec, erasure-coding	daimin	daimin
HDFS-16403	Improve FUSE IO performance by supporting FUSE parameter max_background	Minor	fuse-dfs	daimin	daimin
HADOOP-18136	Verify FileUtils.unTar() handling of missing .tar files	Minor	test, util	Steve Loughran	Steve Loughran
HDFS-16529	Remove unnecessary setObserverRead in TestConsistentReadsObserver	Trivial	test	wangzhaohui	wangzhaohui
HDFS-16530	setReplication debug log creates a new string even if debug is disabled	Major	namenode	Stephen O'Donnell	Stephen O'Donnell
HDFS-16427	Add debug log for BlockManager#chooseExcessRedundancyStriped	Minor	erasure-coding	Tao Li	Tao Li
HDFS-16389	Improve NNThroughputBenchmark test mkdirs	Major	benchmarks, namenode	JiangHua Zhu	JiangHua Zhu
MAPREDUCE-7373	Building MapReduce NativeTask fails on Fedora 34+	Major	build, nativetask	Kengo Seki	Kengo Seki
HDFS-16355	Improve the description of dfs.block.scanner.volume.bytes.per.second	Minor	documentation, hdfs	guophilipse	guophilipse
HADOOP-18088	Replace log4j 1.x with reload4j	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-16501	Print the exception when reporting a bad block	Major	datanode	qinyuren	qinyuren
YARN-11116	Migrate Times util from SimpleDateFormat to thread-safe DateTimeFormatter class	Minor	.	Jonathan Turner Eagles	Jonathan Turner Eagles
YARN-10080	Support show app id on localizer thread pool	Major	nodemanager	zhoukang	Ashutosh Gupta
HADOOP-18240	Upgrade Yetus to 0.14.0	Major	build	Akira Ajisaka	Ashutosh Gupta
HDFS-16585	Add @VisibleForTesting in Dispatcher.java after HDFS-16268	Trivial	.	Wei-Chiu Chuang	Ashutosh Gupta
HDFS-16610	Make fsck read timeout configurable	Major	hdfs-client	Stephen O'Donnell	Stephen O'Donnell

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-13983	TestOfflineImageViewer crashes in windows	Major	.	Vinayakumar B	Vinayakumar B
YARN-9744	RollingLevelDBTimelineStore.getEntityByTime fails with NPE	Major	timelineserver	Prabhu Joseph	Prabhu Joseph
HDFS-15113	Missing IBR when NameNode restart if open processCommand async feature	Blocker	datanode	Xiaoqiao He	Xiaoqiao He
HADOOP-16985	Handle release package related issues	Major	.	Vinayakumar B	Vinayakumar B
HADOOP-17116	Skip Retry INFO logging on first failover from a proxy	Major	ha	Hanisha Koneru	Hanisha Koneru
HDFS-15651	Client could not obtain block when DN CommandProcessingThread exit	Major	.	Yiqun Lin	Mingxiang Li
HDFS-15963	Unreleased volume references cause an infinite loop	Critical	datanode	Shuyan Zhang	Shuyan Zhang
HDFS-14575	LeaseRenewer#daemon threads leak in DFSClient	Major	.	Tao Yang	Renukaprasad C
HADOOP-17796	Upgrade jetty version to 9.4.43	Major	.	Wei-Chiu Chuang	Renukaprasad C
HDFS-15175	Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog	Critical	.	Yicong Cai	Wan Chang
HDFS-16177	Bug fix for Util#receiveFile	Minor	.	Tao Li	Tao Li
YARN-10814	YARN shouldn't start with empty hadoop.http.authentication.signature.secret.file	Major	.	Benjamin Teke	Tamas Domok
HADOOP-17874	ExceptionsHandler to add terse/suppressed Exceptions in thread-safe manner	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-15129	Datanode caches namenode DNS lookup failure and cannot startup	Minor	ipc	Karthik Palaniappan	Chris Nauroth
YARN-10901	Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir	Major	nodemanager	Tamas Domok	Tamas Domok
HDFS-16207	Remove NN logs stack trace for non-existent xattr query	Major	namenode	Ahmed Hussein	Ahmed Hussein
HDFS-16187	SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing	Major	snapshots	Srinivasu Majeti	Shashikant Banerjee
HDFS-16198	Short circuit read leaks Slot objects when InvalidToken exception is thrown	Major	.	Eungsop Yoo	Eungsop Yoo
YARN-10870	Missing user filtering check -> yarn.webapp.filter-entity-list-by-user for RM Scheduler page	Major	yarn	Siddharth Ahuja	Gergely Pollák
HADOOP-17919	Fix command line example in Hadoop Cluster Setup documentation	Minor	documentation	Rintaro Ikeda	Rintaro Ikeda
HDFS-16235	Deadlock in LeaseRenewer for static remove method	Major	hdfs	angerszhu	angerszhu
HDFS-16181	[SBN Read] Fix metric of RpcRequestCacheMissAmount can't display when tailEditLog form JN	Critical	.	wangzhaohui	wangzhaohui
HADOOP-17925	BUILDING.txt should not encourage to activate docs profile on building binary artifacts	Minor	documentation	Rintaro Ikeda	Masatake Iwasaki
HADOOP-16532	Fix TestViewFsTrash to use the correct homeDir.	Minor	test, viewfs	Steve Loughran	Xing Lin
HDFS-16268	Balancer stuck when moving striped blocks due to NPE	Major	balancer & mover, erasure-coding	Leon Gao	Leon Gao
HDFS-7612	TestOfflineEditsViewer.testStored() uses incorrect default value for cacheDir	Major	test	Konstantin Shvachko	Michael Kuchenbecker
HDFS-16311	Metric metadataOperationRate calculation error in DataNodeVolumeMetrics	Major	.	Tao Li	Tao Li
HDFS-16182	numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage	Major	namanode	Max Xie	Max Xie
HADOOP-17999	No-op implementation of setWriteChecksum and setVerifyChecksum in ViewFileSystem	Major	.	Abhishek Das	Abhishek Das
HDFS-16329	Fix log format for BlockManager	Minor	.	Tao Li	Tao Li
HDFS-16330	Fix incorrect placeholder for Exception logs in DiskBalancer	Major	.	Viraj Jasani	Viraj Jasani
HDFS-16328	Correct disk balancer param desc	Minor	documentation, hdfs	guophilipse	guophilipse
HDFS-16343	Add some debug logs when the dfsUsed are not used during Datanode startup	Major	datanode	Mukul Kumar Singh	Mukul Kumar Singh
YARN-10991	Fix to ignore the grouping "[]" for resourcesStr in parseResourcesString method	Minor	distributed-shell	Ashutosh Gupta	Ashutosh Gupta
HADOOP-17975	Fallback to simple auth does not work for a secondary DistributedFileSystem instance	Major	ipc	István Fajth	István Fajth
HDFS-16350	Datanode start time should be set after RPC server starts successfully	Minor	.	Viraj Jasani	Viraj Jasani
YARN-11007	Correct words in YARN documents	Minor	documentation	guophilipse	guophilipse
HDFS-16332	Expired block token causes slow read due to missing handling in sasl handshake	Major	datanode, dfs, dfsclient	Shinya Yoshida	Shinya Yoshida
YARN-9063	ATS 1.5 fails to start if RollingLevelDb files are corrupt or missing	Major	timelineserver, timelineservice	Tarun Parimi	Ashutosh Gupta
HDFS-16333	fix balancer bug when transfer an EC block	Major	balancer & mover, erasure-coding	qinyuren	qinyuren
HDFS-16373	Fix MiniDFSCluster restart in case of multiple namenodes	Major	.	Ayush Saxena	Ayush Saxena
HDFS-16377	Should CheckNotNull before access FsDatasetSpi	Major	.	Tao Li	Tao Li
YARN-6862	Nodemanager resource usage metrics sometimes are negative	Major	nodemanager	YunFan Zhou	Benjamin Teke
YARN-10178	Global Scheduler async thread crash caused by 'Comparison method violates its general contract	Major	capacity scheduler	tuyu	Andras Gyori
HDFS-16395	Remove useless NNThroughputBenchmark#dummyActionNoSynch()	Major	benchmarks, namenode	JiangHua Zhu	JiangHua Zhu
HADOOP-18063	Remove unused import AbstractJavaKeyStoreProvider in Shell class	Minor	.	JiangHua Zhu	JiangHua Zhu
HDFS-16409	Fix typo: testHasExeceptionsReturnsCorrectValue -> testHasExceptionsReturnsCorrectValue	Trivial	.	Ashutosh Gupta	Ashutosh Gupta
HDFS-16408	Ensure LeaseRecheckIntervalMs is greater than zero	Major	namenode	Jingxuan Fu	Jingxuan Fu
YARN-11055	In cgroups-operations.c some fprintf format strings don't end with "\n"	Minor	nodemanager	Gera Shegalov	Gera Shegalov
HDFS-16303	Losing over 100 datanodes in state decommissioning results in full blockage of all datanode decommissioning	Major	.	Kevin Wikant	Kevin Wikant
HDFS-16443	Fix edge case where DatanodeAdminDefaultMonitor doubly enqueues a DatanodeDescriptor on exception	Major	hdfs	Kevin Wikant	Kevin Wikant
HDFS-16449	Fix hadoop web site release notes and changelog not available	Minor	documentation	guophilipse	guophilipse
HADOOP-18192	Fix multiple_bindings warning about slf4j-reload4j	Major	.	Masatake Iwasaki	Masatake Iwasaki
HDFS-16479	EC: NameNode should not send a reconstruction work when the source datanodes are insufficient	Critical	ec, erasure-coding	Yuanbo Liu	Takanobu Asanuma
HDFS-16509	Fix decommission UnsupportedOperationException: Remove unsupported	Major	namenode	daimin	daimin
HDFS-16456	EC: Decommission a rack with only on dn will fail when the rack number is equal with replication	Critical	ec, namenode	caozhiqiang	caozhiqiang
HDFS-16437	ReverseXML processor doesn't accept XML files without the SnapshotDiffSection.	Critical	hdfs	yanbin.zhang	yanbin.zhang
HDFS-16507	[SBN read] Avoid purging edit log which is in progress	Critical	.	Tao Li	Tao Li
YARN-10720	YARN WebAppProxyServlet should support connection timeout to prevent proxy server from hanging	Critical	.	Qi Zhu	Qi Zhu
HDFS-16428	Source path with storagePolicy cause wrong typeConsumed while rename	Major	hdfs, namenode	lei w	lei w
YARN-11014	YARN incorrectly validates maximum capacity resources on the validation API	Major	.	Benjamin Teke	Benjamin Teke
YARN-11075	Explicitly declare serialVersionUID in LogMutation class	Major	.	Benjamin Teke	Benjamin Teke
HDFS-11041	Unable to unregister FsDatasetState MBean if DataNode is shutdown twice	Trivial	datanode	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-16538	EC decoding failed due to not enough valid inputs	Major	erasure-coding	qinyuren	qinyuren
HDFS-16544	EC decoding failed due to invalid buffer	Major	erasure-coding	qinyuren	qinyuren
HDFS-16546	Fix UT TestOfflineImageViewer#testReverseXmlWithoutSnapshotDiffSection to branch branch-3.2	Major	test	daimin	daimin
HDFS-16552	Fix NPE for TestBlockManager	Major	.	Tao Li	Tao Li
MAPREDUCE-7246	In MapredAppMasterRest#Mapreduce_Application_Master_Info_API, the datatype of appId should be "string".	Major	documentation	jenny	Ashutosh Gupta
YARN-10187	Removing hadoop-yarn-project/hadoop-yarn/README as it is no longer maintained.	Minor	documentation	N Sanketh Reddy	Ashutosh Gupta
HDFS-16185	Fix comment in LowRedundancyBlocks.java	Minor	documentation	Akira Ajisaka	Ashutosh Gupta
HADOOP-17479	Fix the examples of hadoop config prefix	Minor	documentation	Akira Ajisaka	Ashutosh Gupta
HDFS-16579	Fix build failure for TestBlockManager on branch-3.2	Major	.	Tao Li	Tao Li
YARN-11092	Upgrade jquery ui to 1.13.1	Major	.	D M Murali Krishna Reddy	Ashutosh Gupta
YARN-11133	YarnClient gets the wrong EffectiveMinCapacity value	Major	api	Zilong Zhu	Zilong Zhu
YARN-10850	TimelineService v2 lists containers for all attempts when filtering for one	Major	timelinereader	Benjamin Teke	Benjamin Teke
YARN-11126	ZKConfigurationStore Java deserialisation vulnerability	Major	yarn	Tamas Domok	Tamas Domok
YARN-11162	Set the zk acl for nodes created by ZKConfigurationStore.	Major	resourcemanager	Owen O'Malley	Owen O'Malley
HDFS-16586	Purge FsDatasetAsyncDiskService threadgroup; it causes BPServiceActor$CommandProcessingThread IllegalThreadStateException 'fatal exception and exit'	Major	datanode	Michael Stack	Michael Stack
HADOOP-18251	Fix failure of extracting JIRA id from commit message in git_jira_fix_version_check.py	Minor	build	Masatake Iwasaki	Masatake Iwasaki
HDFS-16583	DatanodeAdminDefaultMonitor can get stuck in an infinite loop	Major	.	Stephen O'Donnell	Stephen O'Donnell
HDFS-16623	IllegalArgumentException in LifelineSender	Major	.	ZanderXu	ZanderXu
HDFS-16064	Determine when to invalidate corrupt replicas based on number of usable replicas	Major	datanode, namenode	Kevin Wikant	Kevin Wikant
HADOOP-18100	Change scope of inner classes in InodeTree to make them accessible outside package	Major	.	Abhishek Das	Abhishek Das
HADOOP-18334	Fix create-release to address removal of GPG_AGENT_INFO in branch-3.2	Major	build	Masatake Iwasaki	Masatake Iwasaki

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
MAPREDUCE-7342	Stop RMService in TestClientRedirect.testRedirect()	Minor	.	Zhengxi Li	Zhengxi Li
MAPREDUCE-7311	Fix non-idempotent test in TestTaskProgressReporter	Minor	.	Zhengxi Li	Zhengxi Li
HDFS-15862	Make TestViewfsWithNfs3.testNfsRenameSingleNN() idempotent	Minor	nfs	Zhengxi Li	Zhengxi Li

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-15457	TestFsDatasetImpl fails intermittently	Major	hdfs	Ahmed Hussein	Ahmed Hussein
HDFS-15818	Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig	Minor	test	Leon Gao	Leon Gao
YARN-10503	Support queue capacity in terms of absolute resources with custom resourceType.	Critical	.	Qi Zhu	Qi Zhu
HADOOP-17126	implement non-guava Precondition checkNotNull	Major	.	Ahmed Hussein	Ahmed Hussein
HADOOP-17929	implement non-guava Precondition checkArgument	Major	.	Ahmed Hussein	Ahmed Hussein
HADOOP-17947	Provide alternative to Guava VisibleForTesting	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-17930	implement non-guava Precondition checkState	Major	.	Ahmed Hussein	Ahmed Hussein
HADOOP-17374	AliyunOSS: support ListObjectsV2	Major	fs/oss	wujinhu	wujinhu
HDFS-16336	De-flake TestRollingUpgrade#testRollback	Minor	hdfs, test	Kevin Wikant	Viraj Jasani
HDFS-16171	De-flake testDecommissionStatus	Major	.	Viraj Jasani	Viraj Jasani
HDFS-16169	Fix TestBlockTokenWithDFSStriped#testEnd2End failure	Major	test	Hui Fei	secfree
HDFS-16484	[SPS]: Fix an infinite loop bug in SPSPathIdProcessor thread	Major	.	qinyuren	qinyuren
HADOOP-16663	Backport "HADOOP-16560 [YARN] use protobuf-maven-plugin to generate protobuf classes" to all active branches	Major	.	Duo Zhang	Duo Zhang
HADOOP-16664	Backport "HADOOP-16561 [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes" to all active branches	Major	.	Duo Zhang	Duo Zhang

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-16298	Improve error msg for BlockMissingException	Minor	.	Tao Li	Tao Li
HDFS-16312	Fix typo for DataNodeVolumeMetrics and ProfilingFileIoEvents	Minor	.	Tao Li	Tao Li
HDFS-16326	Simplify the code for DiskBalancer	Minor	.	Tao Li	Tao Li
HDFS-16339	Show the threshold when mover threads quota is exceeded	Minor	.	Tao Li	Tao Li
YARN-10820	Make GetClusterNodesRequestPBImpl thread safe	Major	client	Prabhu Joseph	SwathiChandrashekar
HADOOP-13464	update GSON to 2.7+	Minor	build	Sean Busbey	Igor Dvorzhak
HADOOP-18191	Log retry count while handling exceptions in RetryInvocationHandler	Minor	.	Viraj Jasani	Viraj Jasani
HDFS-16551	Backport HADOOP-17588 to 3.3 and other active old branches.	Major	.	Renukaprasad C	Renukaprasad C
HDFS-16618	sync_file_range error should include more volume and file info	Minor	.	Viraj Jasani	Viraj Jasani
HADOOP-18300	Update Gson to 2.9.0	Minor	build	Igor Dvorzhak	Igor Dvorzhak