Apache Hadoop Changelog

Release 3.2.3 - 2022-03-02

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-15691	Add PathCapabilities to FS and FC to complement StreamCapabilities	Major	.	Steve Loughran	Steve Loughran
HDFS-15711	Add Metrics to HttpFS Server	Major	httpfs	Ahmed Hussein	Ahmed Hussein
HDFS-15759	EC: Verify EC reconstruction correctness on DataNode	Major	datanode, ec, erasure-coding	Toshihiko Uchida	Toshihiko Uchida
HDFS-16337	Show start time of Datanode on Web	Minor	.	tomscut	tomscut

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-16052	Remove Subversion and Forrest from Dockerfile	Minor	build	Akira Ajisaka	Xieming Li
YARN-9783	Remove low-level zookeeper test to be able to build Hadoop against zookeeper 3.5.5	Major	test	Mate Szalay-Beko	Mate Szalay-Beko
HADOOP-16717	Remove GenericsUtil isLog4jLogger dependency on Log4jLoggerAdapter	Major	.	David Mollitor	Xieming Li
YARN-10036	Install yarnpkg and upgrade nodejs in Dockerfile	Major	buid, yarn-ui-v2	Akira Ajisaka	Akira Ajisaka
HADOOP-16811	Use JUnit TemporaryFolder Rule in TestFileUtils	Minor	common, test	David Mollitor	David Mollitor
HDFS-15075	Remove process command timing from BPServiceActor	Major	.	Íñigo Goiri	Xiaoqiao He
HADOOP-16054	Update Dockerfile to use Bionic	Major	build, test	Akira Ajisaka	Akira Ajisaka
HDFS-15574	Remove unnecessary sort of block list in DirectoryScanner	Major	.	Stephen O'Donnell	Stephen O'Donnell
HDFS-15583	Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1	Major	datanode	Stephen O'Donnell	Stephen O'Donnell
HDFS-15567	[SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly.	Major	ha, hdfs-client	Konstantin Shvachko	Konstantin Shvachko
HDFS-15633	Avoid redundant RPC calls for getDiskStatus	Major	dfsclient	Ayush Saxena	Ayush Saxena
YARN-10450	Add cpu and memory utilization per node and cluster-wide metrics	Minor	yarn	Jim Brennan	Jim Brennan
HDFS-15652	Make block size from NNThroughputBenchmark configurable	Minor	benchmarks	Hui Fei	Hui Fei
YARN-10475	Scale RM-NM heartbeat interval based on node utilization	Minor	yarn	Jim Brennan	Jim Brennan
HDFS-15665	Balancer logging improvement	Major	balancer & mover	Konstantin Shvachko	Konstantin Shvachko
HADOOP-17342	Creating a token identifier should not do kerberos name resolution	Major	common	Jim Brennan	Jim Brennan
YARN-10479	RMProxy should retry on SocketTimeout Exceptions	Major	yarn	Jim Brennan	Jim Brennan
HDFS-15623	Respect configured values of rpc.engine	Major	hdfs	Hector Sandoval Chaverri	Hector Sandoval Chaverri
HDFS-14395	Remove WARN Logging From Interrupts in DataStreamer	Minor	hdfs-client	David Mollitor	David Mollitor
HADOOP-17367	Add InetAddress api to ProxyUsers.authorize	Major	performance, security	Ahmed Hussein	Ahmed Hussein
HDFS-15694	Avoid calling UpdateHeartBeatState inside DataNodeDescriptor	Major	.	Ahmed Hussein	Ahmed Hussein
HDFS-15703	Don't generate edits for set operations that are no-op	Major	namenode	Ahmed Hussein	Ahmed Hussein
HADOOP-17392	Remote exception messages should not include the exception class	Major	ipc	Ahmed Hussein	Ahmed Hussein
HDFS-15706	HttpFS: Log more information on request failures	Major	httpfs	Ahmed Hussein	Ahmed Hussein
HADOOP-17389	KMS should log full UGI principal	Major	.	Ahmed Hussein	Ahmed Hussein
HDFS-15720	namenode audit async logger should add some log4j config	Minor	hdfs	Max Xie
HDFS-15704	Mitigate lease monitor's rapid infinite loop	Major	namenode	Ahmed Hussein	Ahmed Hussein
HDFS-15751	Add documentation for msync() API to filesystem.md	Major	documentation	Konstantin Shvachko	Konstantin Shvachko
YARN-10538	Add recommissioning nodes to the list of updated nodes returned to the AM	Major	.	Srinivas S T	Srinivas S T
YARN-4589	Diagnostics for localization timeouts is lacking	Major	.	Chang Li	Chang Li
YARN-10562	Follow up changes for YARN-9833	Major	yarn	Jim Brennan	Jim Brennan
HDFS-15783	Speed up BlockPlacementPolicyRackFaultTolerant#verifyBlockPlacement	Major	block placement	Akira Ajisaka	Akira Ajisaka
HADOOP-17478	Improve the description of hadoop.http.authentication.signature.secret.file	Minor	documentation	Akira Ajisaka	Akira Ajisaka
HDFS-15789	Lease renewal does not require namesystem lock	Major	hdfs	Jim Brennan	Jim Brennan
HADOOP-17501	Fix logging typo in ShutdownHookManager	Major	common	Konstantin Shvachko	Fengnan Li
HADOOP-17354	Move Jenkinsfile outside of the root directory	Major	build	Akira Ajisaka	Akira Ajisaka
HDFS-15799	Make DisallowedDatanodeException terse	Minor	hdfs	Richard	Richard
HDFS-15813	DataStreamer: keep sending heartbeat packets while streaming	Major	hdfs	Jim Brennan	Jim Brennan
MAPREDUCE-7319	Log list of mappers at trace level in ShuffleHandler audit log	Minor	yarn	Jim Brennan	Jim Brennan
HDFS-15821	Add metrics for in-service datanodes	Minor	.	Zehao Chen	Zehao Chen
YARN-10626	Log resource allocation in NM log at container start time	Major	.	Eric Badger	Eric Badger
HDFS-15815	if required storageType are unavailable, log the failed reason during choosing Datanode	Minor	block placement	Yang Yun	Yang Yun
HDFS-15826	Solve the problem of incorrect progress of delegation tokens when loading FsImage	Major	.	JiangHua Zhu	JiangHua Zhu
HDFS-15734	[READ] DirectoryScanner#scan need not check StorageType.PROVIDED	Minor	datanode	Yuxuan Wang	Yuxuan Wang
HADOOP-17538	Add kms-default.xml and httpfs-default.xml to site index	Minor	documentation	Masatake Iwasaki	Masatake Iwasaki
YARN-10613	Config to allow Intra- and Inter-queue preemption to enable/disable conservativeDRF	Minor	capacity scheduler, scheduler preemption	Eric Payne	Eric Payne
YARN-10653	Fixed the findbugs issues introduced by YARN-10647.	Major	.	Qi Zhu	Qi Zhu
MAPREDUCE-7324	ClientHSSecurityInfo class is in wrong META-INF file	Major	.	Eric Badger	Eric Badger
HADOOP-17546	Update Description of hadoop-http-auth-signature-secret in HttpAuthentication.md	Minor	.	Ravuri Sushma sree	Ravuri Sushma sree
YARN-10664	Allow parameter expansion in NM_ADMIN_USER_ENV	Major	yarn	Jim Brennan	Jim Brennan
HADOOP-17570	Apply YETUS-1102 to re-enable GitHub comments	Major	build	Akira Ajisaka	Akira Ajisaka
HADOOP-17594	DistCp: Expose the JobId for applications executing through run method	Major	.	Ayush Saxena	Ayush Saxena
HDFS-15911	Provide blocks moved count in Balancer iteration result	Major	balancer & mover	Viraj Jasani	Viraj Jasani
HDFS-15919	BlockPoolManager should log stack trace if unable to get Namenode addresses	Major	datanode	Stephen O'Donnell	Stephen O'Donnell
HADOOP-16870	Use spotbugs-maven-plugin instead of findbugs-maven-plugin	Major	build	Akira Ajisaka	Akira Ajisaka
HDFS-15932	Improve the balancer error message when process exits abnormally.	Major	.	Renukaprasad C	Renukaprasad C
HDFS-15931	Fix non-static inner classes for better memory management	Major	.	Viraj Jasani	Viraj Jasani
HDFS-15942	Increase Quota initialization threads	Major	namenode	Stephen O'Donnell	Stephen O'Donnell
HDFS-15937	Reduce memory used during datanode layout upgrade	Major	datanode	Stephen O'Donnell	Stephen O'Donnell
HADOOP-17569	Building native code fails on Fedora 33	Major	build, common	Kengo Seki	Masatake Iwasaki
HADOOP-17633	Bump json-smart to 2.4.2 and nimbus-jose-jwt to 9.8 due to CVEs	Major	auth, build	helen huang	Viraj Jasani
HADOOP-16822	Provide source artifacts for hadoop-client-api	Major	.	Karel Kolman	Karel Kolman
HADOOP-17680	Allow ProtobufRpcEngine to be extensible	Major	common	Hector Sandoval Chaverri	Hector Sandoval Chaverri
YARN-10123	Error message around yarn app -stop/start can be improved to highlight that an implementation at framework level is needed for the stop/start functionality to work	Minor	client, documentation	Siddharth Ahuja	Siddharth Ahuja
HADOOP-17756	Increase precommit job timeout from 20 hours to 24 hours.	Major	build	Takanobu Asanuma	Takanobu Asanuma
HDFS-16073	Remove redundant RPC requests for getFileLinkInfo in ClientNamenodeProtocolTranslatorPB	Minor	.	lei w	lei w
HDFS-16074	Remove an expensive debug string concatenation	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-15150	Introduce read write lock to Datanode	Major	datanode	Stephen O'Donnell	Stephen O'Donnell
YARN-10834	Intra-queue preemption: apps that don't use defined custom resource won't be preempted.	Major	.	Eric Payne	Eric Payne
HADOOP-17749	Remove lock contention in SelectorPool of SocketIOWithTimeout	Major	common	Xuesen Liang	Xuesen Liang
HADOOP-17775	Remove JavaScript package from Docker environment	Major	build	Masatake Iwasaki	Masatake Iwasaki
HADOOP-17794	Add a sample configuration to use ZKDelegationTokenSecretManager in Hadoop KMS	Major	documentation, kms, security	Akira Ajisaka	Akira Ajisaka
HADOOP-12665	Document hadoop.security.token.service.use_ip	Major	documentation	Arpit Agarwal	Akira Ajisaka
YARN-10456	RM PartitionQueueMetrics records are named QueueMetrics in Simon metrics registry	Major	resourcemanager	Eric Payne	Eric Payne
HDFS-15650	Make the socket timeout for computing checksum of striped blocks configurable	Minor	datanode, ec, erasure-coding	Yushi Hayasaka	Yushi Hayasaka
YARN-10858	[UI2] YARN-10826 breaks Queue view	Major	yarn-ui-v2	Andras Gyori	Masatake Iwasaki
YARN-10860	Make max container per heartbeat configs refreshable	Major	.	Eric Badger	Eric Badger
HADOOP-17813	Checkstyle - Allow line length: 100	Major	.	Akira Ajisaka	Viraj Jasani
HADOOP-17819	Add extensions to ProtobufRpcEngine RequestHeaderProto	Major	common	Hector Sandoval Chaverri	Hector Sandoval Chaverri
HDFS-16153	Avoid evaluation of LOG.debug statement in QuorumJournalManager	Trivial	.	wangzhaohui	wangzhaohui
HDFS-16154	TestMiniJournalCluster failing intermittently because of not reseting UserGroupInformation completely	Minor	.	wangzhaohui	wangzhaohui
HADOOP-17849	Exclude spotbugs-annotations from transitive dependencies on branch-3.2	Major	.	Masatake Iwasaki	Masatake Iwasaki
HDFS-16173	Improve CopyCommands#Put#executor queue configurability	Major	fs	JiangHua Zhu	JiangHua Zhu
HDFS-15160	ReplicaMap, Disk Balancer, Directory Scanner and various FsDatasetImpl methods should use datanode readlock	Major	datanode	Stephen O'Donnell	Stephen O'Donnell
HDFS-14997	BPServiceActor processes commands from NameNode asynchronously	Major	datanode	Xiaoqiao He	Xiaoqiao He
HDFS-16241	Standby close reconstruction thread	Major	.	zhanghuazong	zhanghuazong
HDFS-16286	Debug tool to verify the correctness of erasure coding on file	Minor	erasure-coding, tools	daimin	daimin
HADOOP-17998	Allow get command to run with multi threads.	Major	fs	Chengwei Wang	Chengwei Wang
HADOOP-18023	Allow cp command to run with multi threads.	Major	fs	Chengwei Wang	Chengwei Wang
HADOOP-17643	WASB : Make metadata checks case insensitive	Major	.	Anoop Sam John	Anoop Sam John
HDFS-16386	Reduce DataNode load when FsDatasetAsyncDiskService is working	Major	datanode	JiangHua Zhu	JiangHua Zhu
HDFS-16430	Validate maximum blocks in EC group when adding an EC policy	Minor	ec, erasure-coding	daimin	daimin
HDFS-16403	Improve FUSE IO performance by supporting FUSE parameter max_background	Minor	fuse-dfs	daimin	daimin
HADOOP-18093	Better exception handling for testFileStatusOnMountLink() in ViewFsBaseTest.java	Trivial	.	Xing Lin	Xing Lin
HADOOP-18155	Refactor tests in TestFileUtil	Trivial	common	Gautham Banasandra	Gautham Banasandra

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-15939	Filter overlapping objenesis class in hadoop-client-minicluster	Minor	build	Xiaoyu Yao	Xiaoyu Yao
YARN-8936	Bump up Atsv2 hbase versions	Major	.	Rohith Sharma K S	Vrushali C
HDFS-14189	Fix intermittent failure of TestNameNodeMetrics	Major	.	Ayush Saxena	Ayush Saxena
YARN-9246	NPE when executing a command yarn node -status or -states without additional arguments	Minor	client	Masahiro Tanaka	Masahiro Tanaka
YARN-7266	Timeline Server event handler threads locked	Major	ATSv2, timelineserver	Venkata Puneet Ravuri	Prabhu Joseph
YARN-9990	Testcase fails with "Insufficient configured threads: required=16 < max=10"	Major	.	Prabhu Joseph	Prabhu Joseph
YARN-10020	Fix build instruction of hadoop-yarn-ui	Minor	yarn-ui-v2	Masatake Iwasaki	Masatake Iwasaki
YARN-10037	Upgrade build tools for YARN Web UI v2	Major	build, security, yarn-ui-v2	Akira Ajisaka	Masatake Iwasaki
HDFS-15187	CORRUPT replica mismatch between namenodes after failover	Critical	.	Ayush Saxena	Ayush Saxena
HDFS-15200	Delete Corrupt Replica Immediately Irrespective of Replicas On Stale Storage	Critical	.	Ayush Saxena	Ayush Saxena
HDFS-15113	Missing IBR when NameNode restart if open processCommand async feature	Blocker	datanode	Xiaoqiao He	Xiaoqiao He
HDFS-15210	EC : File write hanged when DN is shutdown by admin command.	Major	ec	Surendra Singh Lilhore	Surendra Singh Lilhore
HADOOP-16768	SnappyCompressor test cases wrongly assume that the compressed data is always smaller than the input data	Major	io, test	zhao bo	Akira Ajisaka
HDFS-11041	Unable to unregister FsDatasetState MBean if DataNode is shutdown twice	Trivial	datanode	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-17068	client fails forever when namenode ipaddr changed	Major	hdfs-client	Sean Chow	Sean Chow
HDFS-15378	TestReconstructStripedFile#testErasureCodingWorkerXmitsWeight is failing on trunk	Major	.	Hemanth Boyina	Hemanth Boyina
YARN-10331	Upgrade node.js to 10.21.0	Critical	build, yarn-ui-v2	Akira Ajisaka	Akira Ajisaka
HADOOP-17119	Jetty upgrade to 9.4.x causes MR app fail with IOException	Major	.	Bilwa S T	Bilwa S T
HADOOP-17138	Fix spotbugs warnings surfaced after upgrade to 4.0.6	Minor	.	Masatake Iwasaki	Masatake Iwasaki
HDFS-15439	Setting dfs.mover.retry.max.attempts to negative value will retry forever.	Major	balancer & mover	AMC-team	AMC-team
YARN-10430	Log improvements in NodeStatusUpdaterImpl	Minor	nodemanager	Bilwa S T	Bilwa S T
HDFS-15438	Setting dfs.disk.balancer.max.disk.errors = 0 will fail the block copy	Major	balancer & mover	AMC-team	AMC-team
YARN-10438	Handle null containerId in ClientRMService#getContainerReport()	Major	resourcemanager	Raghvendra Singh	Shubham Gupta
HDFS-15628	HttpFS server throws NPE if a file is a symlink	Major	fs, httpfs	Ahmed Hussein	Ahmed Hussein
HDFS-15627	Audit log deletes before collecting blocks	Major	logging, namenode	Ahmed Hussein	Ahmed Hussein
HADOOP-17309	Javadoc warnings and errors are ignored in the precommit jobs	Major	build, documentation	Akira Ajisaka	Akira Ajisaka
HADOOP-17310	Touch command with -c option is broken	Major	.	Ayush Saxena	Ayush Saxena
HDFS-15639	[JDK 11] Fix Javadoc errors in hadoop-hdfs-client	Major	.	Takanobu Asanuma	Takanobu Asanuma
HDFS-15622	Deleted blocks linger in the replications queue	Major	hdfs	Ahmed Hussein	Ahmed Hussein
HDFS-15641	DataNode could meet deadlock if invoke refreshNameNode	Critical	.	Hongbing Wang	Hongbing Wang
MAPREDUCE-7302	Upgrading to JUnit 4.13 causes testcase TestFetcher.testCorruptedIFile() to fail	Major	test	Peter Bacsko	Peter Bacsko
HDFS-15644	Failed volumes can cause DNs to stop block reporting	Major	block placement, datanode	Ahmed Hussein	Ahmed Hussein
HADOOP-17236	Bump up snakeyaml to 1.26 to mitigate CVE-2017-18640	Major	.	Brahma Reddy Battula	Brahma Reddy Battula
YARN-10467	ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainers	Major	resourcemanager	Haibo Chen	Haibo Chen
HADOOP-17329	mvn site commands fails due to MetricsSystemImpl changes	Major	.	Xiaoqiao He	Xiaoqiao He
HDFS-15651	Client could not obtain block when DN CommandProcessingThread exit	Major	.	Yiqun Lin	Mingxiang Li
HADOOP-17340	TestLdapGroupsMapping failing -string mismatch in exception validation	Major	test	Steve Loughran	Steve Loughran
HADOOP-17352	Update PATCH_NAMING_RULE in the personality file	Minor	build	Akira Ajisaka	Akira Ajisaka
HDFS-15485	Fix outdated properties of JournalNode when performing rollback	Minor	.	Deegue	Deegue
HADOOP-17358	Improve excessive reloading of Configurations	Major	conf	Ahmed Hussein	Ahmed Hussein
HDFS-15538	Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xml	Major	.	Xieming Li	Xieming Li
HADOOP-17362	Doing hadoop ls on Har file triggers too many RPC calls	Major	fs	Ahmed Hussein	Ahmed Hussein
YARN-10485	TimelineConnector swallows InterruptedException	Major	.	Ahmed Hussein	Ahmed Hussein
HADOOP-17360	Log the remote address for authentication success	Minor	ipc	Ahmed Hussein	Ahmed Hussein
HADOOP-17346	Fair call queue is defeated by abusive service principals	Major	common, ipc	Ahmed Hussein	Ahmed Hussein
YARN-10470	When building new web ui with root user, the bower install should support it.	Major	build, yarn-ui-v2	Qi Zhu	Qi Zhu
YARN-10498	Fix Yarn CapacityScheduler Markdown document	Trivial	documentation	zhaoshengjie	zhaoshengjie
HDFS-15695	NN should not let the balancer run in safemode	Major	namenode	Ahmed Hussein	Ahmed Hussein
YARN-10511	Update yarn.nodemanager.env-whitelist value in docs	Minor	documentation	Andrea Scarpino	Andrea Scarpino
HDFS-15707	NNTop counts don't add up as expected	Major	hdfs, metrics, namenode	Ahmed Hussein	Ahmed Hussein
HDFS-15709	EC: Socket file descriptor leak in StripedBlockChecksumReconstructor	Major	datanode, ec, erasure-coding	Yushi Hayasaka	Yushi Hayasaka
YARN-10491	Fix deprecation warnings in SLSWebApp.java	Minor	build	Akira Ajisaka	Ankit Kumar
HADOOP-13571	ServerSocketUtil.getPort() should use loopback address, not 0.0.0.0	Major	.	Eric Badger	Eric Badger
HDFS-15725	Lease Recovery never completes for a committed block which the DNs never finalize	Major	namenode	Stephen O'Donnell	Stephen O'Donnell
HDFS-15170	EC: Block gets marked as CORRUPT in case of failover and pipeline recovery	Critical	erasure-coding	Ayush Saxena	Ayush Saxena
HDFS-15719	[Hadoop 3] Both NameNodes can crash simultaneously due to the short JN socket timeout	Critical	.	Wei-Chiu Chuang	Wei-Chiu Chuang
YARN-10560	Upgrade node.js to 10.23.1 and yarn to 1.22.5 in Web UI v2	Major	webapp, yarn-ui-v2	Akira Ajisaka	Akira Ajisaka
YARN-10528	maxAMShare should only be accepted for leaf queues, not parent queues	Major	.	Siddharth Ahuja	Siddharth Ahuja
HADOOP-17438	Increase docker memory limit in Jenkins	Major	build, scripts, test, yetus	Ahmed Hussein	Ahmed Hussein
MAPREDUCE-7310	Clear the fileMap in JHEventHandlerForSigtermTest	Minor	test	Zhengxi Li	Zhengxi Li
HADOOP-16947	Stale record should be remove when MutableRollingAverages generating aggregate data.	Major	.	Haibin Huang	Haibin Huang
HDFS-15632	AbstractContractDeleteTest should set recursive parameter to true for recursive test cases.	Major	.	Konstantin Shvachko	Anton Kutuzov
HDFS-10498	Intermittent test failure org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength.testSnapshotfileLength	Major	hdfs, snapshots	Hanisha Koneru	Jim Brennan
HADOOP-17506	Fix typo in BUILDING.txt	Trivial	documentation	Gautham Banasandra	Gautham Banasandra
HDFS-15795	EC: Wrong checksum when reconstruction was failed by exception	Major	datanode, ec, erasure-coding	Yushi Hayasaka	Yushi Hayasaka
HDFS-15779	EC: fix NPE caused by StripedWriter.clearBuffers during reconstruct block	Major	.	Hongbing Wang	Hongbing Wang
HDFS-15798	EC: Reconstruct task failed, and It would be XmitsInProgress of DN has negative number	Major	.	Haiyang Hu	Haiyang Hu
YARN-10428	Zombie applications in the YARN queue using FAIR + sizebasedweight	Critical	capacityscheduler	Guang Yang	Andras Gyori
YARN-10607	User environment is unable to prepend PATH when mapreduce.admin.user.env also sets PATH	Major	.	Eric Badger	Eric Badger
HADOOP-17516	Upgrade ant to 1.10.9	Major	.	Akira Ajisaka	Akira Ajisaka
YARN-10500	TestDelegationTokenRenewer fails intermittently	Major	test	Akira Ajisaka	Masatake Iwasaki
HADOOP-17534	Upgrade Jackson databind to 2.10.5.1	Major	build	Adam Roberts	Akira Ajisaka
MAPREDUCE-7323	Remove job_history_summary.py	Major	.	Akira Ajisaka	Akira Ajisaka
YARN-10647	Fix TestRMNodeLabelsManager failed after YARN-10501.	Major	.	Qi Zhu	Qi Zhu
HADOOP-17510	Hadoop prints sensitive Cookie information.	Major	.	Renukaprasad C	Renukaprasad C
HDFS-15422	Reported IBR is partially replaced with stored info when queuing.	Critical	namenode	Kihwal Lee	Stephen O'Donnell
YARN-10651	CapacityScheduler crashed with NPE in AbstractYarnScheduler.updateNodeResource()	Major	.	Haibo Chen	Haibo Chen
MAPREDUCE-7320	ClusterMapReduceTestCase does not clean directories	Major	.	Ahmed Hussein	Ahmed Hussein
HDFS-14013	Skip any credentials stored in HDFS when starting ZKFC	Major	hdfs	Krzysztof Adamski	Stephen O'Donnell
HDFS-15849	ExpiredHeartbeats metric should be of Type.COUNTER	Major	metrics	Konstantin Shvachko	Qi Zhu
YARN-10672	All testcases in TestReservations are flaky	Major	.	Szilard Nemeth	Szilard Nemeth
HADOOP-17557	skip-dir option is not processed by Yetus	Major	build, precommit, yetus	Ahmed Hussein	Ahmed Hussein
HDFS-15875	Check whether file is being truncated before truncate	Major	.	Hui Fei	Hui Fei
HADOOP-17582	Replace GitHub App Token with GitHub OAuth token	Major	build	Akira Ajisaka	Akira Ajisaka
YARN-10687	Add option to disable/enable free disk space checking and percentage checking for full and not-full disks	Major	nodemanager	Qi Zhu	Qi Zhu
HADOOP-17586	Upgrade org.codehaus.woodstox:stax2-api to 4.2.1	Major	.	Ayush Saxena	Ayush Saxena
HADOOP-17585	Correct timestamp format in the docs for the touch command	Major	.	Stephen O'Donnell	Stephen O'Donnell
YARN-10588	Percentage of queue and cluster is zero in WebUI	Major	.	Bilwa S T	Bilwa S T
MAPREDUCE-7322	revisiting TestMRIntermediateDataEncryption	Major	job submission, security, test	Ahmed Hussein	Ahmed Hussein
HADOOP-17592	Fix the wrong CIDR range example in Proxy User documentation	Minor	documentation	Kwangsun Noh	Kwangsun Noh
YARN-10706	Upgrade com.github.eirslett:frontend-maven-plugin to 1.11.2	Major	buid	Mingliang Liu	Mingliang Liu
MAPREDUCE-7325	Intermediate data encryption is broken in LocalJobRunner	Major	job submission, security	Ahmed Hussein	Ahmed Hussein
YARN-10697	Resources are displayed in bytes in UI for schedulers other than capacity	Major	.	Bilwa S T	Bilwa S T
HADOOP-17602	Upgrade JUnit to 4.13.1	Major	build, security, test	Ahmed Hussein	Ahmed Hussein
HDFS-15900	RBF: empty blockpool id on dfsrouter caused by UNAVAILABLE NameNode	Major	rbf	Harunobu Daikoku	Harunobu Daikoku
YARN-10501	Can't remove all node labels after add node label without nodemanager port	Critical	yarn	caozhiqiang	caozhiqiang
YARN-10716	Fix typo in ContainerRuntime	Trivial	documentation	Wanqiang Ji	xishuhai
HDFS-15950	Remove unused hdfs.proto import	Major	hdfs-client	Gautham Banasandra	Gautham Banasandra
HDFS-15949	Fix integer overflow	Major	libhdfs++	Gautham Banasandra	Gautham Banasandra
HDFS-15948	Fix test4tests for libhdfspp	Critical	build, libhdfs++	Gautham Banasandra	Gautham Banasandra
HADOOP-17608	Fix TestKMS failure	Major	kms	Akira Ajisaka	Akira Ajisaka
YARN-10460	Upgrading to JUnit 4.13 causes tests in TestNodeStatusUpdater to fail	Major	nodemanager, test	Peter Bacsko	Peter Bacsko
HADOOP-17641	ITestWasbUriAndConfiguration.testCanonicalServiceName() failing now mockaccount exists	Minor	fs/azure, test	Steve Loughran	Steve Loughran
HADOOP-17655	Upgrade Jetty to 9.4.40	Blocker	.	Akira Ajisaka	Akira Ajisaka
YARN-10749	Can't remove all node labels after add node label without nodemanager port, broken by YARN-10647	Major	.	D M Murali Krishna Reddy	D M Murali Krishna Reddy
HDFS-15621	Datanode DirectoryScanner uses excessive memory	Major	datanode	Stephen O'Donnell	Stephen O'Donnell
YARN-10756	Remove additional junit 4.11 dependency from javadoc	Major	build, test, timelineservice	ANANDA G B	Akira Ajisaka
YARN-10555	Missing access check before getAppAttempts	Critical	webapp	lujie	lujie
HADOOP-17703	checkcompatibility.py errors out when specifying annotations	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-14922	Build of Mapreduce Native Task module fails with unknown opcode "bswap"	Major	.	Anup Halarnkar	Anup Halarnkar
HADOOP-17718	Explicitly set locale in the Dockerfile	Blocker	build	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-17700	ExitUtil#halt info log should log HaltException	Major	.	Viraj Jasani	Viraj Jasani
YARN-10770	container-executor permission is wrong in SecureContainer.md	Major	documentation	Akira Ajisaka	Siddharth Ahuja
HDFS-15915	Race condition with async edits logging due to updating txId outside of the namesystem log	Major	hdfs, namenode	Konstantin Shvachko	Konstantin Shvachko
HDFS-16040	RpcQueueTime metric counts requeued calls as unique events.	Major	hdfs	Simbarashe Dzinamarira	Simbarashe Dzinamarira
YARN-10809	testWithHbaseConfAtHdfsFileSystem consistently failing	Major	.	Viraj Jasani	Viraj Jasani
HDFS-16055	Quota is not preserved in snapshot INode	Major	hdfs	Siyao Meng	Siyao Meng
HDFS-16068	WebHdfsFileSystem has a possible connection leak in connection with HttpFS	Major	.	Takanobu Asanuma	Takanobu Asanuma
YARN-10767	Yarn Logs Command retrying on Standby RM for 30 times	Major	.	D M Murali Krishna Reddy	D M Murali Krishna Reddy
HDFS-15618	Improve datanode shutdown latency	Major	datanode	Ahmed Hussein	Ahmed Hussein
HADOOP-17760	Delete hadoop.ssl.enabled and dfs.https.enable from docs and core-default.xml	Major	documentation	Takanobu Asanuma	Takanobu Asanuma
HDFS-13671	Namenode deletes large dir slowly caused by FoldedTreeSet#removeAndGet	Major	.	Yiqun Lin	Haibin Huang
HDFS-16061	DFTestUtil.waitReplication can produce false positives	Major	hdfs	Ahmed Hussein	Ahmed Hussein
HDFS-14575	LeaseRenewer#daemon threads leak in DFSClient	Major	.	Tao Yang	Renukaprasad C
YARN-10826	[UI2] Upgrade Node.js to at least v12.22.1	Major	yarn-ui-v2	Akira Ajisaka	Masatake Iwasaki
YARN-10828	Backport YARN-9789 to branch-3.2	Major	.	Tarun Parimi	Tarun Parimi
HADOOP-17769	Upgrade JUnit to 4.13.2	Major	.	Ahmed Hussein	Ahmed Hussein
YARN-10824	Title not set for JHS and NM webpages	Major	.	Rajshree Mishra	Bilwa S T
HDFS-16092	Avoid creating LayoutFlags redundant objects	Major	.	Viraj Jasani	Viraj Jasani
HDFS-16108	Incorrect log placeholders used in JournalNodeSyncer	Minor	.	Viraj Jasani	Viraj Jasani
MAPREDUCE-7353	Mapreduce job fails when NM is stopped	Major	.	Bilwa S T	Bilwa S T
HDFS-16121	Iterative snapshot diff report can generate duplicate records for creates, deletes and Renames	Major	snapshots	Srinivasu Majeti	Shashikant Banerjee
HDFS-15796	ConcurrentModificationException error happens on NameNode occasionally	Critical	hdfs	Daniel Ma	Daniel Ma
HADOOP-17793	Better token validation	Major	.	Artem Smotrakov	Artem Smotrakov
HDFS-16042	DatanodeAdminMonitor scan should be delay based	Major	datanode	Ahmed Hussein	Ahmed Hussein
HDFS-16127	Improper pipeline close recovery causes a permanent write failure or data loss.	Major	.	Kihwal Lee	Kihwal Lee
HADOOP-17028	ViewFS should initialize target filesystems lazily	Major	client-mounts, fs, viewfs	Uma Maheswara Rao G	Abhishek Das
HDFS-12920	HDFS default value change (with adding time unit) breaks old version MR tarball work with Hadoop 3.x	Critical	configuration, hdfs	Junping Du	Akira Ajisaka
YARN-10813	Set default capacity of root for node labels	Major	.	Andras Gyori	Andras Gyori
YARN-9551	TestTimelineClientV2Impl.testSyncCall fails intermittently	Minor	ATSv2, test	Prabhu Joseph	Andras Gyori
HDFS-15175	Multiple CloseOp shared block instance causes the standby namenode to crash when rolling editlog	Critical	.	Yicong Cai	Wan Chang
YARN-10789	RM HA startup can fail due to race conditions in ZKConfigurationStore	Major	.	Tarun Parimi	Tarun Parimi
YARN-6221	Entities missing from ATS when summary log file info got returned to the ATS before the domain log	Critical	yarn	Sushmitha Sreenivasan	Xiaomin Zhang
MAPREDUCE-7258	HistoryServerRest.html#Task_Counters_API, modify the jobTaskCounters's itemName from "taskcounterGroup" to "taskCounterGroup".	Minor	documentation	jenny	jenny
YARN-8990	Fix fair scheduler race condition in app submit and queue cleanup	Blocker	fairscheduler	Wilfred Spiegelenburg	Wilfred Spiegelenburg
YARN-8992	Fair scheduler can delete a dynamic queue while an application attempt is being added to the queue	Major	fairscheduler	Haibo Chen	Wilfred Spiegelenburg
HADOOP-17370	Upgrade commons-compress to 1.21	Major	common	Dongjoon Hyun	Akira Ajisaka
HADOOP-17844	Upgrade JSON smart to 2.4.7	Major	.	Renukaprasad C	Renukaprasad C
HADOOP-17850	Upgrade ZooKeeper to 3.4.14 in branch-3.2	Major	.	Akira Ajisaka	Masatake Iwasaki
HDFS-16177	Bug fix for Util#receiveFile	Minor	.	tomscut	tomscut
YARN-10814	YARN shouldn't start with empty hadoop.http.authentication.signature.secret.file	Major	.	Benjamin Teke	Tamas Domok
HADOOP-17858	Avoid possible class loading deadlock with VerifierNone initialization	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-17886	Upgrade ant to 1.10.11	Major	.	Ahmed Hussein	Ahmed Hussein
YARN-10901	Permission checking error on an existing directory in LogAggregationFileController#verifyAndCreateRemoteLogDir	Major	nodemanager	Tamas Domok	Tamas Domok
HDFS-16187	SnapshotDiff behaviour with Xattrs and Acls is not consistent across NN restarts with checkpointing	Major	snapshots	Srinivasu Majeti	Shashikant Banerjee
HDFS-16198	Short circuit read leaks Slot objects when InvalidToken exception is thrown	Major	.	Eungsop Yoo	Eungsop Yoo
HADOOP-17917	Backport HADOOP-15993 to branch-3.2 which address CVE-2014-4611	Major	.	Brahma Reddy Battula	Brahma Reddy Battula
HDFS-16233	Do not use exception handler to implement copy-on-write for EnumCounters	Major	namenode	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-16235	Deadlock in LeaseRenewer for static remove method	Major	hdfs	angerszhu	angerszhu
HADOOP-17940	Upgrade Kafka to 2.8.1	Major	.	Takanobu Asanuma	Takanobu Asanuma
HDFS-16272	Int overflow in computing safe length during EC block recovery	Critical	3.1.1	daimin	daimin
HADOOP-17971	Exclude IBM Java security classes from being shaded/relocated	Major	build	Nicholas Marion	Nicholas Marion
HADOOP-17972	Backport HADOOP-17683 for branch-3.2	Major	security	Ananya Singh	Ananya Singh
HADOOP-17993	Disable JIRA plugin for YETUS on Hadoop	Critical	build	Gautham Banasandra	Gautham Banasandra
HDFS-16182	numOfReplicas is given the wrong value in BlockPlacementPolicyDefault$chooseTarget can cause DataStreamer to fail with Heterogeneous Storage	Major	namanode	Max Xie	Max Xie
HDFS-16350	Datanode start time should be set after RPC server starts successfully	Minor	.	Viraj Jasani	Viraj Jasani
HADOOP-13500	Synchronizing iteration of Configuration properties object	Major	conf	Jason Darrell Lowe	Dhananjay Badaya
HDFS-16317	Backport HDFS-14729 for branch-3.2	Major	security	Ananya Singh	Ananya Singh
HDFS-14099	Unknown frame descriptor when decompressing multiple frames in ZStandardDecompressor	Major	.	xuzq	xuzq
HDFS-16410	Insecure Xml parsing in OfflineEditsXmlLoader	Minor	.	Ashutosh Gupta	Ashutosh Gupta
HDFS-16420	Avoid deleting unique data blocks when deleting redundancy striped blocks	Critical	ec, erasure-coding	qinyuren	Jackson Wang
HDFS-16428	Source path with storagePolicy cause wrong typeConsumed while rename	Major	hdfs, namenode	lei w	lei w
HDFS-16437	ReverseXML processor doesn't accept XML files without the SnapshotDiffSection.	Critical	hdfs	yanbin.zhang	yanbin.zhang
HDFS-16422	Fix thread safety of EC decoding during concurrent preads	Critical	dfsclient, ec, erasure-coding	daimin	daimin

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-9338	Timeline related testcases are failing	Major	.	Prabhu Joseph	Abhishek Modi
HDFS-15092	TestRedudantBlocks#testProcessOverReplicatedAndRedudantBlock sometimes fails	Minor	test	Hui Fei	Hui Fei

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-15775	[JDK9] Add missing javax.activation-api dependency	Critical	test	Akira Ajisaka	Akira Ajisaka
YARN-9875	FSSchedulerConfigurationStore fails to update with hdfs path	Major	capacityscheduler	Prabhu Joseph	Prabhu Joseph
HADOOP-16764	Rewrite Python example codes using Python3	Minor	documentation	Kengo Seki	Kengo Seki
HADOOP-16905	Update jackson-databind to 2.10.3 to relieve us from the endless CVE patches	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
YARN-10337	TestRMHATimelineCollectors fails on hadoop trunk	Major	test, yarn	Ahmed Hussein	Bilwa S T
HDFS-15464	ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount links	Major	viewfsOverloadScheme	Uma Maheswara Rao G	Uma Maheswara Rao G
HDFS-15478	When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.	Major	.	Uma Maheswara Rao G	Uma Maheswara Rao G
HDFS-15459	TestBlockTokenWithDFSStriped fails intermittently	Major	hdfs	Ahmed Hussein	Ahmed Hussein
HDFS-15461	TestDFSClientRetries#testGetFileChecksum fails intermittently	Major	dfsclient, test	Ahmed Hussein	Ahmed Hussein
HDFS-9776	TestHAAppend#testMultipleAppendsDuringCatchupTailing is flaky	Major	.	Vinayakumar B	Ahmed Hussein
HDFS-15457	TestFsDatasetImpl fails intermittently	Major	hdfs	Ahmed Hussein	Ahmed Hussein
HADOOP-17330	Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-3.2	Major	fs/azure	Sally Zuo	Sally Zuo
HDFS-15643	EC: Fix checksum computation in case of native encoders	Blocker	.	Ahmed Hussein	Ayush Saxena
HADOOP-17325	WASB: Test failures	Major	fs/azure, test	Sneha Vijayarajan	Steve Loughran
HDFS-15716	TestUpgradeDomainBlockPlacementPolicy flaky	Major	namenode, test	Ahmed Hussein	Ahmed Hussein
HDFS-15762	TestMultipleNNPortQOP#testMultipleNNPortOverwriteDownStream fails intermittently	Minor	.	Toshihiko Uchida	Toshihiko Uchida
HDFS-15672	TestBalancerWithMultipleNameNodes#testBalancingBlockpoolsWithBlockPoolPolicy fails on trunk	Major	.	Ahmed Hussein	Masatake Iwasaki
HDFS-15818	Fix TestFsDatasetImpl.testReadLockCanBeDisabledByConfig	Minor	test	Leon Gao	Leon Gao
HADOOP-16748	Migrate to Python 3 and upgrade Yetus to 0.13.0	Major	.	Akira Ajisaka	Akira Ajisaka
HDFS-15890	Improve the Logs for File Concat Operation	Minor	namenode	Bhavik Patel	Bhavik Patel
HDFS-13975	TestBalancer#testMaxIterationTime fails sporadically	Major	.	Jason Darrell Lowe	Toshihiko Uchida
YARN-10688	ClusterMetrics should support GPU capacity related metrics.	Major	metrics, resourcemanager	Qi Zhu	Qi Zhu
HDFS-15902	Improve the log for HTTPFS server operation	Minor	httpfs	Bhavik Patel	Bhavik Patel
HDFS-15940	Some tests in TestBlockRecovery are consistently failing	Major	.	Viraj Jasani	Viraj Jasani
YARN-10702	Add cluster metric for amount of CPU used by RM Event Processor	Minor	yarn	Jim Brennan	Jim Brennan
HADOOP-17630	[JDK 15] TestPrintableString fails due to Unicode 13.0 support	Major	.	Akira Ajisaka	Akira Ajisaka
YARN-10723	Change CS nodes page in UI to support custom resource.	Major	.	Qi Zhu	Qi Zhu
HADOOP-17112	whitespace not allowed in paths when saving files to s3a via committer	Blocker	fs/s3	Krzysztof Adamski	Krzysztof Adamski
HADOOP-17661	mvn versions:set fails to parse pom.xml	Blocker	build	Wei-Chiu Chuang	Wei-Chiu Chuang
YARN-10642	Race condition: AsyncDispatcher can get stuck by the changes introduced in YARN-8995	Critical	resourcemanager	zhengchenyu	zhengchenyu
HDFS-15659	Set dfs.namenode.redundancy.considerLoad to false in MiniDFSCluster	Major	test	Akira Ajisaka	Ahmed Hussein
HADOOP-17840	Backport HADOOP-17837 to branch-3.2	Minor	.	Bryan Beaudreault	Bryan Beaudreault
HADOOP-17126	implement non-guava Precondition checkNotNull	Major	.	Ahmed Hussein	Ahmed Hussein

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-15870	Remove unused configuration dfs.namenode.stripe.min	Minor	.	tomscut	tomscut
HDFS-15808	Add metrics for FSNamesystem read/write lock hold long time	Major	hdfs	tomscut	tomscut
HDFS-15873	Add namenode address in logs for block report	Minor	datanode, hdfs	tomscut	tomscut
HDFS-15906	Close FSImage and FSNamesystem after formatting is complete	Minor	.	tomscut	tomscut
HDFS-15892	Add metric for editPendingQ in FSEditLogAsync	Minor	.	tomscut	tomscut
HDFS-16078	Remove unused parameters for DatanodeManager.handleLifeline()	Minor	.	tomscut	tomscut
YARN-10278	CapacityScheduler test framework ProportionalCapacityPreemptionPolicyMockFramework need some review	Major	.	Gergely Pollák	Szilard Nemeth
HDFS-15731	Reduce threadCount for unit tests to reduce the memory usage	Major	build, test	Akira Ajisaka	Akira Ajisaka
HADOOP-17571	Upgrade com.fasterxml.woodstox:woodstox-core for security reasons	Major	.	Viraj Jasani	Viraj Jasani
HDFS-15895	DFSAdmin#printOpenFiles has redundant String#format usage	Minor	.	Viraj Jasani	Viraj Jasani
HADOOP-17614	Bump netty to the latest 4.1.61	Blocker	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-17627	Backport to branch-3.2 HADOOP-17371, HADOOP-17621, HADOOP-17625 to update Jetty to 9.4.39	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-15989	Split TestBalancer into two classes	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-17808	ipc.Client not setting interrupt flag after catching InterruptedException	Minor	.	Viraj Jasani	Viraj Jasani
HADOOP-17834	Bump aliyun-sdk-oss to 3.13.0	Major	.	Siyao Meng	Siyao Meng
HADOOP-17955	Bump netty to the latest 4.1.68	Major	.	Takanobu Asanuma	Takanobu Asanuma
HADOOP-18061	Update the year to 2022	Major	.	Ayush Saxena	Ayush Saxena
HADOOP-18125	Utility to identify git commit / Jira fixVersion discrepancies for RC preparation	Major	.	Viraj Jasani	Viraj Jasani