Apache Hadoop Changelog

Release 3.3.6 - 2023-06-19

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-18671	Add recoverLease(), setSafeMode(), isFileClosed() APIs to FileSystem	Major	fs	Wei-Chiu Chuang	Tak-Lon (Stephen) Wu

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-15368	TestBalancerWithHANameNodes#testBalancerWithObserver failed occasionally	Major	.	Xiaoqiao He	Xiaoqiao He
HDFS-15383	RBF: Disable watch in ZKDelegationSecretManager for performance	Major	.	Fengnan Li	Fengnan Li
HDFS-15803	EC: Remove unnecessary method (getWeight) in StripedReconstructionInfo	Trivial	.	Haiyang Hu	Haiyang Hu
HDFS-16024	RBF: Rename data to the Trash should be based on src locations	Major	rbf	Xiangyi Zhu	Xiangyi Zhu
HDFS-16016	BPServiceActor add a new thread to handle IBR	Minor	.	JiangHua Zhu	Viraj Jasani
HADOOP-17749	Remove lock contention in SelectorPool of SocketIOWithTimeout	Major	common	Xuesen Liang	Xuesen Liang
HDFS-16480	Fix typo: indicies -> indices	Minor	.	Jiale Qi	Jiale Qi
HADOOP-18466	Limit the findbugs suppression IS2_INCONSISTENT_SYNC to S3AFileSystem field	Minor	fs/s3	Viraj Jasani	Viraj Jasani
MAPREDUCE-7370	Parallelize MultipleOutputs#close call	Major	.	Prabhu Joseph	Ashutosh Gupta
YARN-11360	Add number of decommissioning/shutdown nodes to YARN cluster metrics.	Major	client, resourcemanager	Chris Nauroth	Chris Nauroth
HADOOP-18472	Upgrade to snakeyaml 1.33	Major	.	PJ Fanning	PJ Fanning
HDFS-16811	Support DecommissionBackoffMonitor parameters reconfigurable	Major	.	Haiyang Hu	Haiyang Hu
HADOOP-18433	Fix main thread name.	Major	common, ipc	zhengchenyu	zhengchenyu
HDFS-16851	RBF: Add a utility to dump the StateStore	Major	rbf	Owen O'Malley	Owen O'Malley
HDFS-16839	It should consider EC reconstruction work when we determine if a node is busy	Major	ec, erasure-coding	Kidd5368	Kidd5368
HDFS-16887	Log start and end of phase/step in startup progress	Minor	namenode	Viraj Jasani	Viraj Jasani
HDFS-16891	Avoid the overhead of copy-on-write exception list while loading inodes sub sections in parallel	Major	namenode	Viraj Jasani	Viraj Jasani
HADOOP-18604	Add compile platform in the hadoop version output	Major	.	Ayush Saxena	Ayush Saxena
HDFS-16888	BlockManager#maxReplicationStreams, replicationStreamsHardLimit, blocksReplWorkMultiplier and PendingReconstructionBlocks#timeout should be volatile	Major	.	Haiyang Hu	Haiyang Hu
HADOOP-18592	Sasl connection failure should log remote address	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-18625	Fix method name of RPC.Builder#setnumReaders	Minor	ipc	Haiyang Hu	Haiyang Hu
HDFS-16882	RBF: Add cache hit rate metric in MountTableResolver#getDestinationForPath	Minor	rbf	farmmamba	farmmamba
HDFS-16907	Add LastHeartbeatResponseTime for BP service actor	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-18628	Server connection should log host name before returning VersionMismatch error	Minor	ipc	Viraj Jasani	Viraj Jasani
HADOOP-18596	Distcp -update between different cloud stores to use modification time while checking for file skip.	Major	tools/distcp	Mehakmeet Singh	Mehakmeet Singh
HADOOP-18215	Enhance WritableName to be able to return aliases for classes that use serializers	Minor	.	Bryan Beaudreault	Bryan Beaudreault
HADOOP-18622	Upgrade ant to 1.10.13	Major	.	Aleksandr Nikolaev	Aleksandr Nikolaev
YARN-11394	Fix hadoop-yarn-server-resourcemanager module Java Doc Errors.	Major	resourcemanager	Shilun Fan	Shilun Fan
HADOOP-18535	Implement token storage solution based on MySQL	Major	.	Hector Sandoval Chaverri	Hector Sandoval Chaverri
HADOOP-18646	Upgrade Netty to 4.1.89.Final	Major	build	Aleksandr Nikolaev	Aleksandr Nikolaev
HADOOP-18684	S3A filesystem to support binding to other URI schemes	Major	.	Harshit Gupta	Harshit Gupta
HADOOP-18590	Publish SBOM artifacts	Major	build	Dongjoon Hyun	Dongjoon Hyun
HADOOP-18597	Simplify single node instructions for creating directories for Map Reduce	Trivial	documentation	Nikita Eshkeev	Nikita Eshkeev
HADOOP-18691	Add a CallerContext getter on the Schedulable interface	Major	.	Christos Bisias	Christos Bisias
HADOOP-18689	Bump jettison from 1.5.3 to 1.5.4 in /hadoop-project	Major	common	Ayush Saxena
HDFS-16988	Improve NameServices info at JournalNode web UI	Minor	.	Zhaohui Wang	Zhaohui Wang
HADOOP-18637	S3A to support upload of files greater than 2 GB using DiskBlocks	Major	fs/s3	Harshit Gupta	Harshit Gupta
HADOOP-18695	S3A: reject multipart copy requests when disabled	Minor	fs/s3	Steve Loughran	Steve Loughran

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-16628	RBF: Correct target directory when move to trash for kerberos login user.	Major	rbf	Xiping Zhang	Xiping Zhang
HDFS-16633	Reserved Space For Replicas is not released on some cases	Major	hdfs	Prabhu Joseph	Ashutosh Gupta
HDFS-16566	Erasure Coding: Recovery may cause excess replicas when busy DN exsits	Major	ec, erasure-coding	Kidd5368	Kidd5368
HADOOP-18471	An unhandled ArrayIndexOutOfBoundsException in DefaultStringifier.storeArray() if provided with an empty input	Minor	common, io	FuzzingTeam	FuzzingTeam
HDFS-16809	EC striped block is not sufficient when doing in maintenance	Major	ec, erasure-coding	dingshun	dingshun
YARN-11390	TestResourceTrackerService.testNodeRemovalNormally: Shutdown nodes should be 0 now expected: <1> but was: <0>	Major	yarn	Bence Kosztolnik	Bence Kosztolnik
HDFS-16852	Register the shutdown hook only when not in shutdown for KeyProviderCache constructor	Minor	hdfs	Xing Lin	Xing Lin
HADOOP-18567	LogThrottlingHelper: the dependent recorder is not triggered correctly	Major	.	Chengbing Liu
YARN-11395	Resource Manager UI, cluster/appattempt/*, can not present FINAL_SAVING state	Critical	yarn	Bence Kosztolnik	Bence Kosztolnik
YARN-11392	ClientRMService implemented getCallerUgi and verifyUserAccessForRMApp methods but forget to use sometimes, caused audit log missing.	Major	yarn	Beibei Zhao	Beibei Zhao
HDFS-16872	Fix log throttling by declaring LogThrottlingHelper as static members	Major	.	Chengbing Liu
HADOOP-18591	Fix a typo in Trash	Minor	documentation	xiaoping.huang	xiaoping.huang
HDFS-16764	ObserverNamenode handles addBlock rpc and throws a FileNotFoundException	Critical	.	ZanderXu	ZanderXu
HADOOP-18584	[NFS GW] Fix regression after netty4 migration	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-18279	Cancel fileMonitoringTimer even if trustManager isn't defined	Major	common, test	Steve Vaughan	Steve Vaughan
HADOOP-18576	Java 11 JavaDoc fails due to missing package comments	Major	build, common	Steve Loughran	Steve Vaughan
HADOOP-18612	Avoid mixing canonical and non-canonical when performing comparisons	Minor	common, test	Steve Vaughan	Steve Vaughan
HDFS-16925	Namenode audit log to only include IP address of client	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-18633	fix test AbstractContractDistCpTest#testDistCpUpdateCheckFileSkip	Major	tools/distcp	Mehakmeet Singh	Mehakmeet Singh
HADOOP-18582	No need to clean tmp files in distcp direct mode	Major	tools/distcp	10000kang	10000kang
HADOOP-18636	LocalDirAllocator cannot recover from directory tree deletion during the life of a filesystem client	Minor	fs, fs/azure, fs/s3	Steve Loughran	Steve Loughran
HDFS-16935	TestFsDatasetImpl.testReportBadBlocks brittle	Minor	test	Steve Loughran	Viraj Jasani
HDFS-16942	Send error to datanode if FBR is rejected due to bad lease	Major	datanode, namenode	Stephen O'Donnell	Stephen O'Donnell
HADOOP-18662	ListFiles with recursive fails with FNF	Major	.	Ayush Saxena	Ayush Saxena
HADOOP-18680	Insufficient heap during full test runs in Docker container.	Minor	build	Chris Nauroth	Chris Nauroth
HADOOP-18714	Wrong StringUtils.join() called in AbstractContractRootDirectoryTest	Trivial	test	Attila Doroszlai	Attila Doroszlai
HADOOP-18705	ABFS should exclude incompatible credential providers	Major	fs/azure	Tamas Domok	Tamas Domok
HADOOP-18660	Filesystem Spelling Mistake	Trivial	fs	Sebastian Baunsgaard	Sebastian Baunsgaard
MAPREDUCE-7437	MR Fetcher class to use an AtomicInteger to generate IDs.	Major	build, client	Steve Loughran	Steve Loughran
HDFS-16672	Fix lease interval comparison in BlockReportLeaseManager	Trivial	namenode	dzcxzl	dzcxzl
YARN-11482	Fix bug of DRF comparison DominantResourceFairnessComparator2 in fair scheduler	Major	fairscheduler	Xiaoqiao He	Xiaoqiao He
HDFS-16897	Fix abundant Broken pipe exception in BlockSender	Minor	hdfs	fanluo	fanluo
HADOOP-18715	Add debug log for getting details of tokenKindMap	Minor	.	Pralabh Kumar	Pralabh Kumar
YARN-11312	[UI2] Refresh buttons don't work after EmberJS upgrade	Minor	yarn-ui-v2	Brian Goerlitz	Susheel Gupta
HADOOP-18724	Open file fails with NumberFormatException for S3AFileSystem	Critical	fs, fs/azure, fs/s3	Ayush Saxena	Steve Loughran
HADOOP-18652	Path.suffix raises NullPointerException	Minor	hdfs-client	Patrick Grandjean	Patrick Grandjean
HDFS-17022	Fix the exception message to print the Identifier pattern	Minor	.	Nishtha Shah	Nishtha Shah
HDFS-17017	Fix the issue of arguments number limit in report command in DFSAdmin.	Major	.	Haiyang Hu	Haiyang Hu
HADOOP-18755	openFile builder new optLong() methods break hbase-filesystem	Major	fs	Steve Loughran	Steve Loughran
HDFS-17011	Fix the metric of "HttpPort" at DataNodeInfo	Minor	.	Zhaohui Wang	Zhaohui Wang
HDFS-17003	Erasure Coding: invalidate wrong block after reporting bad blocks from datanode	Critical	namenode	farmmamba	farmmamba
HADOOP-18718	Fix several maven build warnings	Minor	build	Dongjoon Hyun	Dongjoon Hyun

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-11388	Prevent resource leaks in TestClientRMService.	Minor	test	Chris Nauroth	Chris Nauroth

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-15654	TestBPOfferService#testMissBlocksWhenReregister fails intermittently	Major	datanode	Ahmed Hussein	Ahmed Hussein
HDFS-15674	TestBPOfferService#testMissBlocksWhenReregister fails on trunk	Major	.	Ahmed Hussein	Masatake Iwasaki
HADOOP-18380	fs.s3a.prefetch.block.size to be read through longBytesOption	Major	fs/s3	Steve Loughran	Viraj Jasani
HADOOP-18186	s3a prefetching to use SemaphoredDelegatingExecutor for submitting work	Major	fs/s3	Steve Loughran	Viraj Jasani
HADOOP-18377	hadoop-aws maven build to add a prefetch profile to run all tests with prefetching	Major	fs/s3, test	Steve Loughran	Viraj Jasani
HADOOP-18455	s3a prefetching Executor should be closed	Major	fs/s3	Viraj Jasani	Viraj Jasani
HADOOP-18378	Implement readFully(long position, byte[] buffer, int offset, int length)	Minor	fs/s3	Ahmar Suhail	Alessandro Passaro
HADOOP-18189	S3PrefetchingInputStream to support status probes when closed	Minor	fs/s3	Steve Loughran	Viraj Jasani
HADOOP-18156	Address JavaDoc warnings in classes like MarkerTool, S3ObjectAttributes, etc.	Minor	fs/s3	Mukund Thakur	Ankit Saurabh
HADOOP-18482	ITestS3APrefetchingInputStream does not skip if no CSV test file available	Minor	fs/s3	Daniel Carl Jones	Daniel Carl Jones
HADOOP-18531	assertion failure in ITestS3APrefetchingInputStream	Major	fs/s3, test	Steve Loughran	Ashutosh Gupta
HADOOP-18620	Avoid using grizzly-http-* APIs	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-18351	S3A prefetching: Error logging during reads	Minor	.	Ahmar Suhail	Ankit Saurabh
HADOOP-17746	compatibility table in directory_markers.md doesn't render right	Minor	fs/s3, site	Steve Loughran	Masatake Iwasaki
HADOOP-18606	Add reason in in x-ms-client-request-id on a retry API call.	Minor	fs/azure	Pranav Saxena	Pranav Saxena
HADOOP-18146	ABFS: Add changes for expect hundred continue header with append requests	Major	fs/azure	Anmol Asrani	Anmol Asrani
HADOOP-18647	x-ms-client-request-id to have some way that identifies retry of an API.	Minor	fs/azure	Pranav Saxena	Pranav Saxena
HADOOP-18012	ABFS: Enable config controlled ETag check for Rename idempotency	Major	fs/azure	Sneha Vijayarajan	Sree Bhattacharyya
HADOOP-18696	S3A ITestS3ABucketExistence access point test failure	Major	fs/s3, test	Steve Loughran	Steve Loughran
HADOOP-18399	S3A Prefetch - SingleFilePerBlockCache to use LocalDirAllocator	Major	fs/s3	Steve Loughran	Viraj Jasani
HADOOP-18703	Backport S3A prefetching stream to branch-3.3	Major	fs/s3	Steve Loughran	Steve Loughran
HADOOP-18697	Fix transient failure of ITestS3APrefetchingInputStream#testRandomReadLargeFile	Major	fs/s3, test	Steve Loughran	Viraj Jasani
HADOOP-18688	S3A audit header to include count of items in delete ops	Major	fs/s3	Steve Loughran	Viraj Jasani
HADOOP-18740	s3a prefetch cache blocks should be accessed by RW locks	Major	.	Viraj Jasani	Viraj Jasani
HADOOP-18763	Upgrade aws-java-sdk to 1.12.367+	Major	fs/s3	Steve Loughran	Viraj Jasani

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-16822	HostRestrictingAuthorizationFilter should pass through requests if they don't access WebHDFS API	Major	.	Takanobu Asanuma	Takanobu Asanuma
HDFS-16886	Fix documentation for StateStoreRecordOperations#get(Class ..., Query ...)	Trivial	.	Simbarashe Dzinamarira	Simbarashe Dzinamarira
HDFS-16902	Add Namenode status to BPServiceActor metrics and improve logging in offerservice	Major	.	Viraj Jasani	Viraj Jasani
HDFS-16901	RBF: Routers should propagate the real user in the UGI via the caller context	Major	.	Simbarashe Dzinamarira	Simbarashe Dzinamarira
HADOOP-18658	snakeyaml dependency: upgrade to v2.0	Major	.	PJ Fanning	PJ Fanning
HADOOP-18693	Upgrade Apache Derby from 10.10.2.0 to 10.14.2.0 due to CVEs	Major	build, test	PJ Fanning
HADOOP-18712	Upgrade to jetty 9.4.51 due to cve	Major	common	PJ Fanning	PJ Fanning
HADOOP-18727	Fix WriteOperations.listMultipartUploads function description	Trivial	fs/s3	Dongjoon Hyun	Dongjoon Hyun
HADOOP-18761	Remove mysql-connector-java	Blocker	.	Wei-Chiu Chuang	Wei-Chiu Chuang