Apache Hadoop Changelog

Release 3.0.4 - Unreleased (as of 2018-09-02)

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-13448	HDFS Block Placement - Ignore Locality for First Block Replica	Minor	block placement, hdfs-client	BELUGA BEHR	BELUGA BEHR

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-15252	Checkstyle version is not compatible with IDEA's checkstyle plugin	Major	.	Andras Bokor	Andras Bokor
HDFS-13602	Add checkOperation(WRITE) checks in FSNamesystem	Major	ha, namenode	Erik Krogen	Chao Sun
HDFS-13155	BlockPlacementPolicyDefault.chooseTargetInOrder Not Checking Return Value for NULL	Minor	namenode	BELUGA BEHR	Zsolt Venczel
HDFS-13659	Add more test coverage for contentSummary for snapshottable path	Major	namenode, test	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-15499	Performance severe drop when running RawErasureCoderBenchmark with NativeRSRawErasureCoder	Major	.	Sammi Chen	Sammi Chen
HDFS-13653	Make dfs.client.failover.random.order a per nameservice configuration	Major	federation	Ekanth Sethuramalingam	Ekanth Sethuramalingam
YARN-8394	Improve data locality documentation for Capacity Scheduler	Major	.	Weiwei Yang	Weiwei Yang
HDFS-13641	Add metrics for edit log tailing	Major	metrics	Chao Sun	Chao Sun
HDFS-13686	Add overall metrics for FSNamesystemLock	Major	hdfs, namenode	Lukas Majercak	Lukas Majercak
HDFS-13692	StorageInfoDefragmenter floods log when compacting StorageInfo TreeSet	Minor	.	Yiqun Lin	Bharat Viswanadham
HDFS-13703	Avoid allocation of CorruptedBlocks hashmap when no corrupted blocks are hit	Major	performance	Todd Lipcon	Todd Lipcon
HADOOP-15554	Improve JIT performance for Configuration parsing	Minor	conf, performance	Todd Lipcon	Todd Lipcon
HDFS-13714	Fix TestNameNodePrunesMissingStorages test failures on Windows	Major	hdfs, namenode, test	Lukas Majercak	Lukas Majercak
HDFS-13712	BlockReaderRemote.read() logging improvement	Minor	hdfs-client	Gergo Repas	Gergo Repas
HDFS-13719	Docs around dfs.image.transfer.timeout are misleading	Major	.	Kitti Nanasi	Kitti Nanasi
HADOOP-15598	DataChecksum calculate checksum is contented on hashtable synchronization	Major	common	Prasanth Jayachandran	Prasanth Jayachandran
HADOOP-15609	Retry KMS calls when SSLHandshakeException occurs	Major	common, kms	Kitti Nanasi	Kitti Nanasi
HADOOP-15612	Improve exception when tfile fails to load LzoCodec	Major	.	Gera Shegalov	Gera Shegalov
HDFS-11060	make DEFAULT_MAX_CORRUPT_FILEBLOCKS_RETURNED configurable	Minor	hdfs	Lantao Jin	Lantao Jin
HDFS-13727	Log full stack trace if DiskBalancer exits with an unhandled exception	Minor	diskbalancer	Stephen O'Donnell	Gabor Bota
YARN-8155	Improve ATSv2 client logging in RM and NM publisher	Major	.	Rohith Sharma K S	Abhishek Modi
HDFS-13728	Disk Balancer should not fail if volume usage is greater than capacity	Minor	diskbalancer	Stephen O'Donnell	Stephen O'Donnell
YARN-8568	Replace the deprecated zk-address property in the HA config example in ResourceManagerHA.md	Minor	yarn	Antal Bálint Steinbach	Antal Bálint Steinbach
HDFS-13814	Remove super user privilege requirement for NameNode.getServiceStatus	Minor	namenode	Chao Sun	Chao Sun
YARN-8559	Expose mutable-conf scheduler's configuration in RM /scheduler-conf endpoint	Major	resourcemanager	Anna Savarin	Weiwei Yang
HDFS-13813	Exit NameNode if dangling child inode is detected when saving FsImage	Major	hdfs, namenode	Siyao Meng	Siyao Meng
HDFS-13821	RBF: Add dfs.federation.router.mount-table.cache.enable so that users can disable cache	Major	hdfs	Fei Hui	Fei Hui
HDFS-13831	Make block increment deletion number configurable	Major	.	Yiqun Lin	Ryan Wu
YARN-8051	TestRMEmbeddedElector#testCallbackSynchronization is flakey	Major	test	Robert Kanter	Robert Kanter

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-12857	StoragePolicyAdmin should support schema based path	Major	namenode	Surendra Singh Lilhore	Surendra Singh Lilhore
YARN-7835	[Atsv2] Race condition in NM while publishing events if second attempt is launched on the same node	Critical	.	Rohith Sharma K S	Rohith Sharma K S
YARN-7773	YARN Federation used Mysql as state store throw exception, Unknown column 'homeSubCluster' in 'field list'	Blocker	federation	Yiran Wu	Yiran Wu
HDFS-13636	Cross-Site Scripting vulnerability in HttpServer2	Major	.	Haibo Yan	Haibo Yan
HDFS-13339	Volume reference can't be released and may lead to deadlock when DataXceiver does a check volume	Critical	datanode	liaoyuxiangqin	Zsolt Venczel
YARN-8382	cgroup file leak in NM	Major	nodemanager	Hu Ziqian	Hu Ziqian
HDFS-13545	"guarded" is misspelled as "gaurded" in FSPermissionChecker.java	Trivial	documentation	Jianchao Jia	Jianchao Jia
MAPREDUCE-7103	Fix TestHistoryViewerPrinter on windows due to a mismatch line separator	Minor	.	Giovanni Matteo Fumarola	Giovanni Matteo Fumarola
HADOOP-15217	FsUrlConnection does not handle paths with spaces	Major	fs	Joseph Fourny	Zsolt Venczel
HDFS-12950	[oiv] ls will fail in secure cluster	Major	.	Brahma Reddy Battula	Wei-Chiu Chuang
YARN-8359	Exclude containermanager.linux test classes on Windows	Major	.	Giovanni Matteo Fumarola	Jason Lowe
HDFS-13664	Refactor ConfiguredFailoverProxyProvider to make inheritance easier	Minor	hdfs-client	Chao Sun	Chao Sun
HDFS-12670	can't renew HDFS tokens with only the hdfs client jar	Critical	.	Thomas Graves	Arpit Agarwal
HDFS-13667	Typo: Marking all "datandoes" as stale	Trivial	namenode	Wei-Chiu Chuang	Nanda kumar
YARN-8405	RM zk-state-store.parent-path ACLs has been changed since HADOOP-14773	Major	.	Rohith Sharma K S	Íñigo Goiri
YARN-8404	Timeline event publish need to be async to avoid Dispatcher thread leak in case ATS is down	Blocker	.	Rohith Sharma K S	Rohith Sharma K S
HDFS-13673	TestNameNodeMetrics fails on Windows	Minor	test	Zuoming Zhang	Zuoming Zhang
HDFS-13676	TestEditLogRace fails on Windows	Minor	test	Zuoming Zhang	Zuoming Zhang
HDFS-13174	hdfs mover -p /path times out after 20 min	Major	balancer & mover	Istvan Fajth	Istvan Fajth
HADOOP-15523	Shell command timeout given is in seconds whereas it is taken as millisec while scheduling	Major	.	Bilwa S T	Bilwa S T
HDFS-13682	Cannot create encryption zone after KMS auth token expires	Critical	encryption, kms, namenode	Xiao Chen	Xiao Chen
YARN-8444	NodeResourceMonitor crashes on bad swapFree value	Major	.	Jim Brennan	Jim Brennan
YARN-8443	Total #VCores in cluster metrics is wrong when CapacityScheduler reserved some containers	Major	webapp	Tao Yang	Tao Yang
YARN-8457	Compilation is broken with -Pyarn-ui	Major	webapp	Sunil Govindan	Sunil Govindan
YARN-8401	[UI2] new ui is not accessible with out internet connection	Blocker	.	Bibin A Chundatt	Bibin A Chundatt
YARN-8451	Multiple NM heartbeat thread created when a slow NM resync with RM	Major	nodemanager	Botong Huang	Botong Huang
HADOOP-15548	Randomize local dirs	Minor	.	Jim Brennan	Jim Brennan
HDFS-13702	Remove HTrace hooks from DFSClient to reduce CPU usage	Major	performance	Todd Lipcon	Todd Lipcon
HDFS-13635	Incorrect message when block is not found	Major	datanode	Wei-Chiu Chuang	Gabor Bota
HADOOP-15571	Multiple FileContexts created with the same configuration object should be allowed to have different umask	Critical	.	Vinod Kumar Vavilapalli	Vinod Kumar Vavilapalli
HDFS-13121	NPE when request file descriptors when SC read	Minor	hdfs-client	Gang Xie	Zsolt Venczel
YARN-6265	yarn.resourcemanager.fail-fast is used inconsistently	Major	resourcemanager	Daniel Templeton	Yuanbo Liu
YARN-8473	Containers being launched as app tears down can leave containers in NEW state	Major	nodemanager	Jason Lowe	Jason Lowe
HDFS-13723	Occasional "Should be different group" error in TestRefreshUserMappings#testGroupMappingRefresh	Major	security, test	Siyao Meng	Siyao Meng
HDFS-12837	Intermittent failure in TestReencryptionWithKMS	Major	encryption, test	Surendra Singh Lilhore	Xiao Chen
HDFS-13729	Fix broken links to RBF documentation	Minor	documentation	jwhitter	Gabor Bota
YARN-8515	container-executor can crash with SIGPIPE after nodemanager restart	Major	.	Jim Brennan	Jim Brennan
YARN-8421	when moving app, activeUsers is increased, even though app does not have outstanding request	Major	.	kyungwan nam
HDFS-13524	Occasional "All datanodes are bad" error in TestLargeBlock#testLargeBlockSize	Major	.	Wei-Chiu Chuang	Siyao Meng
HADOOP-15610	Hadoop Docker Image Pip Install Fails	Critical	.	Jack Bearden	Jack Bearden
HADOOP-15614	TestGroupsCaching.testExceptionOnBackgroundRefreshHandled reliably fails	Major	.	Kihwal Lee	Weiwei Yang
YARN-8548	AllocationRespose proto setNMToken initBuilder not done	Major	.	Bibin A Chundatt	Bilwa S T
YARN-7748	TestContainerResizing.testIncreaseContainerUnreservedWhenApplicationCompleted fails due to multiple container fail events	Major	capacityscheduler	Haibo Chen	Weiwei Yang
YARN-8577	Fix the broken anchor in SLS site-doc	Minor	documentation	Weiwei Yang	Weiwei Yang
YARN-4606	CapacityScheduler: applications could get starved because computation of #activeUsers considers pending apps	Critical	capacity scheduler, capacityscheduler	Karam Singh	Manikandan R
HDFS-13765	Fix javadoc for FSDirMkdirOp#createParentDirectories	Minor	documentation	Lokesh Jain	Lokesh Jain
YARN-8434	Update federation documentation of Nodemanager configurations	Minor	.	Bibin A Chundatt	Bibin A Chundatt
YARN-8558	NM recovery level db not cleaned up properly on container finish	Critical	.	Bibin A Chundatt	Bibin A Chundatt
HADOOP-15637	LocalFs#listLocatedStatus does not filter out hidden .crc files	Minor	fs	Erik Krogen	Erik Krogen
YARN-8397	Potential thread leak in ActivitiesManager	Major	.	Rohith Sharma K S	Rohith Sharma K S
YARN-6966	NodeManager metrics may return wrong negative values when NM restart	Major	.	Yang Wang	Szilard Nemeth
HDFS-13786	EC: Display erasure coding policy for sub-directories is not working	Major	erasure-coding	Souryakanta Dwivedy	Ayush Saxena
YARN-8331	Race condition in NM container launched after done	Major	.	Yang Wang	Pradeep Ambati
HADOOP-15638	KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x	Major	kms	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-13738	fsck -list-corruptfileblocks has infinite loop if user is not privileged.	Major	tools	Wei-Chiu Chuang	Yuen-Kuei Hsueh
HDFS-13758	DatanodeManager should throw exception if it has BlockRecoveryCommand but the block is not under construction	Major	namenode	Wei-Chiu Chuang	chencan
YARN-8614	Fix few annotation typos in YarnConfiguration	Trivial	.	Sen Zhao	Sen Zhao
HDFS-13746	Still occasional "Should be different group" failure in TestRefreshUserMappings#testGroupMappingRefresh	Major	.	Siyao Meng	Siyao Meng
HDFS-10240	Race between close/recoverLease leads to missing block	Major	.	zhouyingchao	Jinglun
YARN-8612	Fix NM Collector Service Port issue in YarnConfiguration	Major	ATSv2	Prabha Manepalli	Prabha Manepalli
HDFS-13747	Statistic for list_located_status is incremented incorrectly by listStatusIterator	Minor	hdfs-client	Todd Lipcon	Antal Mihalyi
HADOOP-15674	Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite	Major	common	Gabor Bota	Szilard Nemeth
YARN-8640	Restore previous state in container-executor after failure	Major	.	Jim Brennan	Jim Brennan
YARN-8679	[ATSv2] If HBase cluster is down for long time, high chances that NM ContainerManager dispatcher get blocked	Major	.	Rohith Sharma K S	Wangda Tan
HADOOP-14314	The OpenSolaris taxonomy link is dead in InterfaceClassification.md	Major	documentation	Daniel Templeton	Rui Gao
YARN-8649	NPE in localizer hearbeat processing if a container is killed while localizing	Major	.	lujie	lujie
YARN-8719	Typo correction for yarn configuration in OpportunisticContainers(federation) docs	Major	documentation, federation	Y. SREENIVASULU REDDY	Y. SREENIVASULU REDDY
HDFS-13731	ReencryptionUpdater fails with ConcurrentModificationException during processCheckpoints	Major	encryption	Xiao Chen	Zsolt Venczel
HADOOP-15705	Typo in the definition of "stable" in the interface classification	Minor	.	Daniel Templeton	Daniel Templeton
HDFS-13863	FsDatasetImpl should log DiskOutOfSpaceException	Major	hdfs	Fei Hui	Fei Hui
HADOOP-15698	KMS log4j is not initialized properly at startup	Major	kms	Kitti Nanasi	Kitti Nanasi
HADOOP-15706	Typo in compatibility doc: SHOUD -> SHOULD	Trivial	.	Daniel Templeton	Laszlo Kollar
HDFS-13027	Handle possible NPEs due to deleted blocks in race condition	Major	namenode	Vinayakumar B	Vinayakumar B

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-13632	Randomize baseDir for MiniJournalCluster in MiniQJMHACluster for TestDFSAdminWithHA	Minor	.	Anbang Hu	Anbang Hu
HDFS-13651	TestReencryptionHandler fails on Windows	Minor	.	Anbang Hu	Anbang Hu
MAPREDUCE-7102	Fix TestJavaSerialization for Windows due a mismatch line separator	Minor	.	Giovanni Matteo Fumarola	Giovanni Matteo Fumarola
MAPREDUCE-7105	Fix TestNativeCollectorOnlyHandler.testOnCall on Windows because of the path format	Minor	.	Giovanni Matteo Fumarola	Giovanni Matteo Fumarola
HDFS-13652	Randomize baseDir for MiniDFSCluster in TestBlockScanner	Minor	.	Anbang Hu	Anbang Hu
HDFS-13649	Randomize baseDir for MiniDFSCluster in TestReconstructStripedFile and TestReconstructStripedFileWithRandomECPolicy	Minor	.	Anbang Hu	Anbang Hu
HDFS-13650	Randomize baseDir for MiniDFSCluster in TestDFSStripedInputStream and TestDFSStripedInputStreamWithRandomECPolicy	Minor	.	Anbang Hu	Anbang Hu
YARN-8370	Some Node Manager tests fail on Windows due to improper path/file separator	Minor	.	Anbang Hu	Anbang Hu
YARN-8422	TestAMSimulator failing with NPE	Minor	.	Giovanni Matteo Fumarola	Giovanni Matteo Fumarola
HADOOP-15532	TestBasicDiskValidator fails with NoSuchFileException	Minor	.	Íñigo Goiri	Giovanni Matteo Fumarola
HDFS-13563	TestDFSAdminWithHA times out on Windows	Minor	.	Anbang Hu	Lukas Majercak
HDFS-13681	Fix TestStartup.testNNFailToStartOnReadOnlyNNDir test failure on Windows	Major	test	Xiao Liang	Xiao Liang

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-12978	Fine-grained locking while consuming journal stream.	Major	namenode	Konstantin Shvachko	Konstantin Shvachko
HDFS-13637	RBF: Router fails when threadIndex (in ConnectionPool) wraps around Integer.MIN_VALUE	Critical	federation	CR Hota	CR Hota
HDFS-13281	Namenode#createFile should be /.reserved/raw/ aware.	Critical	encryption	Rushabh S Shah	Rushabh S Shah
YARN-4677	RMNodeResourceUpdateEvent update from scheduler can lead to race condition	Major	graceful, resourcemanager, scheduler	Brook Zhou	Wilfred Spiegelenburg
HADOOP-15506	Upgrade Azure Storage Sdk version to 7.0.0 and update corresponding code blocks	Minor	fs/azure	Esfandiar Manii	Esfandiar Manii
HADOOP-15529	ContainerLaunch#testInvalidEnvVariableSubstitutionType is not supported in Windows	Minor	.	Giovanni Matteo Fumarola	Giovanni Matteo Fumarola
HADOOP-15533	Make WASB listStatus messages consistent	Trivial	fs/azure	Esfandiar Manii	Esfandiar Manii
HADOOP-15458	TestLocalFileSystem#testFSOutputStreamBuilder fails on Windows	Minor	test	Xiao Liang	Xiao Liang
HDFS-13726	RBF: Fix RBF configuration links	Minor	documentation	Takanobu Asanuma	Takanobu Asanuma
HDFS-13475	RBF: Admin cannot enforce Router enter SafeMode	Major	.	Wei Yan	Chao Sun
HDFS-13733	RBF: Add Web UI configurations and descriptions to RBF document	Minor	documentation	Takanobu Asanuma	Takanobu Asanuma
HDFS-13743	RBF: Router throws NullPointerException due to the invalid initialization of MountTableResolver	Major	.	Takanobu Asanuma	Takanobu Asanuma
HDFS-13750	RBF: Router ID in RouterRpcClient is always null	Major	.	Takanobu Asanuma	Takanobu Asanuma
YARN-8129	Improve error message for invalid value in fields attribute	Minor	ATSv2	Charan Hebri	Abhishek Modi
HDFS-13848	Refactor NameNode failover proxy providers	Major	ha, hdfs-client	Konstantin Shvachko	Konstantin Shvachko

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-13788	Update EC documentation about rack fault tolerance	Major	documentation, erasure-coding	Xiao Chen	Kitti Nanasi