Apache Hadoop Changelog

Release 2.6.5 - 2016-10-08

IMPORTANT ISSUES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-4785	inconsistent value type of the "type" field for LeafQueueInfo in response of RM REST API - cluster/scheduler	Major	webapp	Jayesh	Varun Vasudev

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-2580	NameNode#main(...) can make use of GenericOptionsParser.	Minor	namenode	Harsh J	Harsh J
HADOOP-11301	[optionally] update jmx cache to drop old metrics	Major	.	Maysam Yabandeh	Maysam Yabandeh
HDFS-9669	TcpPeerServer should respect ipc.server.listen.queue.size	Major	.	Elliott Clark	Elliott Clark
HDFS-9629	Update the footer of Web UI to show year 2016	Major	.	Xiao Chen	Xiao Chen
HADOOP-12805	Annotate CanUnbuffer with @InterfaceAudience.Public	Major	.	Ted Yu	Ted Yu
YARN-4690	Skip object allocation in FSAppAttempt#getResourceUsage when possible	Major	.	Ming Ma	Ming Ma
HADOOP-12800	Copy docker directory from 2.8 to 2.7/2.6 repos to enable pre-commit Jenkins runs	Major	build, yetus	Zhe Zhang	Zhe Zhang
MAPREDUCE-6622	Add capability to set JHS job cache to a task-based limit	Critical	jobhistoryserver	Ray Chiang	Ray Chiang
HADOOP-12789	log classpath of ApplicationClassLoader at INFO level	Minor	util	Sangjin Lee	Sangjin Lee
HDFS-10264	Logging improvements in FSImageFormatProtobuf.Saver	Major	namenode	Konstantin Shvachko	Xiaobing Zhou
HDFS-10377	CacheReplicationMonitor shutdown log message should use INFO level.	Major	logging, namenode	Konstantin Shvachko	Yiqun Lin
HADOOP-13290	Appropriate use of generics in FairCallQueue	Major	ipc	Konstantin Shvachko	Jonathan Hung
HADOOP-13298	Fix the leftover L&N files in hadoop-build-tools/src/main/resources/META-INF/	Minor	.	Xiao Chen	Tsuyoshi Ozawa
YARN-5483	Optimize RMAppAttempt#pullJustFinishedContainers	Major	.	sandflee	sandflee

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-7258	CacheReplicationMonitor rescan schedule log should use DEBUG level instead of INFO level	Minor	namenode	Xiaoyu Yao	Xiaoyu Yao
HDFS-7530	Allow renaming of encryption zone roots	Minor	namenode	Charles Lamb	Charles Lamb
HADOOP-7817	RawLocalFileSystem.append() should give FSDataOutputStream with accurate .getPos()	Major	fs	Kristofer Tomasette	Kanaka Kumar Avvaru
MAPREDUCE-6413	TestLocalJobSubmission is failing with unknown host	Major	test	Jason Lowe	zhihai xu
MAPREDUCE-5817	Mappers get rescheduled on node transition even after all reducers are completed	Major	applicationmaster	Sangjin Lee	Sangjin Lee
HDFS-8845	DiskChecker should not traverse the entire tree	Major	.	Chang Li	Chang Li
HDFS-8581	ContentSummary on / skips further counts on yielding lock	Minor	namenode	tongshiquan	J.Andreina
HADOOP-12348	MetricsSystemImpl creates MetricsSourceAdapter with wrong time unit parameter.	Major	metrics	zhihai xu	zhihai xu
MAPREDUCE-6302	Preempt reducers after a configurable timeout irrespective of headroom	Critical	.	mai shurong	Karthik Kambatla
HADOOP-12482	Race condition in JMX cache update	Major	.	Tony Wu	Tony Wu
HDFS-9347	Invariant assumption in TestQuorumJournalManager.shutdown() is wrong	Major	test	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-12559	KMS connection failures should trigger TGT renewal	Major	security	Zhe Zhang	Zhe Zhang
HADOOP-12682	Fix TestKMS#testKMSRestart* failure	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-12773	HBase classes fail to load with client/job classloader enabled	Major	util	Sangjin Lee	Sangjin Lee
HDFS-9752	Permanent write failures may happen to slow writers during datanode rolling upgrades	Critical	.	Kihwal Lee	Walter Su
HADOOP-12589	Fix intermittent test failure of TestCopyPreserveFlag	Major	test	Tsuyoshi Ozawa	Masatake Iwasaki
HADOOP-12810	FileSystem#listLocatedStatus causes unnecessary RPC calls	Major	fs, fs/s3	Ryan Blue	Ryan Blue
MAPREDUCE-6637	Testcase Failure : TestFileInputFormat.testSplitLocationInfo	Major	test	Brahma Reddy Battula	Brahma Reddy Battula
MAPREDUCE-6635	Unsafe long to int conversion in UncompressedSplitLineReader and IndexOutOfBoundsException	Critical	.	Sergey Shelukhin	Junping Du
YARN-2046	Out of band heartbeats are sent only on container kill and possibly too early	Major	nodemanager	Jason Lowe	Ming Ma
YARN-4722	AsyncDispatcher logs redundant event queue sizes	Major	.	Jason Lowe	Jason Lowe
YARN-4761	NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations on fair scheduler	Major	fairscheduler	Sangjin Lee	Sangjin Lee
MAPREDUCE-6656	[NNBench] OP_DELETE operation isn't working after MAPREDUCE-6363	Blocker	.	J.Andreina	J.Andreina
HADOOP-12958	PhantomReference for filesystem statistics can trigger OOM	Major	.	Jason Lowe	Sangjin Lee
HDFS-10182	Hedged read might overwrite user's buf	Major	.	zhouyingchao	zhouyingchao
YARN-4773	Log aggregation performs extraneous filesystem operations when rolling log aggregation is disabled	Minor	nodemanager	Jason Lowe	Jun Gong
HDFS-10178	Permanent write failures can happen if pipeline recoveries occur for the first packet	Critical	.	Kihwal Lee	Kihwal Lee
HDFS-10271	Extra bytes are getting released from reservedSpace for append	Critical	.	Brahma Reddy Battula	Brahma Reddy Battula
YARN-4794	Deadlock in NMClientImpl	Critical	.	Sumana Sathish	Jian He
HADOOP-13042	Restore lost leveldbjni LICENSE and NOTICE changes	Major	.	Andrew Wang	Andrew Wang
HADOOP-13043	Add LICENSE.txt entries for bundled javascript dependencies	Major	.	Andrew Wang	Andrew Wang
HADOOP-13052	ChecksumFileSystem mishandles crc file permissions	Major	fs	Daryn Sharp	Daryn Sharp
YARN-5009	NMLeveldbStateStoreService database can grow substantially leading to longer recovery times	Major	nodemanager	Jason Lowe	Jason Lowe
MAPREDUCE-6514	Job hangs as ask is not updated after ramping down of all reducers	Blocker	applicationmaster	Varun Saxena	Varun Saxena
MAPREDUCE-6689	MapReduce job can infinitely increase number of reducer resource requests	Blocker	.	Wangda Tan	Wangda Tan
MAPREDUCE-6558	multibyte delimiters with compressed input files generate duplicate records	Major	mrv1, mrv2	Wilfred Spiegelenburg	Wilfred Spiegelenburg
HDFS-9365	Balancer does not work with the HDFS-6376 HA setup	Major	balancer & mover	Tsz Wo Nicholas Sze	Tsz Wo Nicholas Sze
YARN-4459	container-executor should only kill process groups	Major	nodemanager	Jun Gong	Jun Gong
HDFS-10458	getFileEncryptionInfo should return quickly for non-encrypted cluster	Major	encryption, namenode	Zhe Zhang	Zhe Zhang
YARN-5206	RegistrySecurity includes id:pass in exception text if considered invalid	Minor	client, security	Steve Loughran	Steve Loughran
HADOOP-13189	FairCallQueue makes callQueue larger than the configured capacity.	Major	ipc	Konstantin Shvachko	Vinitha Reddy Gankidi
HADOOP-13255	KMSClientProvider should check and renew tgt when doing delegation token operations.	Major	kms	Xiao Chen	Xiao Chen
HADOOP-13192	org.apache.hadoop.util.LineReader cannot handle multibyte delimiters correctly	Critical	util	binde	binde
YARN-5197	RM leaks containers if running container disappears from node update	Critical	resourcemanager	Jason Lowe	Jason Lowe
YARN-5262	Optimize sending RMNodeFinishedContainersPulledByAMEvent for every AM heartbeat	Major	resourcemanager	Rohith Sharma K S	Rohith Sharma K S
HADOOP-13350	Additional fix to LICENSE and NOTICE	Blocker	build	Xiao Chen	Xiao Chen
HADOOP-12893	Verify LICENSE.txt and NOTICE.txt	Blocker	build	Allen Wittenauer	Xiao Chen
HADOOP-13297	Add missing dependency in setting maven-remote-resource-plugin to fix builds	Major	build	Akira Ajisaka	Sean Busbey
YARN-5353	ResourceManager can leak delegation tokens when they are shared across apps	Critical	resourcemanager	Jason Lowe	Jason Lowe
HADOOP-11361	Fix a race condition in MetricsSourceAdapter.updateJmxCache	Major	.	Brahma Reddy Battula	Brahma Reddy Battula
HDFS-10544	Balancer doesn't work with IPFailoverProxyProvider	Major	balancer & mover, ha	Zhe Zhang	Zhe Zhang
YARN-5462	TestNodeStatusUpdater.testNodeStatusUpdaterRetryAndNMShutdown fails intermittently	Major	.	Eric Badger	Eric Badger
HADOOP-13434	Add quoting to Shell class	Major	.	Owen O'Malley	Owen O'Malley
HDFS-9696	Garbage snapshot records lingering forever	Critical	.	Kihwal Lee	Kihwal Lee
HADOOP-13494	ReconfigurableBase can log sensitive information	Major	security	Sean Mackrory	Sean Mackrory
HDFS-9530	ReservedSpace is not cleared for abandoned Blocks	Critical	datanode	Fei Hui	Brahma Reddy Battula
HDFS-10763	Open files can leak permanently due to inconsistent lease update	Critical	.	Kihwal Lee	Kihwal Lee
HADOOP-13579	Fix source-level compatibility after HADOOP-11252	Blocker	.	Akira Ajisaka	Tsuyoshi Ozawa
HDFS-10870	Wrong dfs.namenode.acls.enabled default in HdfsPermissionsGuide.apt.vm	Trivial	documentation	John Zhuge	John Zhuge
YARN-5694	ZKRMStateStore can prevent the transition to standby in branch-2.7 if the ZK node is unreachable	Critical	resourcemanager	Daniel Templeton	Daniel Templeton

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
MAPREDUCE-6191	TestJavaSerialization fails with getting incorrect MR job result	Minor	test	sam liu	sam liu
HDFS-9688	Test the effect of nested encryption zones in HDFS downgrade	Major	encryption, test	Zhe Zhang	Zhe Zhang

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-4393	TestResourceLocalizationService#testFailedDirsResourceRelease fails intermittently	Major	test	Varun Saxena	Varun Saxena
YARN-4573	TestRMAppTransitions.testAppRunningKill and testAppKilledKilled fail on trunk	Major	resourcemanager, test	Takashi Ohnishi	Takashi Ohnishi
HDFS-10653	Optimize conversion from path string to components	Major	hdfs	Daryn Sharp	Daryn Sharp