Apache Hadoop Changelog

Release 2.6.4 - 2016-02-11

INCOMPATIBLE CHANGES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-3154	Should not upload partial logs for MR jobs or other "short-running' applications	Blocker	nodemanager, resourcemanager	Xuan Gong	Xuan Gong
HADOOP-11252	RPC client does not time out by default	Critical	ipc	Wilfred Spiegelenburg	Masatake Iwasaki

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-7694	FSDataInputStream should support "unbuffer"	Major	.	Colin P. McCabe	Colin P. McCabe
HDFS-8722	Optimize datanode writes for small writes and flushes	Critical	.	Kihwal Lee	Kihwal Lee
HDFS-8647	Abstract BlockManager's rack policy into BlockPlacementPolicy	Major	.	Ming Ma	Brahma Reddy Battula
HDFS-9314	Improve BlockPlacementPolicyDefault's picking of excess replicas	Major	.	Ming Ma	Xiao Chen
MAPREDUCE-6436	JobHistory cache issue	Blocker	.	Ryu Kobayashi	Kai Sasaki
HDFS-9415	Document dfs.cluster.administrators and dfs.permissions.superusergroup	Major	documentation	Arpit Agarwal	Xiaobing Zhou

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-2975	FSLeafQueue app lists are accessed without required locks	Blocker	.	Karthik Kambatla	Karthik Kambatla
HDFS-6945	BlockManager should remove a block from excessReplicateMap and decrement ExcessBlocks metric when the block is removed	Critical	namenode	Akira Ajisaka	Akira Ajisaka
HDFS-4660	Block corruption can happen during pipeline recovery	Blocker	datanode	Peng Zhang	Kihwal Lee
YARN-3842	NMProxy should retry on NMNotYetReadyException	Critical	.	Karthik Kambatla	Robert Kanter
YARN-3695	ServerProxy (NMProxy, etc.) shouldn't retry forever for non network exception.	Major	.	Junping Du	Raju Bairishetti
HADOOP-12107	long running apps may have a huge number of StatisticsData instances under FileSystem	Critical	fs	Sangjin Lee	Sangjin Lee
YARN-3849	Too much of preemption activity causing continuos killing of containers across queues	Critical	capacityscheduler	Sunil Govindan	Sunil Govindan
HDFS-8767	RawLocalFileSystem.listStatus() returns null for UNIX pipefile	Critical	.	Haohui Mai	Kanaka Kumar Avvaru
YARN-3535	Scheduler must re-request container resources when RMContainer transitions from ALLOCATED to KILLED	Critical	capacityscheduler, fairscheduler, resourcemanager	Peng Zhang	Peng Zhang
YARN-3857	Memory leak in ResourceManager with SIMPLE mode	Critical	resourcemanager	mujunchao	mujunchao
MAPREDUCE-5982	Task attempts that fail from the ASSIGNED state can disappear	Major	mr-am	Jason Lowe	Chang Li
YARN-3697	FairScheduler: ContinuousSchedulingThread can fail to shutdown	Critical	fairscheduler	zhihai xu	zhihai xu
MAPREDUCE-6492	AsyncDispatcher exit with NPE on TaskAttemptImpl#sendJHStartEventForAssignedFailTask	Critical	.	Bibin A Chundatt	Bibin A Chundatt
YARN-4180	AMLauncher does not retry on failures when talking to NM	Critical	resourcemanager	Anubhav Dhoot	Anubhav Dhoot
HDFS-9178	Slow datanode I/O can cause a wrong node to be marked bad	Critical	.	Kihwal Lee	Kihwal Lee
HDFS-9220	Reading small file (< 512 bytes) that is open for append fails due to incorrect checksum	Blocker	.	Bogdan Raducanu	Jing Zhao
HDFS-9313	Possible NullPointerException in BlockManager if no excess replica can be chosen	Major	.	Ming Ma	Ming Ma
YARN-4354	Public resource localization fails with NPE	Blocker	nodemanager	Jason Lowe	Jason Lowe
YARN-4380	TestResourceLocalizationService.testDownloadingResourcesOnContainerKill fails intermittently	Major	test	Tsuyoshi Ozawa	Varun Saxena
HDFS-9294	DFSClient deadlock when close file and failed to renew lease	Blocker	hdfs-client	DENG FEI	Brahma Reddy Battula
YARN-4452	NPE when submit Unmanaged application	Critical	.	Naganarasimha G R	Naganarasimha G R
HDFS-9445	Datanode may deadlock while handling a bad volume	Blocker	.	Kihwal Lee	Walter Su
MAPREDUCE-6577	MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set	Critical	mr-am	Sangjin Lee	Sangjin Lee
YARN-4546	ResourceManager crash due to scheduling opportunity overflow	Critical	resourcemanager	Jason Lowe	Jason Lowe
HDFS-9600	do not check replication if the block is under construction	Critical	.	Phil Yang	Phil Yang
HDFS-9574	Reduce client failures during datanode restart	Major	.	Kihwal Lee	Kihwal Lee
YARN-4414	Nodemanager connection errors are retried at multiple levels	Major	nodemanager	Jason Lowe	Chang Li
HADOOP-12706	TestLocalFsFCStatistics#testStatisticsThreadLocalDataCleanUp times out occasionally	Major	test	Jason Lowe	Sangjin Lee
YARN-4581	AHS writer thread leak makes RM crash while RM is recovering	Major	resourcemanager	sandflee	sandflee
MAPREDUCE-6554	MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory	Critical	.	Bibin A Chundatt	Bibin A Chundatt
YARN-4598	Invalid event: RESOURCE_FAILED at CONTAINER_CLEANEDUP_AFTER_KILL	Major	nodemanager	tangshangwen	tangshangwen
MAPREDUCE-6619	HADOOP_CLASSPATH is overwritten in MR container	Major	mrv2	shanyu zhao	Junping Du
MAPREDUCE-6618	YarnClientProtocolProvider leaking the YarnClient thread.	Major	.	Xuan Gong	Xuan Gong
MAPREDUCE-6621	Memory Leak in JobClient#submitJobInternal()	Major	.	Xuan Gong	Xuan Gong
MAPREDUCE-6363	[NNBench] Lease mismatch error when running with multiple mappers	Critical	benchmarks	Brahma Reddy Battula	Bibin A Chundatt

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-12736	TestTimedOutTestsListener#testThreadDumpAndDeadlocks sometimes times out	Major	.	Xiao Chen	Xiao Chen

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-3893	Both RM in active state when Admin#transitionToActive failure from refeshAll()	Critical	resourcemanager	Bibin A Chundatt	Bibin A Chundatt
YARN-2902	Killing a container that is localizing can orphan resources in the DOWNLOADING state	Major	nodemanager	Jason Lowe	Varun Saxena