Apache Hadoop Changelog

Release 1.3.0 - Unreleased (as of 2018-09-02)

INCOMPATIBLE CHANGES:

JIRA	Summary	Priority	Component	Reporter	Contributor
MAPREDUCE-5777	Support utf-8 text with BOM (byte order marker)	Major	.	bc Wong	zhihai xu

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
MAPREDUCE-2351	mapred.job.tracker.history.completed.location should support an arbitrary filesystem URI	Major	.	Tom White	Tom White
HADOOP-9450	HADOOP_USER_CLASSPATH_FIRST is not honored; CLASSPATH is PREpended instead of APpended	Major	scripts	Mitch Wyle	Harsh J
HDFS-4521	invalid network topologies should not be cached	Minor	.	Colin P. McCabe	Colin P. McCabe
HADOOP-8873	Port HADOOP-8175 (Add mkdir -p flag) to branch-1	Major	.	Eli Collins	Akira Ajisaka
HDFS-4903	Print trash configuration and trash emptier state in namenode log	Minor	namenode	Suresh Srinivas	Arpit Agarwal
MAPREDUCE-5406	Improve logging around Task Tracker exiting with JVM manager inconsistent state	Major	tasktracker	Chelsey Chang	Chelsey Chang
MAPREDUCE-5408	CLONE - The logging level of the tasks should be configurable by the job	Major	.	Owen O'Malley	Arun C Murthy
MAPREDUCE-5367	Local jobs all use same local working directory	Major	.	Sandy Ryza	Sandy Ryza
HDFS-4963	Improve multihoming support in namenode	Major	namenode	Arpit Agarwal	Arpit Agarwal
HADOOP-9855	Backport HADOOP-6578 to branch-1	Major	.	James Kinley	James Kinley
HDFS-5038	Backport several branch-2 APIs to branch-1	Minor	.	Jing Zhao	Jing Zhao
HDFS-5367	Restoring namenode storage locks namenode due to unnecessary fsimage write	Major	.	yunjiong zhao	yunjiong zhao
MAPREDUCE-5457	Add a KeyOnlyTextOutputReader to enable streaming to write out text files without separators	Major	.	Sandy Ryza	Sandy Ryza
MAPREDUCE-5609	Add debug log message when sending job end notification	Major	.	Robert Kanter	Robert Kanter
MAPREDUCE-3310	Custom grouping comparator cannot be set for Combiners	Major	client	Mathias Herberts	Alejandro Abdelnur
MAPREDUCE-5651	Backport Fair Scheduler queue placement policies to branch-1	Major	scheduler	Sandy Ryza	Ted Malaska
MAPREDUCE-5712	Backport Fair Scheduler pool placement by secondary group	Major	scheduler	Theodore michael Malaska	Theodore michael Malaska
HADOOP-10614	CBZip2InputStream is not threadsafe	Major	.	Xiangrui Meng	Xiangrui Meng
MAPREDUCE-6088	TestTokenCache tests should use their own JobConf instances	Major	mrv1, test	zhihai xu	zhihai xu
HDFS-7312	Update DistCp v1 to optionally not use tmp location (branch-1 only)	Minor	tools	Joseph Prosser	Joseph Prosser

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-7140	IPC Reader threads do not stop when server stops	Critical	.	Todd Lipcon	Todd Lipcon
HDFS-2264	NamenodeProtocol has the wrong value for clientPrincipal in KerberosInfo annotation	Major	namenode	Aaron T. Myers	Aaron T. Myers
MAPREDUCE-5046	backport MAPREDUCE-1423 to mapred.lib.CombineFileInputFormat	Major	client	Sangjin Lee
MAPREDUCE-5047	keep.failed.task.files=true causes job failure on secure clusters	Major	task, tasktracker	Sandy Ryza	Sandy Ryza
MAPREDUCE-5070	TestClusterStatus.testClusterMetrics fails on JDK7	Major	test	Sandy Ryza	Sandy Ryza
MAPREDUCE-5072	TestDelegationTokenRenewal.testDTRenewal fails in MR1 on jdk7	Major	test	Sandy Ryza	Sandy Ryza
MAPREDUCE-5073	TestJobStatusPersistency.testPersistency fails on JDK7	Major	test	Sandy Ryza	Sandy Ryza
HDFS-4622	Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1	Trivial	.	Jing Zhao	Jing Zhao
MAPREDUCE-2817	MiniRMCluster hardcodes 'mapred.local.dir' configuration to 'build/test/mapred/local'	Minor	test	Alejandro Abdelnur	Robert Kanter
MAPREDUCE-5133	TestSubmitJob.testSecureJobExecution is flaky due to job dir deletion race	Major	test	Sandy Ryza	Sandy Ryza
MAPREDUCE-5218	Annotate (comment) internal classes as Private	Minor	.	Karthik Kambatla	Karthik Kambatla
MAPREDUCE-5217	DistCp fails when launched by Oozie in a secure cluster	Major	distcp, security	Venkat Ranganathan	Venkat Ranganathan
MAPREDUCE-5095	TestShuffleExceptionCount#testCheckException fails occasionally with JDK7	Major	.	Arpit Agarwal	Arpit Agarwal
MAPREDUCE-5250	Searching for ';' in JobTracker History throws ArrayOutOfBoundException	Minor	.	Karthik Kambatla	Karthik Kambatla
HDFS-4944	WebHDFS cannot create a file path containing characters that must be URI-encoded, such as space.	Major	webhdfs	Chris Nauroth	Chris Nauroth
HADOOP-9678	TestRPC#testStopsAllThreads intermittently fails on Windows	Major	.	Ivan Mitic	Ivan Mitic
HADOOP-9307	BufferedFSInputStream.read returns wrong results after certain seeks	Major	fs	Todd Lipcon	Todd Lipcon
HDFS-5003	TestNNThroughputBenchmark failed caused by existing directories	Minor	test	Xi Fang	Xi Fang
MAPREDUCE-5405	Job recovery can fail if task log directory symlink from prior run still exists	Major	mrv1	Chris Nauroth	Chris Nauroth
MAPREDUCE-5288	ResourceEstimator#getEstimatedTotalMapOutputSize suffers from divide by zero issues	Major	mrv1	Harsh J	Karthik Kambatla
HADOOP-9507	LocalFileSystem rename() is broken in some cases when destination exists	Minor	fs	Mostafa Elhemali	Chris Nauroth
MAPREDUCE-4366	mapred metrics shows negative count of waiting maps and reduces	Major	jobtracker	Thomas Graves	Sandy Ryza
HADOOP-9768	chown and chgrp reject users and groups with spaces on platforms where spaces are otherwise acceptable	Major	fs	Chris Nauroth	Chris Nauroth
HADOOP-9801	Configuration#writeXml uses platform defaulting encoding, which may mishandle multi-byte characters.	Major	conf	Chris Nauroth	Chris Nauroth
HDFS-4898	BlockPlacementPolicyWithNodeGroup.chooseRemoteRack() fails to properly fallback to local rack	Minor	namenode	Eric Sirianni	Tsz Wo Nicholas Sze
HDFS-5211	Race condition between DistributedFileSystem#close and FileSystem#close can cause return of a closed DistributedFileSystem instance from the FileSystem cache.	Major	hdfs-client	Chris Nauroth	Chris Nauroth
MAPREDUCE-5508	JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob	Critical	jobtracker	Xi Fang	Xi Fang
HDFS-5245	shouldRetry() in WebHDFSFileSystem generates excessive warnings	Minor	webhdfs	Haohui Mai	Haohui Mai
HADOOP-10009	Backport HADOOP-7808 to branch-1	Major	.	Haohui Mai	Haohui Mai
MAPREDUCE-5569	FloatSplitter is not generating correct splits	Major	.	Nathan Roberts	Nathan Roberts
MAPREDUCE-5512	TaskTracker hung after failed reconnect to the JobTracker	Major	tasktracker	Ivan Mitic	Ivan Mitic
HDFS-4794	Browsing filesystem via webui throws kerberos exception when NN service RPC is enabled in a secure cluster	Major	security	Benoy Antony	Benoy Antony
HADOOP-10142	Avoid groups lookup for unprivileged users such as "dr.who"	Major	.	Vinayakumar B	Vinayakumar B
HADOOP-10090	Jobtracker metrics not updated properly after execution of a mapreduce job	Major	metrics	Ivan Mitic	Ivan Mitic
MAPREDUCE-5702	TaskLogServlet#printTaskLog has spurious HTML closing tags	Trivial	task	Karthik Kambatla	Robert Kanter
MAPREDUCE-5698	Backport MAPREDUCE-1285 to branch-1	Major	.	Yongjun Zhang	Yongjun Zhang
HDFS-5685	DistCp will fail to copy with -delete switch	Major	hdfs-client	Yongjun Zhang	Yongjun Zhang
MAPREDUCE-5710	Backport MAPREDUCE-1305 to branch-1	Major	.	Yongjun Zhang	Yongjun Zhang
HDFS-5944	LeaseManager:findLeaseWithPrefixPath can't handle path like /a/b/ right and cause SecondaryNameNode failed do checkpoint	Major	namenode	yunjiong zhao	yunjiong zhao
HDFS-5516	WebHDFS does not require user name when anonymous http requests are disallowed.	Major	webhdfs	Chris Nauroth	Miodrag Radulovic
MAPREDUCE-5808	Port output replication factor configurable for terasort to Hadoop 1.x	Minor	examples	Chuan Liu	Chuan Liu
MAPREDUCE-5822	FairScheduler does not preempt due to fairshare-starvation when fairshare is 1	Major	scheduler	Anubhav Dhoot	Anubhav Dhoot
HDFS-6141	WebHdfsFileSystem#toUrl does not perform character escaping.	Major	webhdfs	Chris Nauroth	Chris Nauroth
HADOOP-10562	Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager	Critical	.	Suresh Srinivas	Suresh Srinivas
MAPREDUCE-5877	Inconsistency between JT/TT for tasks taking a long time to launch	Critical	jobtracker, tasktracker	Karthik Kambatla	Karthik Kambatla
MAPREDUCE-5979	FairScheduler: zero weight can cause sort failures	Major	scheduler	Anubhav Dhoot	Anubhav Dhoot
MAPREDUCE-5966	MR1 FairScheduler use of custom weight adjuster is not thread safe for comparisons	Major	scheduler	Anubhav Dhoot	Anubhav Dhoot
MAPREDUCE-5968	Work directory is not deleted when downloadCacheObject throws IOException	Major	mrv1	zhihai xu	zhihai xu
HDFS-6822	In branch-1, Namenode and datanode fails to replace "_HOST" to hostname for hadoop.http.authentication.kerberos.principal	Major	security	Jing Zhao	Jing Zhao
MAPREDUCE-6012	DBInputSplit creates invalid ranges on Oracle	Major	.	Julien Serdaru	Wei Yan
HADOOP-11035	distcp on mr1(branch-1) fails with NPE using a short relative source path.	Major	tools	zhihai xu	zhihai xu
MAPREDUCE-6009	Map-only job with new-api runs wrong OutputCommitter when cleanup scheduled in a reduce slot	Blocker	client, job submission	Gera Shegalov	Gera Shegalov
MAPREDUCE-6147	Support mapreduce.input.fileinputformat.split.maxsize	Minor	mrv1	zhihai xu	zhihai xu
MAPREDUCE-6170	TestUlimit failure on JDK8	Major	contrib/streaming	bc Wong	bc Wong
HDFS-7503	Namenode restart after large deletions can cause slow processReport (due to logging)	Major	namenode	Arpit Agarwal	Arpit Agarwal
MAPREDUCE-6196	Fix BigDecimal ArithmeticException in PiEstimator	Minor	.	Ray Chiang	Ray Chiang
MAPREDUCE-6198	NPE from JobTracker#resolveAndAddToTopology in MR1 cause initJob and heartbeat failure.	Major	mrv1	zhihai xu	zhihai xu
MAPREDUCE-5272	A Minor Error in Javadoc of TestMRWithDistributedCache in Branch-1	Trivial	test	Zhijie Shen	Zhijie Shen
MAPREDUCE-5556	mapred docs have incorrect classpath	Trivial	.	Allen Wittenauer	Harsh J
MAPREDUCE-5660	Log info about possible thrashing (when using memory-based scheduling in Capacity Scheduler) is not printed	Trivial	capacity-sched, mrv1, tasktracker	Adam Kawa	Adam Kawa
MAPREDUCE-5183	In, TaskTracker#reportProgress logging of 0.0-1.0 progress is followed by percent sign	Minor	mrv1, tasktracker	Sandy Ryza	Niranjan Singh
HDFS-6649	Documentation for setrep is wrong	Trivial	documentation	Alexander Fahlke	Akira Ajisaka
MAPREDUCE-6076	Zero map split input length combine with none zero map split input length may cause MR1 job hung sometimes.	Major	mrv1	zhihai xu	zhihai xu

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-9624	TestFSMainOperationsLocalFileSystem failed when the Hadoop test root path has "X" in its name	Minor	test	Xi Fang	Xi Fang

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
MAPREDUCE-4049	plugin for generic shuffle service	Major	performance, task, tasktracker	Avner BenHanoch	Avner BenHanoch
HADOOP-9573	Fix test-patch script to work with the enhanced PreCommit-Admin script.	Major	build	Giridharan Kesavan	Giridharan Kesavan