Apache Hadoop Changelog

Release 2.8.3 - 2017-12-12

INCOMPATIBLE CHANGES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-13588	ConfServlet should respect Accept request header	Major	conf	Weiwei Yang	Weiwei Yang
HADOOP-14260	Configuration.dumpConfiguration should redact sensitive information	Major	conf, security	Vihang Karajgaonkar	John Zhuge

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-13933	Add haadmin -getAllServiceState option to get the HA state of all the NameNodes/ResourceManagers	Major	tools	Surendra Singh Lilhore	Surendra Singh Lilhore
HDFS-10480	Add an admin command to list currently open files	Major	.	Kihwal Lee	Manoj Govindassamy

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-13628	Support to retrieve specific property from configuration via REST API	Major	conf	Weiwei Yang	Weiwei Yang
HDFS-12143	Improve performance of getting and removing inode features	Major	namenode	Daryn Sharp	Daryn Sharp
HDFS-12171	Reduce IIP object allocations for inode lookup	Major	namenode	Daryn Sharp	Daryn Sharp
HADOOP-14455	ViewFileSystem#rename should support be supported within same nameservice with different mountpoints	Major	viewfs	Brahma Reddy Battula	Brahma Reddy Battula
HADOOP-14627	Support MSI and DeviceCode token provider in ADLS	Major	fs/adl	Atul Sikaria	Atul Sikaria
YARN-7037	Optimize data transfer with zero-copy approach for containerlogs REST API in NMWebServices	Major	nodemanager	Tao Yang	Tao Yang
HADOOP-14827	Allow StopWatch to accept a Timer parameter for tests	Minor	common, test	Erik Krogen	Erik Krogen
HDFS-12131	Add some of the FSNamesystem JMX values as metrics	Minor	hdfs, namenode	Erik Krogen	Erik Krogen
HADOOP-14844	Remove requirement to specify TenantGuid for MSI Token Provider	Major	fs/adl	Atul Sikaria	Atul Sikaria
HADOOP-14864	FSDataInputStream#unbuffer UOE should include stream class name	Minor	fs	John Zhuge	Bharat Viswanadham
HDFS-12441	Suppress UnresolvedPathException in namenode log	Minor	.	Kihwal Lee	Kihwal Lee
MAPREDUCE-6958	Shuffle audit logger should log size of shuffle transfer	Minor	.	Jason Lowe	Jason Lowe
HDFS-12420	Add an option to disallow 'namenode format -force'	Major	.	Ajay Kumar	Ajay Kumar
HADOOP-14521	KMS client needs retry logic	Major	.	Rushabh S Shah	Rushabh S Shah
HDFS-12603	Enable async edit logging by default	Major	namenode	Andrew Wang	Andrew Wang
HDFS-12642	Log block and datanode details in BlockRecoveryWorker	Major	datanode	Xiao Chen	Xiao Chen
HADOOP-14880	[KMS] Document&test missing KMS client side configs	Minor	.	Wei-Chiu Chuang	Gabor Bota
HDFS-12619	Do not catch and throw unchecked exceptions if IBRs fail to process	Minor	namenode	Wei-Chiu Chuang	Wei-Chiu Chuang
YARN-4163	Audit getQueueInfo and getApplications calls	Major	.	Chang Li	Chang Li
MAPREDUCE-6975	Logging task counters	Major	task	Prabhu Joseph	Prabhu Joseph

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-5444	Fix failing unit tests in TestLinuxContainerExecutorWithMocks	Major	nodemanager	Yufei Gu	Yufei Gu
MAPREDUCE-6808	Log map attempts as part of shuffle handler audit log	Major	.	Jonathan Eagles	Gergő Pásztor
HADOOP-14578	Bind IPC connections to kerberos UPN host for proxy users	Major	ipc	Daryn Sharp	Daryn Sharp
HADOOP-14677	mvn clean compile fails	Major	build	Andras Bokor	Andras Bokor
HADOOP-14702	Fix formatting issue and regression caused by conversion from APT to Markdown	Minor	documentation	Doris Gu	Doris Gu
YARN-6965	Duplicate instantiation in FairSchedulerQueueInfo	Minor	fairscheduler	Masahiro Tanaka	Masahiro Tanaka
HDFS-11738	Hedged pread takes more time when block moved from initial locations	Major	hdfs-client	Vinayakumar B	Vinayakumar B
HDFS-12318	Fix IOException condition for openInfo in DFSInputStream	Major	.	legend	legend
HDFS-12336	Listing encryption zones still fails when deleted EZ is not a direct child of snapshottable directory	Minor	encryption, hdfs	Wellington Chevreuil	Wellington Chevreuil
MAPREDUCE-6931	Remove TestDFSIO "Total Throughput" calculation	Critical	benchmarks, test	Dennis Huo	Dennis Huo
YARN-7116	CapacityScheduler Web UI: Queue's AM usage is always show on per-user's AM usage.	Major	capacity scheduler, webapp	Wangda Tan	Wangda Tan
HADOOP-14824	Update ADLS SDK to 2.2.2 for MSI fix	Major	fs/adl	Atul Sikaria	Atul Sikaria
YARN-7120	CapacitySchedulerPage NPE in "Aggregate scheduler counts" section	Minor	.	Eric Payne	Eric Payne
YARN-7164	TestAMRMClientOnRMRestart fails sporadically with bind address in use	Major	test	Jason Lowe	Jason Lowe
HDFS-12369	Edit log corruption due to hard lease recovery of not-closed file which has snapshots	Major	namenode	Xiao Chen	Xiao Chen
HADOOP-14867	Update HDFS Federation setup document, for incorrect property name for secondary name node http address	Major	.	Bharat Viswanadham	Bharat Viswanadham
YARN-4727	Unable to override the $HADOOP_CONF_DIR env variable for container	Major	nodemanager	Terence Yim	Jason Lowe
MAPREDUCE-6957	shuffle hangs after a node manager connection timeout	Major	mrv2	Jooseong Kim	Jooseong Kim
HDFS-12424	Datatable sorting on the Datanode Information page in the Namenode UI is broken	Major	.	Shawna Martell	Shawna Martell
HDFS-12323	NameNode terminates after full GC thinking QJM unresponsive if full GC is much longer than timeout	Major	namenode, qjm	Erik Krogen	Erik Krogen
MAPREDUCE-6960	Shuffle Handler prints disk error stack traces for every read failure.	Major	.	Kuhu Shukla	Kuhu Shukla
HDFS-11799	Introduce a config to allow setting up write pipeline with fewer nodes than replication factor	Major	.	Yongjun Zhang	Brahma Reddy Battula
YARN-6771	Use classloader inside configuration class to make new classes	Major	.	Jongyoul Lee	Jongyoul Lee
HDFS-12526	FSDirectory should use Time.monotonicNow for durations	Minor	.	Chetna Chaudhari	Bharat Viswanadham
HDFS-12371	"BlockVerificationFailures" and "BlocksVerified" show up as 0 in Datanode JMX	Major	metrics	Sai Nukavarapu	Hanisha Koneru
MAPREDUCE-6966	DistSum should use Time.monotonicNow for measuring durations	Minor	.	Chetna Chaudhari	Chetna Chaudhari
HDFS-12531	Fix conflict in the javadoc of UnderReplicatedBlocks.java in branch-2	Minor	documentation	Akira Ajisaka	Bharat Viswanadham
HDFS-12495	TestPendingInvalidateBlock#testPendingDeleteUnknownBlocks fails intermittently	Major	.	Eric Badger	Eric Badger
HADOOP-14891	Remove references to Guava Objects.toStringHelper	Major	.	Jonathan Eagles	Jonathan Eagles
HADOOP-14902	LoadGenerator#genFile write close timing is incorrectly calculated	Major	fs	Jason Lowe	Hanisha Koneru
YARN-7084	TestSchedulingMonitor#testRMStarts fails sporadically	Major	.	Jason Lowe	Jason Lowe
YARN-7226	Whitelisted variables do not support delayed variable expansion	Major	nodemanager	Jason Lowe	Jason Lowe
YARN-7285	ContainerExecutor always launches with priorities due to yarn-default property	Minor	nodemanager	Jason Lowe	Jason Lowe
YARN-7245	Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user	Major	capacity scheduler, yarn	Eric Payne	Eric Payne
HADOOP-14912	FairCallQueue may defer servicing calls	Major	ipc	Daryn Sharp	Daryn Sharp
HDFS-12659	Update TestDeadDatanode#testNonDFSUsedONDeadNodeReReg to increase heartbeat recheck interval	Minor	.	Ajay Kumar	Ajay Kumar
HDFS-12485	expunge may fail to remove trash from encryption zone	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
YARN-7333	container-executor fails to remove entries from a directory that is not writable or executable	Critical	.	Jason Lowe	Jason Lowe
HADOOP-14966	Handle JDK-8071638 for hadoop-common	Blocker	.	Bibin A Chundatt	Bibin A Chundatt
HDFS-9914	Fix configurable WebhDFS connect/read timeout	Blocker	hdfs-client, webhdfs	Xiaoyu Yao	Xiaoyu Yao
YARN-7244	ShuffleHandler is not aware of disks that are added	Major	.	Kuhu Shukla	Kuhu Shukla
HADOOP-14990	Clean up jdiff xml files added for 2.8.2 release	Blocker	.	Subru Krishnan	Junping Du
HADOOP-14919	BZip2 drops records when reading data in splits	Critical	.	Aki Tanaka	Jason Lowe
YARN-7370	Preemption properties should be refreshable	Major	capacity scheduler, scheduler preemption	Eric Payne	Gergely Novák
YARN-7361	Improve the docker container runtime documentation	Major	.	Shane Kumpf	Shane Kumpf
YARN-7469	Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit	Major	capacity scheduler, yarn	Eric Payne	Eric Payne
HADOOP-15047	Python is required for -Preleasedoc but not documented in branch-2.8	Major	build, documentation	Akira Ajisaka	Bharat Viswanadham
YARN-7496	CS Intra-queue preemption user-limit calculations are not in line with LeafQueue user-limit calculations	Major	.	Eric Payne	Eric Payne
HDFS-12832	INode.getFullPathName may throw ArrayIndexOutOfBoundsException lead to NameNode exit	Critical	namenode	DENG FEI	Konstantin Shvachko
HDFS-12638	Delete copy-on-truncate block along with the original block, when deleting a file being truncated	Blocker	hdfs	Jiandan Yang	Konstantin Shvachko

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-14678	AdlFilesystem#initialize swallows exception when getting user name	Minor	fs/adl	John Zhuge	John Zhuge
HADOOP-14892	MetricsSystemImpl should use Time.monotonicNow for measuring durations	Minor	.	Chetna Chaudhari	Chetna Chaudhari
HADOOP-14881	LoadGenerator should use Time.monotonicNow() to measure durations	Major	.	Chetna Chaudhari	Bharat Viswanadham
HADOOP-14893	WritableRpcEngine should use Time.monotonicNow	Minor	.	Chetna Chaudhari	Chetna Chaudhari
HDFS-12386	Add fsserver defaults call to WebhdfsFileSystem.	Minor	webhdfs	Rushabh S Shah	Rushabh S Shah