Apache Hadoop Changelog

Release 2.8.2 - 2017-10-24

INCOMPATIBLE CHANGES:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-13119	Add ability to secure log servlet using proxy users	Major	.	Jeffrey E Rodriguez	Yuanbo Liu
HADOOP-14174	Set default ADLS access token provider type to ClientCredential	Major	fs/adl	John Zhuge	John Zhuge

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-7764	DirectoryScanner shouldn't abort the scan if one directory had an error	Major	datanode	Rakesh R	Rakesh R
HDFS-7541	Upgrade Domains in HDFS	Major	.	Ming Ma	Ming Ma
HDFS-10683	Make class Token$PrivateToken private	Minor	.	John Zhuge	John Zhuge
HADOOP-13688	Stop bundling HTML source code in javadoc JARs	Major	build	Andrew Wang	Andrew Wang
HADOOP-13737	Cleanup DiskChecker interface	Major	util	Arpit Agarwal	Arpit Agarwal
HADOOP-13496	Include file lengths in Mismatch in length error for distcp	Minor	.	Ted Yu	Ted Yu
HDFS-10534	NameNode WebUI should display DataNode usage histogram	Major	namenode, ui	Zhe Zhang	Kai Sasaki
HADOOP-14050	Add process name to kms process	Minor	kms, scripts	Rushabh S Shah	Rushabh S Shah
HDFS-11390	Add process name to httpfs process	Minor	httpfs, scripts	John Zhuge	Weiwei Yang
HDFS-11333	Print a user friendly error message when plugins are not found	Minor	namenode	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-11466	Change dfs.namenode.write-lock-reporting-threshold-ms default from 1000ms to 5000ms	Major	namenode	Andrew Wang	Andrew Wang
HDFS-11432	Federation : Support fully qualified path for Quota/Snapshot/cacheadmin/cryptoadmin commands	Major	federation	Brahma Reddy Battula	Brahma Reddy Battula
HADOOP-14169	Implement listStatusIterator, listLocatedStatus for ViewFs	Minor	viewfs	Erik Krogen	Erik Krogen
HADOOP-14233	Delay construction of PreCondition.check failure message in Configuration#set	Major	.	Jonathan Eagles	Jonathan Eagles
HADOOP-14240	Configuration#get return value optimization	Major	.	Jonathan Eagles	Jonathan Eagles
YARN-6339	Improve performance for createAndGetApplicationReport	Major	.	yunjiong zhao	yunjiong zhao
HDFS-9705	Refine the behaviour of getFileChecksum when length = 0	Minor	.	Kai Zheng	Sammi Chen
HDFS-11628	Clarify the behavior of HDFS Mover in documentation	Major	documentation	Xiaobing Zhou	Xiaobing Zhou
HADOOP-14104	Client should always ask namenode for kms provider path.	Major	kms	Rushabh S Shah	Rushabh S Shah
HADOOP-14276	Add a nanosecond API to Time/Timer/FakeTimer	Minor	util	Erik Krogen	Erik Krogen
HDFS-11558	BPServiceActor thread name is too long	Minor	datanode	Tsz Wo Nicholas Sze	Xiaobing Zhou
HDFS-11648	Lazy construct the IIP pathname	Major	.	Daryn Sharp	Daryn Sharp
HDFS-11634	Optimize BlockIterator when iterating starts in the middle.	Major	.	Konstantin Shvachko	Konstantin Shvachko
HDFS-11384	Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike	Major	balancer & mover	yunjiong zhao	Konstantin Shvachko
YARN-6457	Allow custom SSL configuration to be supplied in WebApps	Major	webapp, yarn	Sanjay M Pujare	Sanjay M Pujare
HDFS-11641	Reduce cost of audit logging by using FileStatus instead of HdfsFileStatus	Major	hdfs	Daryn Sharp	Daryn Sharp
YARN-6493	Print requested node partition in assignContainer logs	Major	.	Jonathan Hung	Jonathan Hung
HDFS-11891	DU#refresh should print the path of the directory when an exception is caught	Minor	.	Chen Liang	Chen Liang
HADOOP-14440	Add metrics for connections dropped	Major	.	Eric Badger	Eric Badger
HDFS-11861	ipc.Client.Connection#sendRpcRequest should log request name	Trivial	ipc	John Zhuge	John Zhuge
HDFS-11345	Document the configuration key for FSNamesystem lock fairness	Minor	documentation, namenode	Zhe Zhang	Erik Krogen
YARN-6738	LevelDBCacheTimelineStore should reuse ObjectMapper instances	Major	timelineserver	Zoltan Haindrich	Zoltan Haindrich
HADOOP-14515	Specifically configure zookeeper-related log levels in KMS log4j	Major	kms	Xiao Chen	Xiao Chen
HDFS-11881	NameNode consumes a lot of memory for snapshot diff report generation	Major	hdfs, snapshots	Manoj Govindassamy	Manoj Govindassamy
HDFS-12042	Lazy initialize AbstractINodeDiffList#diffs for snapshots to reduce memory consumption	Major	.	Misha Dmitriev	Misha Dmitriev
HDFS-12078	Add time unit to the description of property dfs.namenode.stale.datanode.interval in hdfs-default.xml	Minor	documentation, hdfs	Weiwei Yang	Weiwei Yang
HADOOP-14629	Improve exception checking in FileContext related JUnit tests	Major	fs, test	Andras Bokor	Andras Bokor
HDFS-12137	DN dataset lock should be fair	Critical	datanode	Daryn Sharp	Daryn Sharp
HADOOP-14659	UGI getShortUserName does not need to search the Subject	Major	common	Daryn Sharp	Daryn Sharp
YARN-6768	Improve performance of yarn api record toString and fromString	Major	.	Jonathan Eagles	Jonathan Eagles
YARN-5892	Support user-specific minimum user limit percentage in Capacity Scheduler	Major	capacityscheduler	Eric Payne	Eric Payne
YARN-6917	Queue path is recomputed from scratch on every allocation	Minor	capacityscheduler	Jason Lowe	Eric Payne
HDFS-12301	NN File Browser UI: Navigate to a path when enter is pressed	Trivial	ui	Ravi Prakash	Ravi Prakash
HADOOP-14251	Credential provider should handle property key deprecation	Critical	security	John Zhuge	John Zhuge
MAPREDUCE-6937	Backport MAPREDUCE-6870 to branch-2 while preserving compatibility	Major	.	Zhe Zhang	Peter Bacsko
YARN-5547	NMLeveldbStateStore should be more tolerant of unknown keys	Major	nodemanager	Jason Lowe	Ajith S
YARN-6930	Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager	Major	nodemanager	Vinod Kumar Vavilapalli	Shane Kumpf

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-524	TestYarnVersionInfo failing if generated properties doesn't include an SVN URL	Minor	api	Steve Loughran	Steve Loughran
YARN-1471	The SLS simulator is not running the preemption policy for CapacityScheduler	Minor	.	Carlo Curino	Carlo Curino
HADOOP-11703	git should ignore .DS_Store files on Mac OS X	Major	.	Abin Shahab	Abin Shahab
YARN-4612	Fix rumen and scheduler load simulator handle killed tasks properly	Major	.	Ming Ma	Ming Ma
YARN-4594	container-executor fails to remove directory tree when chmod required	Major	nodemanager	Colin P. McCabe	Colin P. McCabe
YARN-4731	container-executor should not follow symlinks in recursive_unlink_children	Blocker	.	Bibin A Chundatt	Colin P. McCabe
YARN-4812	TestFairScheduler#testContinuousScheduling fails intermittently	Major	fairscheduler	Karthik Kambatla	Karthik Kambatla
YARN-4927	TestRMHA#testTransitionedToActiveRefreshFail fails with FairScheduler	Major	test	Karthik Kambatla	Bibin A Chundatt
YARN-4562	YARN WebApp ignores the configuration passed to it for keystore settings	Major	.	Sergey Shelukhin	Sergey Shelukhin
HDFS-9276	Failed to Update HDFS Delegation Token for long running application in HA mode	Major	fs, ha, security	Liangliang Gu	Liangliang Gu
YARN-5333	Some recovered apps are put into default queue when RM HA	Major	.	Jun Gong	Jun Gong
HADOOP-13437	KMS should reload whitelist and default key ACLs when hot-reloading	Major	kms	Xiao Chen	Xiao Chen
HDFS-8312	Trash does not descent into child directories to check for permissions	Critical	fs, security	Eric Yang	Weiwei Yang
YARN-5920	Fix deadlock in TestRMHA.testTransitionedToStandbyShouldNotHang	Major	test	Rohith Sharma K S	Varun Saxena
HADOOP-13867	FilterFileSystem should override rename(.., options) to take effect of Rename options called via FilterFileSystem implementations	Major	.	Vinayakumar B	Vinayakumar B
HADOOP-13508	FsPermission string constructor does not recognize sticky bit	Major	.	Atul Sikaria	Atul Sikaria
YARN-5988	RM unable to start in secure setup	Blocker	.	Ajith S	Ajith S
YARN-6054	TimelineServer fails to start when some LevelDb state files are missing.	Critical	.	Ravi Prakash	Ravi Prakash
HADOOP-13928	TestAdlFileContextMainOperationsLive.testGetFileContext1 runtime error	Major	fs/adl, test	John Zhuge	John Zhuge
HADOOP-13976	Path globbing does not match newlines	Major	.	Eric Badger	Eric Badger
HDFS-11316	TestDataNodeVolumeFailure#testUnderReplicationAfterVolFailure fails in trunk	Minor	.	Yiqun Lin	Yiqun Lin
HADOOP-14044	Synchronization issue in delegation token cancel functionality	Major	.	Hrishikesh Gadre	Hrishikesh Gadre
HDFS-11377	Balancer hung due to no available mover threads	Major	balancer & mover	yunjiong zhao	yunjiong zhao
YARN-6031	Application recovery has failed when node label feature is turned off during RM recovery	Minor	scheduler	Ying Zhang	Ying Zhang
YARN-6137	Yarn client implicitly invoke ATS client which accesses HDFS	Major	.	Yesha Vora	Li Lu
HADOOP-13433	Race in UGI.reloginFromKeytab	Major	security	Duo Zhang	Duo Zhang
HADOOP-14058	Fix NativeS3FileSystemContractBaseTest#testDirWithDifferentMarkersWorks	Minor	fs/s3, test	Akira Ajisaka	Yiqun Lin
HDFS-11084	Add a regression test for sticky bit support of OIV ReverseXML processor	Major	tools	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-11391	Numeric usernames do no work with WebHDFS FS (write access)	Major	webhdfs	Pierre Villard	Pierre Villard
HDFS-11177	'storagepolicies -getStoragePolicy' command should accept URI based path.	Major	shell	Surendra Singh Lilhore	Surendra Singh Lilhore
HDFS-11404	Increase timeout on TestShortCircuitLocalRead.testDeprecatedGetBlockLocalPathInfoRpc	Major	.	Eric Badger	Eric Badger
HADOOP-14114	S3A can no longer handle unencoded + in URIs	Minor	fs/s3	Sean Mackrory	Sean Mackrory
MAPREDUCE-6841	Fix dead link in MapReduce tutorial document	Minor	documentation	Akira Ajisaka	Victor Nee
MAPREDUCE-6753	Variable in byte printed directly in mapreduce client	Major	client	Nemo Chen	Kai Sasaki
YARN-6263	NMTokenSecretManagerInRM.createAndGetNMToken is not thread safe	Major	yarn	Haibo Chen	Haibo Chen
HADOOP-14026	start-build-env.sh: invalid docker image name	Major	build	Gergő Pásztor	Gergő Pásztor
HDFS-11441	Add escaping to error message in KMS web UI	Minor	security	Aaron T. Myers	Aaron T. Myers
MAPREDUCE-6855	Specify charset when create String in CredentialsTestJob	Minor	.	Akira Ajisaka	Kai Sasaki
YARN-6165	Intra-queue preemption occurs even when preemption is turned off for a specific queue.	Major	capacity scheduler, scheduler preemption	Eric Payne	Eric Payne
YARN-6310	OutputStreams in AggregatedLogFormat.LogWriter can be left open upon exceptions	Major	yarn	Haibo Chen	Haibo Chen
YARN-6321	TestResources test timeouts are too aggressive	Major	test	Jason Lowe	Eric Badger
HDFS-11512	Increase timeout on TestShortCircuitLocalRead#testSkipWithVerifyChecksum	Minor	.	Eric Badger	Eric Badger
HDFS-11499	Decommissioning stuck because of failing recovery	Major	hdfs, namenode	Lukas Majercak	Lukas Majercak
HDFS-11395	RequestHedgingProxyProvider#RequestHedgingInvocationHandler hides the Exception thrown from NameNode	Major	ha	Nanda kumar	Nanda kumar
YARN-4051	ContainerKillEvent lost when container is still recovering and application finishes	Critical	nodemanager	sandflee	sandflee
YARN-6217	TestLocalCacheDirectoryManager test timeout is too aggressive	Major	test	Jason Lowe	Miklos Szegedi
HDFS-11132	Allow AccessControlException in contract tests when getFileStatus on subdirectory of existing files	Major	fs/adl, test	Vishwajeet Dusane	Vishwajeet Dusane
HADOOP-14204	S3A multipart commit failing, "UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort"	Critical	fs/s3	Steve Loughran	Steve Loughran
HADOOP-14205	No FileSystem for scheme: adl	Major	fs/adl	John Zhuge	John Zhuge
HDFS-11561	HttpFS doc errors	Trivial	documentation, httpfs, test	Yuanbo Liu	Yuanbo Liu
HADOOP-9631	ViewFs should use underlying FileSystem's server side defaults	Major	fs, viewfs	Lohit Vijayarenu	Erik Krogen
HADOOP-14214	DomainSocketWatcher::add()/delete() should not self interrupt while looping await()	Critical	hdfs-client	Mingliang Liu	Mingliang Liu
HADOOP-14195	CredentialProviderFactory$getProviders is not thread-safe	Major	security	Vihang Karajgaonkar	Vihang Karajgaonkar
HADOOP-14211	FilterFs and ChRootedFs are too aggressive about enforcing "authorityNeeded"	Major	viewfs	Erik Krogen	Erik Krogen
MAPREDUCE-6866	Fix getNumMapTasks() documentation in JobConf	Minor	documentation	Joe Mészáros	Joe Mészáros
MAPREDUCE-6868	License check for jdiff output files should be ignored	Major	build	Akira Ajisaka	Akira Ajisaka
HDFS-10506	OIV's ReverseXML processor cannot reconstruct some snapshot details	Major	tools	Colin P. McCabe	Akira Ajisaka
HDFS-11486	Client close() should not fail fast if the last block is being decommissioned	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
YARN-6359	TestRM#testApplicationKillAtAcceptedState fails rarely due to race condition	Major	test	Robert Kanter	Robert Kanter
YARN-5368	Memory leak in timeline server	Critical	timelineserver	Wataru Yukawa	Jonathan Eagles
HADOOP-14247	FileContextMainOperationsBaseTest should clean up test root path	Minor	fs, test	Mingliang Liu	Mingliang Liu
YARN-6352	Header injections are possible in application proxy servlet	Major	resourcemanager, security	Naganarasimha G R	Naganarasimha G R
MAPREDUCE-6873	MR Job Submission Fails if MR framework application path not on defaultFS	Minor	mrv2	Erik Krogen	Erik Krogen
HADOOP-14256	[S3A DOC] Correct the format for "Seoul" example	Minor	documentation, fs/s3	Brahma Reddy Battula	Brahma Reddy Battula
MAPREDUCE-6850	Shuffle Handler keep-alive connections are closed from the server side	Major	.	Jonathan Eagles	Jonathan Eagles
HDFS-11592	Closing a file has a wasteful preconditions in NameNode	Major	namenode	Eric Badger	Eric Badger
YARN-6354	LeveldbRMStateStore can parse invalid keys when recovering reservations	Major	resourcemanager	Jason Lowe	Jason Lowe
HADOOP-14268	Fix markdown itemization in hadoop-aws documents	Minor	documentation, fs/s3	Akira Ajisaka	Akira Ajisaka
YARN-6436	TestSchedulingPolicy#testParseSchedulingPolicy timeout is too low	Major	test	Jason Lowe	Eric Badger
YARN-6420	RM startup failure due to wrong order in nodelabel editlog	Critical	.	Bibin A Chundatt	Bibin A Chundatt
YARN-6403	Invalid local resource request can raise NPE and make NM exit	Major	nodemanager	Tao Yang	Tao Yang
HDFS-11538	Move ClientProtocol HA proxies into hadoop-hdfs-client	Blocker	hdfs-client	Andrew Wang	Huafeng Wang
YARN-6437	TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently	Major	test	Jason Lowe	Jason Lowe
HDFS-11629	Revert HDFS-11431 hadoop-hdfs-client JAR does not include ConfiguredFailoverProxyProvider.	Major	.	Andrew Wang	Andrew Wang
HADOOP-13996	Fix some release build issues	Blocker	build	Andrew Wang	Andrew Wang
HDFS-11608	HDFS write crashed with block size greater than 2 GB	Critical	hdfs-client	Xiaobing Zhou	Xiaobing Zhou
YARN-6288	Exceptions during aggregated log writes are mishandled	Critical	log-aggregation	Akira Ajisaka	Akira Ajisaka
HADOOP-14066	VersionInfo should be marked as public API	Critical	common	Thejas M Nair	Akira Ajisaka
HADOOP-14293	Initialize FakeTimer with a less trivial value	Major	test	Andrew Wang	Andrew Wang
YARN-6461	TestRMAdminCLI has very low test timeouts	Major	test	Jason Lowe	Eric Badger
HDFS-11163	Mover should move the file blocks to default storage once policy is unset	Major	balancer & mover	Surendra Singh Lilhore	Surendra Singh Lilhore
YARN-6450	TestContainerManagerWithLCE requires override for each new test added to ContainerManagerTest	Major	test	Jason Lowe	Jason Lowe
YARN-3760	FSDataOutputStream leak in AggregatedLogFormat.LogWriter.close()	Critical	nodemanager	Daryn Sharp	Haibo Chen
YARN-5994	TestCapacityScheduler.testAMLimitUsage fails intermittently	Major	.	Eric Badger	Eric Badger
YARN-6480	Timeout is too aggressive for TestAMRestart.testPreemptedAMRestartOnRMRestart	Major	.	Eric Badger	Eric Badger
HADOOP-14311	Add python2.7-dev to Dockerfile	Major	.	Allen Wittenauer	Allen Wittenauer
YARN-6304	Skip rm.transitionToActive call to RM if RM is already active.	Major	resourcemanager	Rohith Sharma K S	Rohith Sharma K S
HDFS-11615	FSNamesystemLock metrics can be inaccurate due to millisecond precision	Major	hdfs	Erik Krogen	Erik Krogen
HADOOP-14318	Remove non-existent setfattr command option from FileSystemShell.md	Minor	documentation	Doris Gu	Doris Gu
HADOOP-14315	Python example in the rack awareness document doesn't work due to bad indentation	Minor	documentation	Kengo Seki	Kengo Seki
HDFS-11689	New exception thrown by DFSClient#isHDFSEncryptionEnabled broke hacky hive code	Major	.	Yongjun Zhang	Yongjun Zhang
HDFS-11691	Add a proper scheme to the datanode links in NN web UI	Major	.	Kihwal Lee	Kihwal Lee
HADOOP-14341	Support multi-line value for ssl.server.exclude.cipher.list	Major	.	John Zhuge	John Zhuge
YARN-5617	AMs only intended to run one attempt can be run more than once	Major	resourcemanager	Jason Lowe	Jason Lowe
HADOOP-14346	CryptoOutputStream throws IOException on flush() if stream is closed	Major	.	Pierre Lacave	Pierre Lacave
HDFS-11709	StandbyCheckpointer should handle an non-existing legacyOivImageDir gracefully	Critical	ha, namenode	Zhe Zhang	Erik Krogen
YARN-5894	fixed license warning caused by de.ruedigermoeller:fst:jar:2.24	Blocker	yarn	Haibo Chen	Haibo Chen
HADOOP-14320	TestIPC.testIpcWithReaderQueuing fails intermittently	Major	.	Eric Badger	Eric Badger
YARN-6536	TestAMRMClient.testAMRMClientWithSaslEncryption fails intermittently	Major	.	Eric Badger	Jason Lowe
HDFS-11609	Some blocks can be permanently lost if nodes are decommissioned while dead	Blocker	namenode	Kihwal Lee	Kihwal Lee
HDFS-8498	Blocks can be committed with wrong size	Critical	hdfs-client	Daryn Sharp	Jing Zhao
HDFS-11714	Newly added NN storage directory won't get initialized and cause space exhaustion	Critical	.	Kihwal Lee	Kihwal Lee
HADOOP-14371	License error in TestLoadBalancingKMSClientProvider.java	Major	.	hu xiaodong	hu xiaodong
HADOOP-14369	NetworkTopology calls expensive toString() when logging	Major	.	Íñigo Goiri	Íñigo Goiri
HADOOP-14306	TestLocalFileSystem tests have very low timeouts	Major	.	Eric Badger	Eric Badger
HADOOP-14372	TestSymlinkLocalFS timeouts are too low	Major	.	Eric Badger	Eric Badger
HADOOP-14207	"dfsadmin -refreshCallQueue" fails with DecayRpcScheduler	Blocker	rpc-server	Surendra Singh Lilhore	Surendra Singh Lilhore
HDFS-11702	Remove indefinite caching of key provider uri in DFSClient	Major	hdfs-client	Rushabh S Shah	Rushabh S Shah
HADOOP-14374	License error in GridmixTestUtils.java	Major	.	lixinglong	lixinglong
HADOOP-14100	Upgrade Jsch jar to latest version to fix vulnerability in old versions	Critical	.	Vinayakumar B	Vinayakumar B
HADOOP-14377	Increase Common test timeouts from 1 second to 10 seconds	Major	.	Eric Badger	Eric Badger
HADOOP-14373	License error In org.apache.hadoop.metrics2.util.Servers	Major	.	hu xiaodong	hu xiaodong
YARN-6552	Increase YARN test timeouts from 1 second to 10 seconds	Major	.	Eric Badger	Eric Badger
MAPREDUCE-6882	Increase MapReduce test timeouts from 1 second to 10 seconds	Major	.	Eric Badger	Eric Badger
HADOOP-14405	Fix performance regression due to incorrect use of DataChecksum	Major	native, performance	LiXin Ge	LiXin Ge
HDFS-11745	Increase HDFS test timeouts from 1 second to 10 seconds	Major	.	Eric Badger	Eric Badger
HDFS-11755	Underconstruction blocks can be considered missing	Major	.	Nathan Roberts	Nathan Roberts
YARN-5543	ResourceManager SchedulingMonitor could potentially terminate the preemption checker thread	Major	capacityscheduler, resourcemanager	Min Shen	Min Shen
HDFS-11674	reserveSpaceForReplicas is not released if append request failed due to mirror down and replica recovered	Critical	datanode	Vinayakumar B	Vinayakumar B
HADOOP-14376	Memory leak when reading a compressed file using the native library	Major	common, io	Eli Acherkan	Eli Acherkan
HDFS-11818	TestBlockManager.testSufficientlyReplBlocksUsesNewRack fails intermittently	Major	.	Eric Badger	Nathan Roberts
YARN-6598	History server getApplicationReport NPE when fetching report for pre-2.8 job	Blocker	timelineserver	Jason Lowe	Jason Lowe
YARN-6603	NPE in RMAppsBlock	Major	resourcemanager	Jason Lowe	Jason Lowe
HDFS-11833	HDFS architecture documentation describes outdated placement policy	Minor	.	dud	Chen Liang
HADOOP-14412	HostsFileReader#getHostDetails is very expensive on large clusters	Major	util	Jason Lowe	Jason Lowe
YARN-6577	Remove unused ContainerLocalization classes	Minor	nodemanager	ZhangBing Lin	ZhangBing Lin
YARN-6618	TestNMLeveldbStateStoreService#testCompactionCycle can fail if compaction occurs more than once	Minor	test	Jason Lowe	Jason Lowe
HDFS-11849	JournalNode startup failure exception should be logged in log file	Major	journal-node	Surendra Singh Lilhore	Surendra Singh Lilhore
HDFS-11864	Document Metrics to track usage of memory for writes	Major	documentation	Brahma Reddy Battula	Yiqun Lin
YARN-6615	AmIpFilter drops query parameters on redirect	Major	.	Wilfred Spiegelenburg	Wilfred Spiegelenburg
HADOOP-14449	The ASF Header in ComparableVersion.java and SSLHostnameVerifier.java is not correct	Minor	common, documentation	ZhangBing Lin	ZhangBing Lin
HADOOP-14166	Reset the DecayRpcScheduler AvgResponseTime metric to zero when queue is not used	Major	common	Surendra Singh Lilhore	Surendra Singh Lilhore
HDFS-11661	GetContentSummary uses excessive amounts of memory	Blocker	namenode	Nathan Roberts	Wei-Chiu Chuang
YARN-6141	ppc64le on Linux doesn't trigger __linux get_executable codepath	Major	nodemanager	Sonia Garudi	Ayappan
HDFS-11445	FSCK shows overall health status as corrupt even one replica is corrupt	Critical	.	Brahma Reddy Battula	Brahma Reddy Battula
YARN-6643	TestRMFailover fails rarely due to port conflict	Major	test	Robert Kanter	Robert Kanter
HDFS-11817	A faulty node can cause a lease leak and NPE on accessing data	Critical	.	Kihwal Lee	Kihwal Lee
YARN-6641	Non-public resource localization on a bad disk causes subsequent containers failure	Major	.	Kuhu Shukla	Kuhu Shukla
HDFS-11078	Fix NPE in LazyPersistFileScrubber	Major	.	Íñigo Goiri	Íñigo Goiri
HADOOP-14464	hadoop-aws doc header warning #5 line wrapped	Trivial	documentation, fs/s3	John Zhuge	John Zhuge
HDFS-5042	Completed files lost after power failure	Critical	.	Dave Latham	Vinayakumar B
YARN-6649	RollingLevelDBTimelineServer throws RuntimeException if object decoding ever fails runtime exception	Critical	.	Jonathan Eagles	Jonathan Eagles
HDFS-11893	Fix TestDFSShell.testMoveWithTargetPortEmpty failure.	Major	test	Konstantin Shvachko	Brahma Reddy Battula
HDFS-11741	Long running balancer may fail due to expired DataEncryptionKey	Major	balancer & mover	Wei-Chiu Chuang	Wei-Chiu Chuang
HDFS-11856	Ability to re-add Upgrading Nodes (remote) to pipeline for future pipeline updates	Major	hdfs-client, rolling upgrades	Vinayakumar B	Vinayakumar B
HADOOP-14474	Use OpenJDK 7 instead of Oracle JDK 7 to avoid oracle-java7-installer failures	Major	build	Akira Ajisaka	Akira Ajisaka
HDFS-10816	TestComputeInvalidateWork#testDatanodeReRegistration fails due to race between test and replication monitor	Major	.	Eric Badger	Eric Badger
HDFS-11932	BPServiceActor thread name is not correctly set	Major	hdfs	Chen Liang	Chen Liang
HDFS-11708	Positional read will fail if replicas moved to different DNs after stream is opened	Critical	hdfs-client	Vinayakumar B	Vinayakumar B
HDFS-11711	DN should not delete the block On "Too many open files" Exception	Critical	datanode	Brahma Reddy Battula	Brahma Reddy Battula
MAPREDUCE-6676	NNBench should Throw IOException when rename and delete fails	Major	.	Brahma Reddy Battula	Brahma Reddy Battula
HADOOP-14208	Fix typo in the top page in branch-2.8	Trivial	common, documentation	Akira Ajisaka	Wenxin He
HDFS-11945	Internal lease recovery may not be retried for a long time	Major	namenode	Kihwal Lee	Kihwal Lee
HADOOP-14511	WritableRpcEngine.Invocation#toString NPE on null parameters	Minor	ipc	John Zhuge	John Zhuge
HADOOP-14512	WASB atomic rename should not throw exception if the file is neither in src nor in dst when doing the rename	Major	fs/azure	Duo Xu	Duo Xu
YARN-6585	RM fails to start when upgrading from 2.7 to 2.8 for clusters with node labels.	Blocker	.	Eric Payne	Sunil Govindan
HDFS-11967	TestJMXGet fails occasionally	Major	.	Arpit Agarwal	Arpit Agarwal
YARN-6719	Fix findbugs warnings in SLSCapacityScheduler.java	Major	.	Akira Ajisaka	Akira Ajisaka
HADOOP-14540	Replace MRv1 specific terms in HostsFileReader	Minor	documentation	Akira Ajisaka	hu xiaodong
HDFS-11995	HDFS Architecture documentation incorrectly describes writing to a local temporary file.	Minor	documentation	Chris Nauroth	Nanda kumar
HDFS-11736	OIV tests should not write outside 'target' directory.	Major	.	Konstantin Shvachko	Yiqun Lin
YARN-6713	Fix dead link in the Javadoc of FairSchedulerEventLog.java	Minor	documentation	Akira Ajisaka	Weiwei Yang
HADOOP-14533	Size of args cannot be less than zero in TraceAdmin#run as its linkedlist	Trivial	common, tracing	Weisen Han	Weisen Han
HDFS-11960	Successfully closed files can stay under-replicated.	Critical	.	Kihwal Lee	Kihwal Lee
HADOOP-14146	KerberosAuthenticationHandler should authenticate with SPN in AP-REQ	Major	security	Daryn Sharp	Daryn Sharp
YARN-6467	CSQueueMetrics needs to update the current metrics for default partition only	Major	capacity scheduler	Naganarasimha G R	Manikandan R
HADOOP-14024	KMS JMX endpoint throws ClassNotFoundException	Critical	kms	Andrew Wang	John Zhuge
YARN-6749	TestAppSchedulingInfo.testPriorityAccounting fails consistently	Major	.	Eric Badger	Naganarasimha G R
MAPREDUCE-6905	Fix meaningless operations in TestDFSIO in some situation.	Major	tools/rumen	LiXin Ge	LiXin Ge
MAPREDUCE-6909	LocalJobRunner fails when run on a node from multiple users	Blocker	client	Jason Lowe	Jason Lowe
HADOOP-13414	Hide Jetty Server version header in HTTP responses	Major	security	Vinayakumar B	Surendra Singh Lilhore
HDFS-12089	Fix ambiguous NN retry log message	Major	webhdfs	Eric Badger	Eric Badger
MAPREDUCE-6911	TestMapreduceConfigFields.testCompareXmlAgainstConfigurationClass fails consistently	Major	.	Eric Badger	Eric Badger
YARN-6708	Nodemanager container crash after ext3 folder limit	Critical	.	Bibin A Chundatt	Bibin A Chundatt
HADOOP-14563	LoadBalancingKMSClientProvider#warmUpEncryptedKeys swallows IOException	Major	.	Rushabh S Shah	Rushabh S Shah
MAPREDUCE-6246	DBOutputFormat.java appending extra semicolon to query which is incompatible with DB2	Major	mrv1, mrv2	ramtin	Gergely Novák
YARN-6428	Queue AM limit is not honored in CS always	Major	.	Bibin A Chundatt	Bibin A Chundatt
YARN-6770	[Docs] A small mistake in the example of TimelineClient	Trivial	docs	Jinjiang Ling	Jinjiang Ling
HADOOP-10829	Iteration on CredentialProviderFactory.serviceLoader is thread-unsafe	Major	security	Benoy Antony	Benoy Antony
YARN-6809	Fix typo in ResourceManagerHA.md	Trivial	documentation	Akira Ajisaka	Yeliang Cang
MAPREDUCE-5621	mr-jobhistory-daemon.sh doesn't have to execute mkdir and chown all the time	Minor	jobhistoryserver	Shinichi Yamashita	Shinichi Yamashita
YARN-6797	TimelineWriter does not fully consume the POST response	Major	timelineclient	Jason Lowe	Jason Lowe
HDFS-11502	Datanode UI should display hostname based on JMX bean instead of window.location.hostname	Major	hdfs	Jeffrey E Rodriguez	Jeffrey E Rodriguez
HADOOP-14646	FileContextMainOperationsBaseTest#testListStatusFilterWithSomeMatches never runs	Minor	test	Andras Bokor	Andras Bokor
MAPREDUCE-6697	Concurrent task limits should only be applied when necessary	Major	mrv2	Jason Lowe	Nathan Roberts
YARN-6654	RollingLevelDBTimelineStore backwards incompatible after fst upgrade	Blocker	.	Jonathan Eagles	Jonathan Eagles
YARN-6805	NPE in LinuxContainerExecutor due to null PrivilegedOperationException exit code	Major	nodemanager	Jason Lowe	Jason Lowe
YARN-3260	AM attempt fail to register before RM processes launch event	Critical	resourcemanager	Jason Lowe	Bibin A Chundatt
HDFS-12140	Remove BPOfferService lock contention to get block pool id	Critical	datanode	Daryn Sharp	Daryn Sharp
HDFS-12112	TestBlockManager#testBlockManagerMachinesArray sometimes fails with NPE	Minor	.	Wei-Chiu Chuang	Wei-Chiu Chuang
MAPREDUCE-6910	MapReduceTrackingUriPlugin can not return the right URI of history server with HTTPS	Major	jobhistoryserver	Lantao Jin	Lantao Jin
HDFS-12158	Secondary Namenode's web interface lack configs for X-FRAME-OPTIONS protection	Major	namenode	Mukul Kumar Singh	Mukul Kumar Singh
YARN-6837	Null LocalResource visibility or resource type can crash the nodemanager	Major	.	Jinjiang Ling	Jinjiang Ling
HDFS-11472	Fix inconsistent replica size after a data pipeline failure	Critical	datanode	Wei-Chiu Chuang	Erik Krogen
HDFS-12177	NameNode exits due to setting BlockPlacementPolicy loglevel to Debug	Major	block placement	Jiandan Yang	Jiandan Yang
HDFS-11742	Improve balancer usability after HDFS-8818	Blocker	.	Kihwal Lee	Kihwal Lee
HDFS-11896	Non-dfsUsed will be doubled on dead node re-registration	Blocker	.	Brahma Reddy Battula	Brahma Reddy Battula
YARN-5728	TestMiniYarnClusterNodeUtilization.testUpdateNodeUtilization timeout	Major	test	Akira Ajisaka	Akira Ajisaka
YARN-6628	Unexpected jackson-core-2.2.3 dependency introduced	Blocker	timelineserver	Jason Lowe	Jonathan Eagles
YARN-5731	Preemption calculation is not accurate when reserved containers are present in queue.	Major	capacity scheduler	Sunil Govindan	Wangda Tan
HADOOP-14683	FileStatus.compareTo binary compatible issue	Blocker	.	Sergey Shelukhin	Akira Ajisaka
YARN-6872	Ensure apps could run given NodeLabels are disabled post RM switchover/restart	Major	resourcemanager	Sunil Govindan	Sunil Govindan
YARN-6846	Nodemanager can fail to fully delete application local directories when applications are killed	Critical	nodemanager	Jason Lowe	Jason Lowe
MAPREDUCE-6927	MR job should only set tracking url if history was successfully written	Major	.	Eric Badger	Eric Badger
YARN-6890	If UI is not secured, we allow user to kill other users' job even yarn cluster is secured.	Critical	.	Sumana Sathish	Junping Du
HDFS-10326	Disable setting tcp socket send/receive buffers for write pipelines	Major	datanode, hdfs	Daryn Sharp	Daryn Sharp
HDFS-12157	Do fsyncDirectory(..) outside of FSDataset lock	Critical	datanode	Vinayakumar B	Vinayakumar B
HDFS-12278	LeaseManager operations are inefficient in 2.8.	Blocker	namenode	Rushabh S Shah	Rushabh S Shah
YARN-6987	Log app attempt during InvalidStateTransition	Major	.	Jonathan Eagles	Jonathan Eagles
YARN-7020	TestAMRMProxy#testAMRMProxyTokenRenewal is flakey	Major	.	Robert Kanter	Robert Kanter
YARN-2416	InvalidStateTransitonException in ResourceManager if AMLauncher does not receive response for startContainers() call in time	Critical	resourcemanager	Jian Fang	Jonathan Eagles
YARN-7048	Fix tests faking kerberos to explicitly set ugi auth type	Major	yarn	Daryn Sharp	Daryn Sharp
HADOOP-14687	AuthenticatedURL will reuse bad/expired session cookies	Critical	common	Daryn Sharp	Daryn Sharp
YARN-6640	AM heartbeat stuck when responseId overflows MAX_INT	Blocker	.	Botong Huang	Botong Huang
HDFS-12319	DirectoryScanner will throw IllegalStateException when Multiple BP's are present	Blocker	.	Brahma Reddy Battula	Brahma Reddy Battula
HDFS-12299	Race Between update pipeline and DN Re-Registration	Critical	.	Brahma Reddy Battula	Brahma Reddy Battula
YARN-7052	RM SchedulingMonitor gives no indication why the spawned thread crashed.	Critical	yarn	Eric Payne	Eric Payne
YARN-7087	NM failed to perform log aggregation due to absent container	Blocker	log-aggregation	Jason Lowe	Jason Lowe
YARN-7051	Avoid concurrent modification exception in FifoIntraQueuePreemptionPlugin	Critical	capacity scheduler, scheduler preemption, yarn	Eric Payne	Eric Payne
HDFS-12364	[branch-2.8.2] Fix the Compile Error after HDFS-12299	Blocker	hdfs	Jiandan Yang	Jiandan Yang
YARN-7112	TestAMRMProxy is failing with invalid request	Major	.	Jason Lowe	Jason Lowe
YARN-7076	yarn application -list -appTypes <appType> is not working	Blocker	.	Jian He	Jian He
MAPREDUCE-6641	TestTaskAttempt fails in trunk	Major	test	Tsuyoshi Ozawa	Haibo Chen
YARN-7083	Log aggregation deletes/renames while file is open	Critical	nodemanager	Daryn Sharp	Jason Lowe
HADOOP-14814	Fix incompatible API change on FsServerDefaults to HADOOP-14104	Blocker	.	Junping Du	Junping Du
YARN-7023	Incorrect ReservationId.compareTo() implementation	Minor	reservation system	Oleg Danilov	Oleg Danilov
HADOOP-14842	Hadoop 2.8.2 release build process get stuck due to java issue	Blocker	build	Junping Du	Junping Du
YARN-5195	RM intermittently crashed with NPE while handling APP_ATTEMPT_REMOVED event when async-scheduling enabled in CapacityScheduler	Major	resourcemanager	Karam Singh	sandflee
YARN-7034	DefaultLinuxContainerRuntime and DockerLinuxContainerRuntime sends client environment variables to container-executor	Blocker	nodemanager	Miklos Szegedi	Miklos Szegedi
YARN-7325	Remove unused container variable in DockerLinuxContainerRuntime	Minor	nodemanager	Shane Kumpf	Shane Kumpf
YARN-7246	Fix the default docker binary path	Blocker	nodemanager	Shane Kumpf	Shane Kumpf
YARN-7333	container-executor fails to remove entries from a directory that is not writable or executable	Critical	.	Jason Lowe	Jason Lowe
YARN-7230	Document DockerContainerRuntime for branch-2.8 with proper scope and claim as an experimental feature	Blocker	documentation	Junping Du	Shane Kumpf
HADOOP-14958	CLONE - Fix source-level compatibility after HADOOP-11252	Blocker	.	Junping Du	Junping Du
YARN-7647	NM print inappropriate error log when node-labels is enabled	Minor	.	Yang Wang	Yang Wang
YARN-7249	Fix CapacityScheduler NPE issue when a container preempted while the node is being removed	Blocker	.	Wangda Tan	Wangda Tan

TESTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HDFS-9300	TestDirectoryScanner.testThrottle() is still a little flakey	Major	balancer & mover, test	Daniel Templeton	Daniel Templeton
HADOOP-13178	TestShellBasedIdMapping.testStaticMapUpdate doesn't work on OS X	Major	test	Allen Wittenauer	Kai Sasaki
YARN-5343	TestContinuousScheduling#testSortedNodes fails intermittently	Minor	.	sandflee	Yufei Gu
MAPREDUCE-6831	Flaky test TestJobImpl.testKilledDuringKillAbort	Major	mrv2	Peter Bacsko	Peter Bacsko
HDFS-11290	TestFSNameSystemMBean should wait until JMX cache is cleared	Major	test	Akira Ajisaka	Erik Krogen
YARN-5349	TestWorkPreservingRMRestart#testUAMRecoveryOnRMWorkPreservingRestart fail intermittently	Minor	.	sandflee	Jason Lowe

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-14032	Reduce fair call queue priority inversion	Major	ipc	Daryn Sharp	Daryn Sharp
HADOOP-14034	Allow ipc layer exceptions to selectively close connections	Major	ipc	Daryn Sharp	Daryn Sharp
HADOOP-14033	Reduce fair call queue lock contention	Major	ipc	Daryn Sharp	Daryn Sharp
HADOOP-14135	Remove URI parameter in AWSCredentialProvider constructors	Major	fs/s3	Mingliang Liu	Mingliang Liu
HADOOP-14196	Azure Data Lake doc is missing required config entry	Major	fs/adl	Atul Sikaria	Atul Sikaria
HADOOP-14197	Fix ADLS doc for credential provider	Major	documentation, fs/adl	John Zhuge	John Zhuge
HADOOP-14230	TestAdlFileSystemContractLive fails to clean up	Minor	fs/adl, test	John Zhuge	John Zhuge
HADOOP-14038	Rename ADLS credential properties	Minor	fs/adl	John Zhuge	John Zhuge
HADOOP-14321	Explicitly exclude S3A root dir ITests from parallel runs	Minor	fs/s3, test	Steve Loughran	Steve Loughran
HADOOP-14241	Add ADLS sensitive config keys to default list	Minor	fs, fs/adl, security	John Zhuge	John Zhuge
HADOOP-14349	Rename ADLS CONTRACT_ENABLE_KEY	Minor	fs/adl	Mingliang Liu	Mingliang Liu
HDFS-9005	Provide configuration support for upgrade domain	Major	.	Ming Ma	Ming Ma
HDFS-9016	Display upgrade domain information in fsck	Major	.	Ming Ma	Ming Ma
HDFS-9922	Upgrade Domain placement policy status marks a good block in violation when there are decommissioned nodes	Minor	.	Chris Trezzo	Chris Trezzo
HADOOP-14035	Reduce fair call queue backoff's impact on clients	Major	ipc	Daryn Sharp	Daryn Sharp
YARN-6682	Improve performance of AssignmentInformation datastructures	Major	.	Daryn Sharp	Daryn Sharp
YARN-6680	Avoid locking overhead for NO_LABEL lookups	Major	resourcemanager	Daryn Sharp	Daryn Sharp
YARN-6681	Eliminate double-copy of child queues in canAssignToThisQueue	Major	resourcemanager	Daryn Sharp	Daryn Sharp
YARN-2113	Add cross-user preemption within CapacityScheduler's leaf-queue	Major	capacity scheduler	Vinod Kumar Vavilapalli	Sunil Govindan
YARN-6775	CapacityScheduler: Improvements to assignContainers, avoid unnecessary canAssignToUser/Queue calls	Major	capacityscheduler	Nathan Roberts	Nathan Roberts
YARN-6988	container-executor fails for docker when command length > 4096 B	Major	yarn	Eric Badger	Eric Badger
HDFS-12473	Change hosts JSON file format	Major	.	Ming Ma	Ming Ma
HDFS-9754	Avoid unnecessary getBlockCollection calls in BlockManager	Major	namenode	Jing Zhao	Jing Zhao

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor
HADOOP-14344	Revert HADOOP-13606 swift FS to add a service load metadata file	Major	.	John Zhuge	John Zhuge
HDFS-11717	Add unit test for HDFS-11709 StandbyCheckpointer should handle non-existing legacyOivImageDir gracefully	Minor	ha, namenode	Erik Krogen	Erik Krogen