Apache Hadoop Changelog

Release 3.0.0 - 2017-12-13

INCOMPATIBLE CHANGES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-6623	Add support to turn off launching privileged containers in the container-executor	Blocker	nodemanager	Varun Vasudev	Varun Vasudev
HADOOP-14816	Update Dockerfile to use Xenial	Major	build, test	Allen Wittenauer	Allen Wittenauer
HADOOP-14957	ReconfigurationTaskStatus is exposing guava Optional in its public api	Major	common	Haibo Chen	Xiao Chen
MAPREDUCE-6983	Moving logging APIs over to slf4j in hadoop-mapreduce-client-core	Major	.	Jinjiang Ling	Jinjiang Ling
HDFS-12682	ECAdmin -listPolicies will always show SystemErasureCodingPolicies state as DISABLED	Blocker	erasure-coding	Xiao Chen	Xiao Chen

NEW FEATURES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-1492	truly shared cache for jars (jobjar/libjar)	Major	.	Sangjin Lee	Chris Trezzo
HDFS-10467	Router-based HDFS federation	Major	fs	Íñigo Goiri	Íñigo Goiri
YARN-5734	OrgQueue for easy CapacityScheduler queue configuration management	Major	.	Min Shen	Min Shen
MAPREDUCE-5951	Add support for the YARN Shared Cache	Major	.	Chris Trezzo	Chris Trezzo
YARN-6871	Add additional deSelects params in RMWebServices#getAppReport	Major	resourcemanager, router	Giovanni Matteo Fumarola	Tanuj Nayak
HADOOP-14840	Tool to estimate resource requirements of an application pipeline based on prior executions	Major	tools	Subru Krishnan	Rui Li
YARN-3813	Support Application timeout feature in YARN.	Major	scheduler	nijel	Rohith Sharma K S

IMPROVEMENTS:

JIRA	Summary	Priority	Component	Reporter	Contributor
MAPREDUCE-6829	Add peak memory usage counter for each task	Major	mrv2	Yufei Gu	Miklos Szegedi
YARN-7045	Remove FSLeafQueue#addAppSchedulable	Major	fairscheduler	Yufei Gu	Sen Zhao
YARN-7240	Add more states and transitions to stabilize the NM Container state machine	Major	.	Arun Suresh	kartheek muthyala
HADOOP-14909	Fix the word of "erasure encoding" in the top page	Trivial	documentation	Takanobu Asanuma	Takanobu Asanuma
HADOOP-14095	Document caveats about the default JavaKeyStoreProvider in KMS	Major	documentation, kms	Xiao Chen	Xiao Chen
HADOOP-14928	Update site release notes for 3.0.0 GA	Major	site	Andrew Wang	Andrew Wang
HDFS-12420	Add an option to disallow 'namenode format -force'	Major	.	Ajay Kumar	Ajay Kumar
HADOOP-14521	KMS client needs retry logic	Major	.	Rushabh S Shah	Rushabh S Shah
YARN-2162	add ability in Fair Scheduler to optionally configure maxResources in terms of percentage	Major	fairscheduler, scheduler	Ashwin Shankar	Yufei Gu
YARN-7207	Cache the RM proxy server address	Major	RM	Yufei Gu	Yufei Gu
HADOOP-14939	Update project release notes with HDFS-10467 for 3.0.0	Major	.	Íñigo Goiri	Íñigo Goiri
HDFS-12573	Divide the total block metrics into replica and ec	Major	erasure-coding, metrics, namenode	Takanobu Asanuma	Takanobu Asanuma
HDFS-12553	Add nameServiceId to QJournalProtocol	Major	journal-node	Bharat Viswanadham	Bharat Viswanadham
HDFS-12603	Enable async edit logging by default	Major	namenode	Andrew Wang	Andrew Wang
HDFS-12642	Log block and datanode details in BlockRecoveryWorker	Major	datanode	Xiao Chen	Xiao Chen
HADOOP-14938	Configuration.updatingResource map should be initialized lazily	Major	.	Misha Dmitriev	Misha Dmitriev
HDFS-12613	Native EC coder should implement release() as idempotent function.	Major	erasure-coding	Lei (Eddy) Xu	Lei (Eddy) Xu
MAPREDUCE-6972	Enable try-with-resources for RecordReader	Major	.	Zoltan Haindrich	Zoltan Haindrich
HADOOP-14880	[KMS] Document&test missing KMS client side configs	Minor	.	Wei-Chiu Chuang	Gabor Bota
HDFS-12619	Do not catch and throw unchecked exceptions if IBRs fail to process	Minor	namenode	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-14771	hadoop-client does not include hadoop-yarn-client	Critical	common	Haibo Chen	Ajay Kumar
YARN-7359	TestAppManager.testQueueSubmitWithNoPermission() should be scheduler agnostic	Minor	.	Haibo Chen	Haibo Chen
HDFS-12448	Make sure user defined erasure coding policy ID will not overflow	Major	erasure-coding	Sammi Chen	Huafeng Wang
HADOOP-14944	Add JvmMetrics to KMS	Major	kms	Xiao Chen	Xiao Chen
YARN-7261	Add debug message for better download latency monitoring	Major	nodemanager	Yufei Gu	Yufei Gu
YARN-7357	Several methods in TestZKRMStateStore.TestZKRMStateStoreTester.TestZKRMStateStoreInternal should have @Override annotations	Trivial	resourcemanager	Daniel Templeton	Sen Zhao
YARN-4163	Audit getQueueInfo and getApplications calls	Major	.	Chang Li	Chang Li
HADOOP-9657	NetUtils.wrapException to have special handling for 0.0.0.0 addresses and :0 ports	Minor	net	Steve Loughran	Varun Saxena
YARN-7389	Make TestResourceManager Scheduler agnostic	Major	test	Robert Kanter	Robert Kanter
HDFS-12544	SnapshotDiff - support diff generation on any snapshot root descendant directory	Major	hdfs	Manoj Govindassamy	Manoj Govindassamy
YARN-7358	TestZKConfigurationStore and TestLeveldbConfigurationStore should explicitly set capacity scheduler	Minor	resourcemanager	Haibo Chen	Haibo Chen
YARN-7320	Duplicate LiteralByteStrings in SystemCredentialsForAppsProto.credentialsForApp_	Major	.	Misha Dmitriev	Misha Dmitriev
YARN-7262	Add a hierarchy into the ZKRMStateStore for delegation token znodes to prevent jute buffer overflow	Major	.	Robert Kanter	Robert Kanter
YARN-7397	Reduce lock contention in FairScheduler#getAppWeight()	Major	fairscheduler	Daniel Templeton	Daniel Templeton
HADOOP-14992	Upgrade Avro patch version	Major	build	Chris Douglas	Bharat Viswanadham
YARN-6413	FileSystem based Yarn Registry implementation	Major	amrmproxy, api, resourcemanager	Ellen Hui	Ellen Hui
HDFS-12482	Provide a configuration to adjust the weight of EC recovery tasks to adjust the speed of recovery	Minor	erasure-coding	Lei (Eddy) Xu	Lei (Eddy) Xu
HDFS-12744	More logs when short-circuit read is failed and disabled	Major	datanode	Weiwei Yang	Weiwei Yang
HDFS-12771	Add genstamp and block size to metasave Corrupt blocks list	Minor	.	Kuhu Shukla	Kuhu Shukla
HADOOP-14987	Improve KMSClientProvider log around delegation token checking	Major	.	Xiaoyu Yao	Xiaoyu Yao
MAPREDUCE-6975	Logging task counters	Major	task	Prabhu Joseph	Prabhu Joseph
YARN-7401	Reduce lock contention in ClusterNodeTracker#getClusterCapacity()	Major	resourcemanager	Daniel Templeton	Daniel Templeton
HDFS-7060	Avoid taking locks when sending heartbeats from the DataNode	Major	.	Haohui Mai	Jiandan Yang
YARN-7413	Support resource type in SLS	Major	scheduler-load-simulator	Yufei Gu	Yufei Gu
YARN-7386	Duplicate Strings in various places in Yarn memory	Major	.	Misha Dmitriev	Misha Dmitriev
HADOOP-14960	Add GC time percentage monitor/alerter	Major	.	Misha Dmitriev	Misha Dmitriev
HADOOP-15037	Add site release notes for OrgQueue and resource types	Major	.	Andrew Wang	Andrew Wang
HADOOP-14876	Create downstream developer docs from the compatibility guidelines	Critical	documentation	Daniel Templeton	Daniel Templeton
HADOOP-14112	Über-jira adl:// Azure Data Lake Phase I: Stabilization	Major	fs/adl	Steve Loughran	John Zhuge
HADOOP-15104	AliyunOSS: change the default value of max error retry	Major	fs/oss	wujinhu	wujinhu
HADOOP-13204	Über-jira: S3a phase III: scale and tuning	Major	fs/s3	Steve Loughran	Steve Loughran
HADOOP-14552	Über-jira: WASB client phase II: performance and testing	Major	fs/azure	Steve Loughran	Thomas Marquardt

BUG FIXES:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-7172	ResourceCalculator.fitsIn() should not take a cluster resource parameter	Major	scheduler	Daniel Templeton	Sen Zhao
HADOOP-14901	ReuseObjectMapper in Hadoop Common	Minor	.	Hanisha Koneru	Hanisha Koneru
YARN-7248	NM returns new SCHEDULED container status to older clients	Blocker	nodemanager	Jason Lowe	Arun Suresh
HADOOP-14902	LoadGenerator#genFile write close timing is incorrectly calculated	Major	fs	Jason Lowe	Hanisha Koneru
YARN-7084	TestSchedulingMonitor#testRMStarts fails sporadically	Major	.	Jason Lowe	Jason Lowe
HDFS-12453	TestDataNodeHotSwapVolumes fails in trunk Jenkins runs	Critical	test	Arpit Agarwal	Lei (Eddy) Xu
HADOOP-14915	method name is incorrect in ConfServlet	Minor	.	Bharat Viswanadham	Bharat Viswanadham
HADOOP-14752	TestCopyFromLocal#testCopyFromLocalWithThreads is fleaky	Major	test	Andras Bokor	Andras Bokor
HDFS-12569	Unset EC policy logs empty payload in edit log	Blocker	erasure-coding	Lei (Eddy) Xu	Lei (Eddy) Xu
YARN-6943	Update Yarn to YARN in documentation	Minor	documentation	Miklos Szegedi	Chetna Chaudhari
YARN-7211	AMSimulator in SLS does't work due to refactor of responseId	Blocker	scheduler-load-simulator	Yufei Gu	Botong Huang
HADOOP-14459	SerializationFactory shouldn't throw a NullPointerException if the serializations list is not defined	Minor	.	Nandor Kollar	Nandor Kollar
YARN-7044	TestContainerAllocation#testAMContainerAllocationWhenDNSUnavailable fails	Major	capacity scheduler, test	Wangda Tan	Akira Ajisaka
YARN-7226	Whitelisted variables do not support delayed variable expansion	Major	nodemanager	Jason Lowe	Jason Lowe
HADOOP-14616	Client fails to read a block with erasure code (XOR, native) when one of the data block is lost	Blocker	.	Ayappan	Huafeng Wang
YARN-7279	Fix typo in helper message of ContainerLauncher	Trivial	.	Elek, Marton	Elek, Marton
YARN-7258	Add Node and Rack Hints to Opportunistic Scheduler	Major	.	Arun Suresh	kartheek muthyala
YARN-7009	TestNMClient.testNMClientNoCleanupOnStop is flaky by design	Major	.	Miklos Szegedi	Miklos Szegedi
HDFS-12567	BlockPlacementPolicyRackFaultTolerant fails with racks with very few nodes	Major	erasure-coding	Andrew Wang	Andrew Wang
HDFS-12494	libhdfs SIGSEGV in setTLSExceptionStrings	Major	libhdfs	John Zhuge	John Zhuge
YARN-7245	Max AM Resource column in Active Users Info section of Capacity Scheduler UI page should be updated per-user	Major	capacity scheduler, yarn	Eric Payne	Eric Payne
HDFS-12606	When using native decoder, DFSStripedStream#close crashes JVM after being called multiple times.	Critical	erasure-coding	Lei (Eddy) Xu	Lei (Eddy) Xu
HDFS-12599	Remove Mockito dependency from DataNodeTestUtils	Minor	test	Ted Yu	Ted Yu
YARN-7309	TestClientRMService#testUpdateApplicationPriorityRequest and TestClientRMService#testUpdatePriorityAndKillAppWithZeroClusterResource test functionality not supported by FairScheduler	Major	test	Robert Kanter	Robert Kanter
HADOOP-14912	FairCallQueue may defer servicing calls	Major	ipc	Daryn Sharp	Daryn Sharp
HDFS-12635	Unnecessary exception declaration of the CellBuffers constructor	Minor	.	Huafeng Wang	Huafeng Wang
HDFS-12622	Fix enumerate in HDFSErasureCoding.md	Minor	documentation	Akira Ajisaka	Yiqun Lin
YARN-7082	TestContainerManagerSecurity failing in trunk	Major	.	Varun Saxena	Akira Ajisaka
HADOOP-13556	Change Configuration.getPropsWithPrefix to use getProps instead of iterator	Major	.	Larry McCay	Larry McCay
HADOOP-13102	Update GroupsMapping documentation to reflect the new changes	Major	documentation	Anu Engineer	Esther Kundin
YARN-7270	Fix unsafe casting from long to int for class Resource and its sub-classes	Major	resourcemanager	Yufei Gu	Yufei Gu
YARN-7310	TestAMRMProxy#testAMRMProxyE2E fails with FairScheduler	Major	test	Robert Kanter	Robert Kanter
YARN-7269	Tracking URL in the app state does not get redirected to ApplicationMaster for Running applications	Critical	.	Sumana Sathish	Tan, Wangda
HDFS-12659	Update TestDeadDatanode#testNonDFSUsedONDeadNodeReReg to increase heartbeat recheck interval	Minor	.	Ajay Kumar	Ajay Kumar
HDFS-12485	expunge may fail to remove trash from encryption zone	Major	.	Wei-Chiu Chuang	Wei-Chiu Chuang
HADOOP-14949	TestKMS#testACLs fails intermittently	Major	kms, test	Xiao Chen	Xiao Chen
YARN-7124	LogAggregationTFileController deletes/renames while file is open	Critical	nodemanager	Daryn Sharp	Jason Lowe
YARN-7333	container-executor fails to remove entries from a directory that is not writable or executable	Critical	.	Jason Lowe	Jason Lowe
YARN-7308	TestApplicationACLs fails with FairScheduler	Major	test	Robert Kanter	Robert Kanter
HADOOP-14948	Document missing config key hadoop.treat.subject.external	Minor	security	Wei-Chiu Chuang	Ajay Kumar
HDFS-12614	FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured	Major	.	Manoj Govindassamy	Manoj Govindassamy
YARN-7341	TestRouterWebServiceUtil#testMergeMetrics is flakey	Major	federation	Robert Kanter	Robert Kanter
HDFS-12612	DFSStripedOutputStream#close will throw if called a second time with a failed streamer	Major	erasure-coding	Andrew Wang	Lei (Eddy) Xu
HADOOP-14958	CLONE - Fix source-level compatibility after HADOOP-11252	Blocker	.	Junping Du	Junping Du
YARN-7294	TestSignalContainer#testSignalRequestDeliveryToNM fails intermittently with Fair scheduler	Major	.	Miklos Szegedi	Miklos Szegedi
YARN-7170	Improve bower dependencies for YARN UI v2	Critical	webapp	Sunil Govindan	Sunil Govindan
YARN-7355	TestDistributedShell should be scheduler agnostic	Major	.	Haibo Chen	Haibo Chen
HDFS-12497	Re-enable TestDFSStripedOutputStreamWithFailure tests	Major	erasure-coding	Andrew Wang	Huafeng Wang
HADOOP-14942	DistCp#cleanup() should check whether jobFS is null	Minor	.	Ted Yu	Andras Bokor
YARN-7318	Fix shell check warnings of SLS.	Major	.	Wangda Tan	Gergely Novák
HDFS-12518	Re-encryption should handle task cancellation and progress better	Major	encryption	Xiao Chen	Xiao Chen
HADOOP-14966	Handle JDK-8071638 for hadoop-common	Blocker	.	Bibin A Chundatt	Bibin A Chundatt
HDFS-12249	dfsadmin -metaSave to output maintenance mode blocks	Minor	namenode	Wei-Chiu Chuang	Wellington Chevreuil
HDFS-12695	Add a link to HDFS router federation document in site.xml	Minor	documentation	Yiqun Lin	Yiqun Lin
YARN-7382	NoSuchElementException in FairScheduler after failover causes RM crash	Blocker	fairscheduler	Robert Kanter	Robert Kanter
YARN-7385	TestFairScheduler#testUpdateDemand and TestFSLeafQueue#testUpdateDemand are failing with NPE	Major	test	Robert Kanter	Yufei Gu
HADOOP-14030	PreCommit TestKDiag failure	Major	security	John Zhuge	Wei-Chiu Chuang
HADOOP-14979	Upgrade maven-dependency-plugin to 3.0.2	Major	build	liyunzhang	liyunzhang
HADOOP-14977	Xenial dockerfile needs ant in main build for findbugs	Trivial	build, test	Allen Wittenauer	Akira Ajisaka
YARN-7339	LocalityMulticastAMRMProxyPolicy should handle cancel request properly	Minor	.	Botong Huang	Botong Huang
HDFS-9914	Fix configurable WebhDFS connect/read timeout	Blocker	hdfs-client, webhdfs	Xiaoyu Yao	Xiaoyu Yao
YARN-7375	Possible NPE in RMWebapp when HA is enabled and the active RM fails	Major	.	Chandni Singh	Chandni Singh
HDFS-12582	Replace HdfsFileStatus constructor with a builder pattern.	Major	.	Bharat Viswanadham	Bharat Viswanadham
HADOOP-14986	Enforce JDK limitations	Major	build	Chris Douglas	Chris Douglas
HADOOP-14991	Add missing figures to Resource Estimator tool	Major	.	Subru Krishnan	Rui Li
YARN-7299	Fix TestDistributedScheduler	Major	.	Jason Lowe	Arun Suresh
YARN-6747	TestFSAppStarvation.testPreemptionEnable fails intermittently	Major	.	Sunil Govindan	Miklos Szegedi
YARN-7336	Unsafe cast from long to int Resource.hashCode() method	Critical	resourcemanager	Daniel Templeton	Miklos Szegedi
YARN-7244	ShuffleHandler is not aware of disks that are added	Major	.	Kuhu Shukla	Kuhu Shukla
HADOOP-14990	Clean up jdiff xml files added for 2.8.2 release	Blocker	.	Subru Krishnan	Junping Du
HADOOP-14919	BZip2 drops records when reading data in splits	Critical	.	Aki Tanaka	Jason Lowe
HDFS-12699	TestMountTable fails with Java 7	Major	.	Íñigo Goiri	Íñigo Goiri
HDFS-12714	Hadoop 3 missing fix for HDFS-5169	Major	native	Joe McDonnell	Joe McDonnell
HDFS-12219	Javadoc for FSNamesystem#getMaxObjects is incorrect	Trivial	.	Erik Krogen	Erik Krogen
YARN-7412	test_docker_util.test_check_mount_permitted() is failing	Critical	nodemanager	Haibo Chen	Eric Badger
MAPREDUCE-6999	Fix typo "onf" in DynamicInputChunk.java	Trivial	.	fang zhenyi	fang zhenyi
YARN-7364	Queue dash board in new YARN UI has incorrect values	Critical	webapp	Sunil Govindan	Sunil Govindan
YARN-7370	Preemption properties should be refreshable	Major	capacity scheduler, scheduler preemption	Eric Payne	Gergely Novák
YARN-7400	incorrect log preview displayed in jobhistory server ui	Major	yarn	Santhosh B Gowda	Xuan Gong
HADOOP-15013	Fix ResourceEstimator findbugs issues	Blocker	.	Allen Wittenauer	Arun Suresh
YARN-7432	Fix DominantResourceFairnessPolicy serializable findbugs issues	Blocker	.	Allen Wittenauer	Daniel Templeton
YARN-7434	Router getApps REST invocation fails with multiple RMs	Critical	.	Subru Krishnan	Íñigo Goiri
HDFS-12725	BlockPlacementPolicyRackFaultTolerant fails with very uneven racks	Major	erasure-coding	Xiao Chen	Xiao Chen
YARN-5085	Add support for change of container ExecutionType	Major	.	Arun Suresh	Arun Suresh
YARN-7428	Add containerId to Localizer failed logs	Minor	nodemanager	Prabhu Joseph	Prabhu Joseph
YARN-7410	Cleanup FixedValueResource to avoid dependency to ResourceUtils	Major	resourcemanager	Sunil Govindan	Wangda Tan
YARN-7360	TestRM.testNMTokenSentForNormalContainer() should be scheduler agnostic	Major	test	Haibo Chen	Haibo Chen
HADOOP-15018	Update JAVA_HOME in create-release for Xenial Dockerfile	Blocker	build	Andrew Wang	Andrew Wang
HDFS-12788	Reset the upload button when file upload fails	Critical	ui, webhdfs	Brahma Reddy Battula	Brahma Reddy Battula
YARN-7453	Fix issue where RM fails to switch to active after first successful start	Blocker	resourcemanager	Rohith Sharma K S	Rohith Sharma K S
YARN-7458	TestContainerManagerSecurity is still flakey	Major	test	Robert Kanter	Robert Kanter
HADOOP-15025	Ensure singleton for ResourceEstimatorService	Major	.	Subru Krishnan	Rui Li
HDFS-12732	Correct spellings of ramdomly to randomly in log.	Trivial	.	hu xiaodong	hu xiaodong
YARN-7454	RMAppAttemptMetrics#getAggregateResourceUsage can NPE due to double lookup	Minor	resourcemanager	Jason Lowe	Jason Lowe
YARN-7388	TestAMRestart should be scheduler agnostic	Major	.	Haibo Chen	Haibo Chen
YARN-7465	start-yarn.sh fails to start ResourceManager unless running as root	Blocker	.	Sean Mackrory
HADOOP-8522	ResetableGzipOutputStream creates invalid gzip files when finish() and resetState() are used	Major	io	Mike Percy	Mike Percy
YARN-7475	Fix Container log link in new YARN UI	Major	.	Sunil Govindan	Sunil Govindan
HADOOP-15036	Update LICENSE.txt for HADOOP-14840	Major	.	Arun Suresh	Arun Suresh
YARN-6078	Containers stuck in Localizing state	Major	.	Jagadish	Billie Rinaldi
YARN-7469	Capacity Scheduler Intra-queue preemption: User can starve if newest app is exactly at user limit	Major	capacity scheduler, yarn	Eric Payne	Eric Payne
HADOOP-15067	GC time percentage reported in JvmMetrics should be a gauge, not counter	Major	.	Misha Dmitriev	Misha Dmitriev
YARN-7290	Method canContainerBePreempted can return true when it shouldn't	Major	fairscheduler	Steven Rand	Steven Rand
HDFS-12754	Lease renewal can hit a deadlock	Major	.	Kuhu Shukla	Kuhu Shukla
YARN-6647	RM can crash during transitionToStandby due to InterruptedException	Critical	resourcemanager	Jason Lowe	Bibin A Chundatt
HDFS-11754	Make FsServerDefaults cache configurable.	Minor	.	Rushabh S Shah	Mikhail Erofeev
YARN-7509	AsyncScheduleThread and ResourceCommitterService are still running after RM is transitioned to standby	Critical	.	Tao Yang	Tao Yang
YARN-7589	TestPBImplRecords fails with NullPointerException	Major	.	Jason Lowe	Daniel Templeton
HADOOP-15058	create-release site build outputs dummy shaded jars due to skipShade	Blocker	.	Andrew Wang	Andrew Wang
HDFS-12889	Router UI is missing robots.txt file	Major	.	Bharat Viswanadham	Bharat Viswanadham
HDFS-12872	EC Checksum broken when BlockAccessToken is enabled	Critical	erasure-coding	Xiao Chen	Xiao Chen
HDFS-11576	Block recovery will fail indefinitely if recovery time > heartbeat interval	Critical	datanode, hdfs, namenode	Lukas Majercak	Lukas Majercak
YARN-7607	Remove the trailing duplicated timestamp in container diagnostics message	Minor	nodemanager	Weiwei Yang	Weiwei Yang
HDFS-12840	Creating a file with non-default EC policy in a EC zone is not correctly serialized in the editlog	Blocker	erasure-coding	Lei (Eddy) Xu	Lei (Eddy) Xu
HADOOP-15080	Aliyun OSS: update oss sdk from 2.8.1 to 2.8.3 to remove its dependency on Cat-x "json-lib"	Blocker	fs/oss	Chris Douglas	Sammi Chen
HADOOP-15059	3.0 deployment cannot work with old version MR tar ball which breaks rolling upgrade	Blocker	security	Junping Du	Jason Lowe
YARN-7591	NPE in async-scheduling mode of CapacityScheduler	Critical	capacityscheduler	Tao Yang	Tao Yang
YARN-7381	Enable the configuration: yarn.nodemanager.log-container-debug-info.enabled by default in yarn-default.xml	Critical	.	Xuan Gong	Xuan Gong

SUB-TASKS:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-4081	Add support for multiple resource types in the Resource class	Major	resourcemanager	Varun Vasudev	Varun Vasudev
YARN-4172	Extend DominantResourceCalculator to account for all resources	Major	resourcemanager	Varun Vasudev	Varun Vasudev
YARN-4715	Add support to read resource types from a config file	Major	nodemanager, resourcemanager	Varun Vasudev	Varun Vasudev
YARN-4829	Add support for binary units	Major	nodemanager, resourcemanager	Varun Vasudev	Varun Vasudev
YARN-4830	Add support for resource types in the nodemanager	Major	nodemanager	Varun Vasudev	Varun Vasudev
YARN-5242	Update DominantResourceCalculator to consider all resource types in calculations	Major	resourcemanager	Varun Vasudev	Varun Vasudev
YARN-5586	Update the Resources class to consider all resource types	Major	nodemanager, resourcemanager	Varun Vasudev	Varun Vasudev
YARN-5216	Expose configurable preemption policy for OPPORTUNISTIC containers running on the NM	Major	distributed-scheduling	Arun Suresh	Hitesh Sharma
YARN-5951	Changes to allow CapacityScheduler to use configuration store	Major	.	Jonathan Hung	Jonathan Hung
YARN-5946	Create YarnConfigurationStore interface and InMemoryConfigurationStore class	Major	.	Jonathan Hung	Jonathan Hung
YARN-6232	Update resource usage and preempted resource calculations to take into account all resource types	Major	resourcemanager	Varun Vasudev	Varun Vasudev
YARN-5948	Implement MutableConfigurationManager for handling storage into configuration store	Major	.	Jonathan Hung	Jonathan Hung
YARN-5952	Create REST API for changing YARN scheduler configurations	Major	.	Jonathan Hung	Jonathan Hung
HDFS-10882	Federation State Store Interface API	Major	fs	Jason Kace	Jason Kace
YARN-6445	[YARN-3926] Performance improvements in resource profile branch with respect to SLS	Major	nodemanager, resourcemanager	Varun Vasudev	Varun Vasudev
HDFS-10630	Federation State Store FS Implementation	Major	hdfs	Íñigo Goiri	Jason Kace
YARN-5949	Add pluggable configuration ACL policy interface and implementation	Major	.	Jonathan Hung	Jonathan Hung
YARN-6575	Support global configuration mutation in MutableConfProvider	Major	.	Jonathan Hung	Jonathan Hung
YARN-5953	Create CLI for changing YARN configurations	Major	.	Jonathan Hung	Jonathan Hung
YARN-6761	Fix build for YARN-3926 branch	Major	nodemanager, resourcemanager	Varun Vasudev	Varun Vasudev
YARN-6786	ResourcePBImpl imports cleanup	Trivial	resourcemanager	Daniel Templeton	Yeliang Cang
YARN-5292	NM Container lifecycle and state transitions to support for PAUSED container state.	Major	.	Hitesh Sharma	Hitesh Sharma
HDFS-12223	Rebasing HDFS-10467	Major	fs	Íñigo Goiri	Íñigo Goiri
HDFS-10687	Federation Membership State Store internal API	Major	hdfs	Íñigo Goiri	Jason Kace
YARN-5947	Create LeveldbConfigurationStore class using Leveldb as backing store	Major	.	Jonathan Hung	Jonathan Hung
YARN-6322	Disable queue refresh when configuration mutation is enabled	Major	.	Jonathan Hung	Jonathan Hung
HDFS-11826	Federation Namenode Heartbeat	Major	fs	Íñigo Goiri	Íñigo Goiri
YARN-6788	Improve performance of resource profile branch	Blocker	nodemanager, resourcemanager	Sunil Govindan	Sunil Govindan
HDFS-10880	Federation Mount Table State Store internal API	Major	fs	Jason Kace	Íñigo Goiri
HDFS-10646	Federation admin tool	Major	fs	Íñigo Goiri	Íñigo Goiri
YARN-6994	Remove last uses of Long from resource types code	Minor	resourcemanager	Daniel Templeton	Daniel Templeton
YARN-6892	Improve API implementation in Resources and DominantResourceCalculator class	Major	nodemanager, resourcemanager	Sunil Govindan	Sunil Govindan
HDFS-11554	[Documentation] Router-based federation documentation	Minor	fs	Íñigo Goiri	Íñigo Goiri
HDFS-12312	Rebasing HDFS-10467 (2)	Major	.	Íñigo Goiri	Íñigo Goiri
YARN-6610	DominantResourceCalculator#getResourceAsValue dominant param is updated to handle multiple resources	Critical	resourcemanager	Daniel Templeton	Daniel Templeton
YARN-7030	Performance optimizations in Resource and ResourceUtils class	Critical	nodemanager, resourcemanager	Wangda Tan	Wangda Tan
YARN-7042	Clean up unit tests after YARN-6610	Major	test	Daniel Templeton	Daniel Templeton
YARN-6789	Add Client API to get all supported resource types from RM	Major	nodemanager, resourcemanager	Sunil Govindan	Sunil Govindan
YARN-6781	ResourceUtils#initializeResourcesMap takes an unnecessary Map parameter	Minor	resourcemanager	Daniel Templeton	Yu-Tang Lin
HDFS-10631	Federation State Store ZooKeeper implementation	Major	fs	Íñigo Goiri	Jason Kace
YARN-7067	Optimize ResourceType information display in UI	Critical	nodemanager, resourcemanager	Wangda Tan	Wangda Tan
YARN-7039	Fix javac and javadoc errors in YARN-3926 branch	Major	nodemanager, resourcemanager	Sunil Govindan	Sunil Govindan
YARN-7024	Fix issues on recovery in LevelDB store	Major	.	Jonathan Hung	Jonathan Hung
YARN-7093	Improve log message in ResourceUtils	Trivial	nodemanager, resourcemanager	Sunil Govindan	Sunil Govindan
YARN-6933	ResourceUtils.DISALLOWED_NAMES check is duplicated	Major	resourcemanager	Daniel Templeton	Manikandan R
YARN-5328	Plan/ResourceAllocation data structure enhancements required to support recurring reservations in ReservationSystem	Major	resourcemanager	Subru Krishnan	Subru Krishnan
HDFS-12384	Fixing compilation issue with BanDuplicateClasses	Major	fs	Íñigo Goiri	Íñigo Goiri
HDFS-12335	Federation Metrics	Major	fs	Giovanni Matteo Fumarola	Íñigo Goiri
YARN-5330	SharingPolicy enhancements required to support recurring reservations in ReservationSystem	Major	resourcemanager	Subru Krishnan	Carlo Curino
YARN-7072	Add a new log aggregation file format controller	Major	.	Xuan Gong	Xuan Gong
YARN-7136	Additional Performance Improvement for Resource Profile Feature	Critical	nodemanager, resourcemanager	Wangda Tan	Wangda Tan
YARN-7137	Move newly added APIs to unstable in YARN-3926 branch	Blocker	nodemanager, resourcemanager	Wangda Tan	Wangda Tan
YARN-6059	Update paused container state in the NM state store	Blocker	.	Hitesh Sharma	Hitesh Sharma
HDFS-12430	Rebasing HDFS-10467 After HDFS-12269 and HDFS-12218	Major	fs	Íñigo Goiri	Íñigo Goiri
YARN-7194	Log aggregation status is always Failed with the newly added log aggregation IndexedFileFormat	Major	.	Xuan Gong	Xuan Gong
YARN-6612	Update fair scheduler policies to be aware of resource types	Major	fairscheduler	Daniel Templeton	Daniel Templeton
HDFS-12450	Fixing TestNamenodeHeartbeat and support non-HA	Major	fs	Íñigo Goiri	Íñigo Goiri
YARN-6840	Implement zookeeper based store for scheduler configuration updates	Major	.	Wangda Tan	Jonathan Hung
YARN-7046	Add closing logic to configuration store	Major	.	Jonathan Hung	Jonathan Hung
HDFS-12381	[Documentation] Adding configuration keys for the Router	Minor	fs	Íñigo Goiri	Íñigo Goiri
YARN-7238	Documentation for API based scheduler configuration management	Major	.	Jonathan Hung	Jonathan Hung
YARN-7252	Removing queue then failing over results in exception	Critical	.	Jonathan Hung	Jonathan Hung
YARN-7251	Misc changes to YARN-5734	Major	.	Jonathan Hung	Jonathan Hung
YARN-7250	Update Shared cache client api to use URLs	Minor	.	Chris Trezzo	Chris Trezzo
YARN-6509	Add a size threshold beyond which yarn logs will require a force option	Major	.	Siddharth Seth	Xuan Gong
YARN-7259	Add size-based rolling policy to LogAggregationIndexedFileController	Major	.	Xuan Gong	Xuan Gong
MAPREDUCE-6971	Moving logging APIs over to slf4j in hadoop-mapreduce-client-app	Major	.	Jinjiang Ling	Jinjiang Ling
YARN-6550	Capture launch_container.sh logs to a separate log file	Major	.	Wangda Tan	Suma Shivaprasad
HDFS-12580	Rebasing HDFS-10467 after HDFS-12447	Major	fs	Íñigo Goiri	Íñigo Goiri
YARN-6916	Moving logging APIs over to slf4j in hadoop-yarn-server-common	Major	.	Akira Ajisaka	Akira Ajisaka
YARN-7219	Make AllocateRequestProto compatible with branch-2/branch-2.8	Critical	yarn	Ray Chiang	Ray Chiang
YARN-6975	Moving logging APIs over to slf4j in hadoop-yarn-server-tests, hadoop-yarn-server-web-proxy and hadoop-yarn-server-router	Major	.	Yeliang Cang	Yeliang Cang
YARN-2037	Add work preserving restart support for Unmanaged AMs	Major	resourcemanager	Karthik Kambatla	Botong Huang
YARN-5329	Placement Agent enhancements required to support recurring reservations in ReservationSystem	Blocker	resourcemanager	Subru Krishnan	Carlo Curino
YARN-6182	Fix alignment issues and missing information in new YARN UI's Queue page	Major	yarn-ui-v2	Akhil PB	Akhil PB
YARN-2960	Add documentation for the YARN shared cache	Major	.	Chris Trezzo	Chris Trezzo
HDFS-12273	Federation UI	Major	fs	Íñigo Goiri	Íñigo Goiri
HDFS-12577	Rename Router tooling	Major	fs	Íñigo Goiri	Íñigo Goiri
HDFS-12541	Extend TestSafeModeWithStripedFile with a random EC policy	Major	erasure-coding, test	Takanobu Asanuma	Takanobu Asanuma
YARN-7304	Merge YARN-5734 branch to branch-3.0	Major	.	Xuan Gong	Xuan Gong
HDFS-12547	Extend TestQuotaWithStripedBlocks with a random EC policy	Major	erasure-coding, test	Takanobu Asanuma	Takanobu Asanuma
YARN-7205	Log improvements for the ResourceUtils	Major	nodemanager, resourcemanager	Jian He	Sunil Govindan
HDFS-12637	Extend TestDistributedFileSystemWithECFile with a random EC policy	Major	erasure-coding, test	Takanobu Asanuma	Takanobu Asanuma
YARN-7275	NM Statestore cleanup for Container updates	Blocker	.	Arun Suresh	kartheek muthyala
YARN-7311	Fix TestRMWebServicesReservation parametrization for fair scheduler	Blocker	fairscheduler, reservation system	Yufei Gu	Yufei Gu
YARN-6546	SLS is slow while loading 10k queues	Major	scheduler-load-simulator	Yufei Gu	Yufei Gu
YARN-7345	GPU Isolation: Incorrect minor device numbers written to devices.deny file	Major	.	Jonathan Hung	Jonathan Hung
YARN-7338	Support same origin policy for cross site scripting prevention.	Major	yarn-ui-v2	Vrushali C	Sunil Govindan
HDFS-12620	Backporting HDFS-10467 to branch-2	Major	.	Íñigo Goiri	Íñigo Goiri
YARN-4090	Make Collections.sort() more efficient by caching resource usage	Major	fairscheduler	Xianyin Xin	Yufei Gu
YARN-7353	Docker permitted volumes don't properly check for directories	Major	yarn	Eric Badger	Eric Badger
YARN-6984	DominantResourceCalculator.isAnyMajorResourceZero() should test all resources	Major	scheduler	Daniel Templeton	Sunil Govindan
YARN-3661	Basic Federation UI	Major	nodemanager, resourcemanager	Giovanni Matteo Fumarola	Íñigo Goiri
YARN-4827	Document configuration of ReservationSystem for FairScheduler	Blocker	capacity scheduler	Subru Krishnan	Yufei Gu
YARN-5516	Add REST API for supporting recurring reservations	Major	resourcemanager	Sangeetha Abdu Jyothi	Sean Po
YARN-6505	Define the strings used in SLS JSON input file format	Major	scheduler-load-simulator	Yufei Gu	Gergely Novák
YARN-7178	Add documentation for Container Update API	Blocker	.	Arun Suresh	Arun Suresh
YARN-7374	Improve performance of DRF comparisons for resource types in fair scheduler	Critical	fairscheduler	Daniel Templeton	Daniel Templeton
YARN-6927	Add support for individual resource types requests in MapReduce	Major	resourcemanager	Daniel Templeton	Gergo Repas
YARN-7407	Moving logging APIs over to slf4j in hadoop-yarn-applications	Major	.	Yeliang Cang	Yeliang Cang
YARN-7379	Moving logging APIs over to slf4j in hadoop-yarn-client	Major	.	Yeliang Cang	Yeliang Cang
YARN-7276	Federation Router Web Service fixes	Major	.	Íñigo Goiri	Íñigo Goiri
HADOOP-14997	Add hadoop-aliyun as dependency of hadoop-cloud-storage	Minor	fs/oss	Genmao Yu	Genmao Yu
YARN-7289	Application lifetime does not work with FairScheduler	Major	resourcemanager	Miklos Szegedi	Miklos Szegedi
YARN-7286	Add support for docker to have no capabilities	Major	yarn	Eric Badger	Eric Badger
HDFS-11467	Support ErasureCoding section in OIV XML/ReverseXML	Blocker	tools	Wei-Chiu Chuang	Huafeng Wang
YARN-7307	Allow client/AM update supported resource types via YARN APIs	Blocker	nodemanager, resourcemanager	Wangda Tan	Sunil Govindan
MAPREDUCE-6997	Moving logging APIs over to slf4j in hadoop-mapreduce-client-hs	Major	.	Akira Ajisaka	Gergely Novák
MAPREDUCE-7001	Moving logging APIs over to slf4j in hadoop-mapreduce-client-shuffle	Trivial	.	Jinjiang Ling	Jinjiang Ling
YARN-7166	Container REST endpoints should report resource types	Major	resourcemanager	Daniel Templeton	Daniel Templeton
YARN-7143	FileNotFound handling in ResourceUtils is inconsistent	Major	resourcemanager	Daniel Templeton	Daniel Templeton
YARN-7437	Rename PlacementSet and SchedulingPlacementSet	Major	.	Wangda Tan	Wangda Tan
YARN-6909	Use LightWeightedResource when number of resource types more than two	Critical	resourcemanager	Daniel Templeton	Sunil Govindan
YARN-7406	Moving logging APIs over to slf4j in hadoop-yarn-api	Major	.	Yeliang Cang	Yeliang Cang
YARN-7442	[YARN-7069] Limit format of resource type name	Blocker	nodemanager, resourcemanager	Wangda Tan	Wangda Tan
YARN-7369	Improve the resource types docs	Major	docs	Daniel Templeton	Daniel Templeton
HADOOP-14993	AliyunOSS: Override listFiles and listLocatedStatus	Major	fs/oss	Genmao Yu	Genmao Yu
YARN-7430	Enable user re-mapping for Docker containers by default	Blocker	security, yarn	Eric Yang	Eric Yang
HADOOP-15024	AliyunOSS: support user agent configuration and include that & Hadoop version information to oss server	Major	fs, fs/oss	Sammi Chen	Sammi Chen
YARN-7541	Node updates don't update the maximum cluster capability for resources other than CPU and memory	Critical	resourcemanager	Daniel Templeton	Daniel Templeton

OTHER:

JIRA	Summary	Priority	Component	Reporter	Contributor
YARN-7317	Fix overallocation resulted from ceiling in LocalityMulticastAMRMProxyPolicy	Minor	.	Botong Huang	Botong Huang
HDFS-12847	Regenerate editsStored and editsStored.xml in HDFS tests	Major	test	Lei (Eddy) Xu	Lei (Eddy) Xu
HADOOP-13493	Compatibility Docs should clarify the policy for what takes precedence when a conflict is found	Critical	documentation	Robert Kanter	Daniel Templeton