Back to Hadoop

Licensed to the Apache Software Foundation (ASF) under one

hadoop-common-project/hadoop-common/src/site/markdown/release/3.2.2/CHANGELOG.3.2.2.md

2.0.5-alpha-rc295.0 KB
Original Source
<!--- # Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. -->

Apache Hadoop Changelog

Release 3.2.2 - 2021-01-03

NEW FEATURES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-15691Add PathCapabilities to FS and FC to complement StreamCapabilitiesMajor.Steve LoughranSteve Loughran
YARN-9760Support configuring application priorities on a workflow levelMajor.Jonathan HungVarun Saxena
HDFS-14905Backport HDFS persistent memory read cache support to branch-3.2Majorcaching, datanodeFeilong HeFeilong He
HDFS-12943Consistent Reads from Standby NodeMajorhdfsKonstantin ShvachkoKonstantin Shvachko
HADOOP-16790Add Write Convenience MethodsMinor.David MollitorDavid Mollitor
HADOOP-17210backport HADOOP-15691 PathCapabilities API to branch-3.2Majorfs, fs/s3Steve LoughranSteve Loughran

IMPROVEMENTS:

JIRASummaryPriorityComponentReporterContributor
YARN-8750Refactor TestQueueMetricsMinorresourcemanagerSzilard NemethSzilard Nemeth
HADOOP-15849Upgrade netty version to 3.10.6Major.Xiao ChenXiao Chen
HDFS-12946Add a tool to check rack configuration against EC policiesMajorerasure-codingXiao ChenKitti Nanasi
HDFS-14113EC : Add Configuration to restrict UserDefined PoliciesMajorerasure-codingAyush SaxenaAyush Saxena
HDFS-14006Refactor name node to allow different token verification implementationsMajor.CR HotaCR Hota
HADOOP-15909KeyProvider class should implement CloseableMajorkmsKuhu ShuklaKuhu Shukla
HDFS-14061Check if the cluster topology supports the EC policy before setting, enabling or adding itMajorerasure-coding, hdfsKitti NanasiKitti Nanasi
HDFS-14187Make warning message more clear when there are not enough data nodes for EC writeMajorerasure-codingKitti NanasiKitti Nanasi
HDFS-14125Use parameterized log format in ECTopologyVerifierTrivialerasure-codingKitti NanasiKitti Nanasi
HDFS-14188Make hdfs ec -verifyClusterSetup command accept an erasure coding policy as a parameterMajorerasure-codingKitti NanasiKitti Nanasi
HADOOP-16126ipc.Client.stop() may sleep too long to wait for all connectionsMajoripcTsz-wo SzeTsz-wo Sze
HADOOP-15014KMS should log the IP address of the clientsMajorkmsZsombor GegesyZsombor Gegesy
HDFS-14460DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configuredMajor.CR HotaCR Hota
HDFS-14624When decommissioning a node, log remaining blocks to replicate periodicallyMajornamenodeStephen O'DonnellStephen O'Donnell
HDFS-13693Remove unnecessary search in INodeDirectory.addChild during image loadingMajornamenodezhouyingchaoLisheng Sun
HDFS-14313Get hdfs used space from FsDatasetImpl#volumeMap#ReplicaInfo in memory instead of df/duMajordatanode, performanceLisheng SunLisheng Sun
HDFS-14678Allow triggerBlockReport to a specific namenodeMajordatanodeLeon GaoLeon Gao
HDFS-14523Remove excess read lock for NetworkToplogyMajor.Wu WeiweiWu Weiwei
HDFS-14497Write lock held by metasave impact following RPC processingMajornamenodeXiaoqiao HeXiaoqiao He
HADOOP-16531Log more detail for slow RPCMajor.Chen ZhangChen Zhang
YARN-9764Print application submission context label in application summaryMajor.Jonathan HungManoj Kumar
YARN-9824Fall back to configured queue ordering policy class nameMajor.Jonathan HungJonathan Hung
HADOOP-16069Support configure ZK_DTSM_ZK_KERBEROS_PRINCIPAL in ZKDelegationTokenSecretManager using principal with Schema /_HOSTMinorcommonluhuachaoluhuachao
YARN-9762Add submission context label to audit logsMajor.Jonathan HungManoj Kumar
HDFS-14850Optimize FileSystemAccessService#getFileSystemConfigurationMajorhttpfs, performanceLisheng SunLisheng Sun
HDFS-14192Track missing DFS operations in Statistics and StorageStatisticsMajor.Ayush SaxenaAyush Saxena
YARN-9356Add more tests to ratio method in TestResourceCalculatorMajor.Szilard NemethZoltan Siegl
HADOOP-16643Update netty4 to the latest 4.1.42Major.Wei-Chiu ChuangLisheng Sun
HADOOP-16640WASB: Override getCanonicalServiceName() to return full url of WASB filesystemMajorfs/azureDa ZhouDa Zhou
HDFS-14915Move Superuser Check Before Taking Lock For Encryption APIMajor.Ayush SaxenaAyush Saxena
HDFS-14921Remove SuperUser Check in Setting Storage Policy in FileStatus During ListingMajor.Ayush SaxenaAyush Saxena
HDFS-14923Remove dead code from HealthMonitorMinor.Hui FeiHui Fei
YARN-9914Use separate configs for free disk space checking for full and not-full disksMinoryarnJim BrennanJim Brennan
MAPREDUCE-7208Tuning TaskRuntimeEstimatorMinor.Ahmed HusseinAhmed Hussein
HDFS-14942Change Log Level to debug in JournalNodeSyncer#syncWithJournalAtIndexMinor.Lisheng SunLisheng Sun
HDFS-14979[Observer Node] Balancer should submit getBlocks to Observer Node when possibleMajorbalancer & mover, hdfsErik KrogenErik Krogen
HADOOP-16705MBeanInfoBuilder puts unnecessary memory pressure on the system with a debug logMajormetricsLukas MajercakLukas Majercak
HADOOP-16712Config ha.failover-controller.active-standby-elector.zk.op.retries is not in core-default.xmlTrivial.Wei-Chiu ChuangXieming Li
HDFS-14952Skip safemode if blockTotal is 0 in new NNTrivialnamenodeRajesh BalamohanXiaoqiao He
YARN-8842Expose metrics for custom resource types in QueueMetricsMajor.Szilard NemethSzilard Nemeth
YARN-9966Code duplication in UserGroupMappingPlacementRuleMajor.Szilard NemethKevin Su
YARN-9937Add missing queue configs in RMWebService#CapacitySchedulerQueueInfoMajorcapacity schedulerPrabhu JosephPrabhu Joseph
HADOOP-16718Allow disabling Server Name Indication (SNI) for JettyMajor.Siyao MengAravindan Vijayan
HADOOP-16729Extract version numbers to head of pom.xmlMinorbuildTamas PenzesTamas Penzes
HADOOP-16735Make it clearer in config default that EnvironmentVariableCredentialsProvider supports AWS_SESSION_TOKENMinordocumentation, fs/s3Mingliang LiuMingliang Liu
YARN-10012Guaranteed and max capacity queue metrics for custom resourcesMajor.Jonathan HungManikandan R
HDFS-15050Optimize log information when DFSInputStream meet CannotObtainBlockLengthExceptionMajordfsclientXiaoqiao HeXiaoqiao He
YARN-10033TestProportionalCapacityPreemptionPolicy not initializing vcores for effective max resourcesMajorcapacity scheduler, testEric PayneEric Payne
YARN-10039Allow disabling app submission from REST endpointsMajor.Jonathan HungJonathan Hung
YARN-9894CapacitySchedulerPerf test for measuring hundreds of apps in a large number of queues.Majorcapacity scheduler, testEric PayneEric Payne
HADOOP-16771Update checkstyle to 8.26 and maven-checkstyle-plugin to 3.1.0MajorbuildAndras BokorAndras Bokor
YARN-10009In Capacity Scheduler, DRC can treat minimum user limit percent as a max when custom resource is definedCriticalcapacity schedulerEric PayneEric Payne
HDFS-12999When reach the end of the block group, it may not need to flush all the data packets(flushAllInternals) twice.Majorerasure-coding, hdfs-clientlufeilufei
HDFS-15074DataNode.DataTransfer thread should catch all the expception and log it.MajordatanodeSurendra Singh LilhoreHemanth Boyina
HDFS-14740Recover data blocks from persistent memory read cache during datanode restartsMajorcaching, datanodeFeilong HeFeilong He
HADOOP-16775DistCp reuses the same temp file within the task attempt for different files.Majortools/distcpAmir ShenavandehAmir Shenavandeh
HDFS-15097Purge log in KMS and HttpFSMinorhttpfs, kmsDoris GuDoris Gu
HADOOP-16753Refactor HAAdminMajorhaAkira AjisakaXieming Li
HDFS-14968Add ability to know datanode stalenessMinordatanode, logging, namenodeAhmed HusseinAhmed Hussein
YARN-7913Improve error handling when application recovery fails with exceptionMajorresourcemanagerGergo RepasWilfred Spiegelenburg
HDFS-15117EC: Add getECTopologyResultForPolicies to DistributedFileSystemMajor.Ayush SaxenaAyush Saxena
HDFS-15119Allow expiration of cached locations in DFSInputStreamMinordfsclientAhmed HusseinAhmed Hussein
MAPREDUCE-7262MRApp helpers block for long intervals (500ms)Minormr-amAhmed HusseinAhmed Hussein
MAPREDUCE-7260Cross origin request support for Job history server web UICriticaljobhistoryserverAdam AntalAdam Antal
YARN-10084Allow inheritance of max app lifetime / default app lifetimeMajorcapacity schedulerEric PayneEric Payne
HDFS-12491Support wildcard in CLASSPATH for libhdfsMajorlibhdfsJohn ZhugeMuhammad Samir Khan
YARN-10116Expose diagnostics in RMAppManager summaryMajor.Jonathan HungJonathan Hung
HADOOP-16739Fix native build failure of hadoop-pipes on CentOS 8Majortools/pipesMasatake IwasakiMasatake Iwasaki
HADOOP-16847Test TestGroupsCaching fail if HashSet iterates in a different orderMinortesttestfixer0testfixer0
HDFS-14758Decrease lease hard limitMinor.Eric PayneHemanth Boyina
HDFS-15086Block scheduled counter never get decremet if the block got deleted before replication.Major3.1.1Surendra Singh LilhoreHemanth Boyina
HDFS-15174Optimize ReplicaCachingGetSpaceUsed by reducing unnecessary io operationsMajor.Lisheng SunLisheng Sun
YARN-9018Add functionality to AuxiliaryLocalPathHandler to return all locations to read for a given pathMajor.Kuhu ShuklaKuhu Shukla
HDFS-14861Reset LowRedundancyBlocks Iterator periodicallyMajornamenodeStephen O'DonnellStephen O'Donnell
HADOOP-16899Update HdfsDesign.md to reduce ambiguityMinordocumentationAkshay NeheAkshay Nehe
HADOOP-16772Extract version numbers to head of pom.xml (addendum)MajorbuildTamas PenzesTamas Penzes
HDFS-15197[SBN read] Change ObserverRetryOnActiveException log to debugMinorhdfsChen LiangChen Liang
HADOOP-16935Backport HADOOP-10848. Cleanup calling of sun.security.krb5.Config to branch-3.2Minor.Siyao MengSiyao Meng
YARN-10200Add number of containers to RMAppManager summaryMajor.Jonathan HungJonathan Hung
HADOOP-16952Add .diff to gitignoreMinor.Ayush SaxenaAyush Saxena
MAPREDUCE-7266historyContext doesn't need to be a class attribute inside JobHistoryServerMinorjobhistoryserverSiddharth AhujaSiddharth Ahuja
YARN-10003YarnConfigurationStore#checkVersion throws exception that belongs to RMStateStoreMajor.Szilard NemethBenjamin Teke
YARN-10212Create separate configuration for max global AM attemptsMajor.Jonathan HungBilwa S T
YARN-5277When localizers fail due to resource timestamps being out, provide more diagnosticsMajornodemanagerSteve LoughranSiddharth Ahuja
YARN-9995Code cleanup in TestSchedConfCLIMinor.Szilard NemethBilwa S T
YARN-9354Resources should be created with ResourceTypesTestHelper instead of TestUtilsTrivial.Szilard NemethAndras Gyori
YARN-10002Code cleanup and improvements in ConfigurationStoreBaseTestMinor.Szilard NemethBenjamin Teke
YARN-9954Configurable max application tags and max tag lengthMajor.Jonathan HungBilwa S T
YARN-10001Add explanation of unimplemented methods in InMemoryConfigurationStoreMajor.Szilard NemethSiddharth Ahuja
HADOOP-17001The suffix name of the unified compression classMajoriobianqibianqi
YARN-9997Code cleanup in ZKConfigurationStoreMinor.Szilard NemethAndras Gyori
YARN-9996Code cleanup in QueueAdminConfigurationMutationACLPolicyMajor.Szilard NemethSiddharth Ahuja
YARN-9998Code cleanup in LeveldbConfigurationStoreMinor.Szilard NemethBenjamin Teke
YARN-9999TestFSSchedulerConfigurationStore: Extend from ConfigurationStoreBaseTest, general code cleanupMinor.Szilard NemethBenjamin Teke
HDFS-15295AvailableSpaceBlockPlacementPolicy should use chooseRandomWithStorageTypeTwoTrial() for better performance.Minor.JinglunJinglun
YARN-10189Code cleanup in LeveldbRMStateStoreMinor.Benjamin TekeBenjamin Teke
HADOOP-16886Add hadoop.http.idle_timeout.ms to core-default.xmlMajor.Wei-Chiu ChuangLisheng Sun
YARN-10260Allow transitioning queue from DRAINING to RUNNING stateMajor.Jonathan HungBilwa S T
HADOOP-17042Hadoop distcp throws "ERROR: Tools helper ///usr/lib/hadoop/libexec/tools/hadoop-distcp.sh was not found"Minortools/distcpAki TanakaAki Tanaka
HADOOP-14698Make copyFromLocal's -t option available for put as wellMajor.Andras BokorAndras Bokor
YARN-6492Generate queue metrics for each partitionMajorcapacity schedulerJonathan HungManikandan R
HADOOP-17047TODO comments exist in trunk while the related issues are already fixed.Trivial.Rungroj MaipraditRungroj Maipradit
HDFS-15406Improve the speed of Datanode Block ScanMajor.Hemanth BoyinaHemanth Boyina
HADOOP-17090Increase precommit job timeout from 5 hours to 20 hoursMajorbuildAkira AjisakaAkira Ajisaka
YARN-10297TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittentlyMajor.Jonathan HungJim Brennan
HADOOP-17127Use RpcMetrics.TIMEUNIT to initialize rpc queueTime and processingTimeMinorcommonJim BrennanJim Brennan
HDFS-15404ShellCommandFencer should expose info about sourceMajor.Chen LiangChen Liang
HADOOP-17147Dead link in hadoop-kms/index.md.vmMinordocumentation, kmsAkira AjisakaXieming Li
YARN-10343Legacy RM UI should include labeled metrics for allocated, total, and reserved resources.Major.Eric PayneEric Payne
YARN-1529Add Localization overhead metrics to NMMajornodemanagerGera ShegalovJim Brennan
YARN-10251Show extended resources on legacy RM UI.Major.Eric PayneEric Payne
HADOOP-17159Make UGI support forceful relogin from keytab ignoring the last login timeMajorsecuritySandeep GuggilamSandeep Guggilam
YARN-10353Log vcores used and cumulative cpu in containers monitorMinoryarnJim BrennanJim Brennan
YARN-10369Make NMTokenSecretManagerInRM sending NMToken for nodeId DEBUGMinoryarnJim BrennanJim Brennan
YARN-10390LeafQueue: retain user limits cache across assignContainers() callsMajorcapacity scheduler, capacityschedulerMuhammad Samir KhanMuhammad Samir Khan
HDFS-15574Remove unnecessary sort of block list in DirectoryScannerMajor.Stephen O'DonnellStephen O'Donnell
HDFS-15583Backport DirectoryScanner improvements HDFS-14476, HDFS-14751 and HDFS-15048 to branch 3.2 and 3.1MajordatanodeStephen O'DonnellStephen O'Donnell
HDFS-15581Access Controlled HTTPFS ProxyMinorhttpfsRichardRichard
HDFS-15415Reduce locking in Datanode DirectoryScannerMajordatanodeStephen O'DonnellStephen O'Donnell
HADOOP-17287Support new Instance by non default constructor by ReflectionUtilsMajor.Baolong MaoBaolong Mao
YARN-10451RM (v1) UI NodesPage can NPE when yarn.io/gpu resource type is defined.Major.Eric PayneEric Payne
YARN-9667Container-executor.c duplicates messages to stdoutMajornodemanager, yarnAdam AntalPeter Bacsko
MAPREDUCE-7301Expose Mini MR Cluster attribute for testingMinortestSwaroopa KadamSwaroopa Kadam
HDFS-15567[SBN Read] HDFS should expose msync() API to allow downstream applications call it explicitly.Majorha, hdfs-clientKonstantin ShvachkoKonstantin Shvachko
YARN-10450Add cpu and memory utilization per node and cluster-wide metricsMinoryarnJim BrennanJim Brennan
YARN-10475Scale RM-NM heartbeat interval based on node utilizationMinoryarnJim BrennanJim Brennan
HDFS-15665Balancer logging improvementMajorbalancer & moverKonstantin ShvachkoKonstantin Shvachko
HADOOP-17342Creating a token identifier should not do kerberos name resolutionMajorcommonJim BrennanJim Brennan
YARN-10479RMProxy should retry on SocketTimeout ExceptionsMajoryarnJim BrennanJim Brennan
HDFS-15623Respect configured values of rpc.engineMajorhdfsHector Sandoval ChaverriHector Sandoval Chaverri
HDFS-14395Remove WARN Logging From Interrupts in DataStreamerMinorhdfs-clientDavid MollitorDavid Mollitor
HADOOP-17367Add InetAddress api to ProxyUsers.authorizeMajorperformance, securityAhmed HusseinAhmed Hussein
MAPREDUCE-7304Enhance the map-reduce Job end notifier to be able to notify the given URL via a custom classMajormrv2Daniel FritsiZoltán Erdmann
MAPREDUCE-7309Improve performance of reading resource request for mapper/reducers from configMajorapplicationmasterWangda TanPeter Bacsko
HADOOP-17389KMS should log full UGI principalMajor.Ahmed HusseinAhmed Hussein
HDFS-15717Improve fsck loggingMajorlogging, namenodeAhmed HusseinAhmed Hussein
HDFS-15751Add documentation for msync() API to filesystem.mdMajordocumentationKonstantin ShvachkoKonstantin Shvachko

BUG FIXES:

JIRASummaryPriorityComponentReporterContributor
HADOOP-15418Hadoop KMSAuthenticationFilter needs to use getPropsByPrefix instead of iterator to avoid ConcurrentModificationExceptionMajorcommonSuma ShivaprasadSuma Shivaprasad
HDFS-14004TestLeaseRecovery2#testCloseWhileRecoverLease fails intermittently in trunkMajor.Ayush SaxenaAyush Saxena
HDFS-13959TestUpgradeDomainBlockPlacementPolicy is flakyMajor.Ayush SaxenaAyush Saxena
YARN-8948PlacementRule interface should be for all YarnSchedulersMajor.Bibin ChundattBibin Chundatt
HADOOP-16013DecayRpcScheduler decay thread should run as a daemonMajoripcErik KrogenErik Krogen
HDFS-14175EC: Native XOR decoder should reset the output buffer before using it.Majorec, hdfsSurendra Singh LilhoreAyush Saxena
HDFS-14202"dfs.disk.balancer.max.disk.throughputInMBperSec" property is not working as per set value.MajordiskbalancerRanith SardarRanith Sardar
HADOOP-16127In ipc.Client, put a new connection could happen after stopMajoripcTsz-wo SzeTsz-wo Sze
YARN-4901QueueMetrics needs to be cleared before MockRM is initializedMajorschedulerDaniel TempletonPeter Bacsko
HADOOP-16161NetworkTopology#getWeightUsingNetworkLocation return unexpected resultMajornetXiaoqiao HeXiaoqiao He
HDFS-14434webhdfs that connect secure hdfs should not use user.name parameterMinorwebhdfsKWON BYUNGCHANGKWON BYUNGCHANG
HDFS-14527Stop all DataNodes may result in NN terminateMajornamenodeXiaoqiao HeXiaoqiao He
HDFS-14494Move Server logging of StatedId inside receiveRequestState()Major.Konstantin ShvachkoShweta
HDFS-14599HDFS-12487 breaks test TestDiskBalancer.testDiskBalancerWithFedClusterWithOneNameServiceEmptyMajordiskbalancerWei-Chiu ChuangXiaoqiao He
HDFS-14618Incorrect synchronization of ArrayList field (ArrayList is thread-unsafe).Critical.Paul WardPaul Ward
HDFS-14610HashMap is not thread safe. Field storageMap is typically synchronized by storageMap. However, in one place, field storageMap is not protected with synchronized.Critical.Paul WardPaul Ward
HDFS-14499Misleading REM_QUOTA value with snapshot and trash feature enabled for a directoryMajorsnapshotsShashikant BanerjeeShashikant Banerjee
HADOOP-16451Update jackson-databind to 2.9.9.1Major.Wei-Chiu ChuangSiyao Meng
HDFS-14647NPE during secure namenode startupMajorhdfsFengnan LiFengnan Li
HADOOP-16461Regression: FileSystem cache lock parses XML within the lockMajorfsGopal VijayaraghavanGopal Vijayaraghavan
HDFS-14660[SBN Read] ObserverNameNode should throw StandbyException for requests not from ObserverProxyProviderMajor.Chao SunChao Sun
HADOOP-16460ABFS: fix for Sever Name Indication (SNI)Majorfs/azureThomas MarqardtSneha Vijayarajan
HDFS-14569Result of crypto -listZones is not formatted properlyMajor.Hemanth BoyinaHemanth Boyina
HADOOP-12282Connection thread's name should be updated after address changing is detectedMajoripczhouyingchaoLisheng Sun
HDFS-14686HttpFS: HttpFSFileSystem#getErasureCodingPolicy always returns nullMajorhttpfsSiyao MengSiyao Meng
HADOOP-15865ConcurrentModificationException in Configuration.overlay() methodMajor.Oleksandr ShevchenkoOleksandr Shevchenko
HADOOP-16487Update jackson-databind to 2.9.9.2Critical.Siyao MengSiyao Meng
HDFS-14759HDFS cat logs an info messageMajor.Eric BadgerEric Badger
HADOOP-16533Update jackson-databind to 2.9.9.3Major.Akira AjisakaAkira Ajisaka
HDFS-14699Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to thresholdCriticalecZhao Yi MingZhao Yi Ming
YARN-9833Race condition when DirectoryCollection.checkDirs() runs during container launchMajor.Peter BacskoPeter Bacsko
YARN-9837YARN Service fails to fetch status for Stopped apps with bigger spec filesMajoryarn-native-servicesTarun ParimiTarun Parimi
YARN-2255YARN Audit logging not added to log4j.propertiesMajor.Varun SaxenaAihua Xu
HDFS-14836FileIoProvider should not increase FileIoErrors metric in datanode volume metricMinor.AiphagoAiphago
HADOOP-16582LocalFileSystem's mkdirs() does not work as expected under viewfs.Major.Kihwal LeeKihwal Lee
HADOOP-16581ValueQueue does not trigger an async refill when number of values falls below watermarkMajorcommon, kmsYuval DeganiYuval Degani
HDFS-14853NPE in DFSNetworkTopology#chooseRandomWithStorageType() when the excludedNode is not presentMajor.Ranith SardarRanith Sardar
HDFS-13660DistCp job fails when new data is appended in the file while the distCp copy job is runningCriticaldistcpMukund ThakurMukund Thakur
HDFS-14808EC: Improper size values for corrupt ec block in LOGMajorecHarshakiran ReddyAyush Saxena
HDFS-14849Erasure Coding: the internal block is replicated many times when datanode is decommissioningMajorec, erasure-codingHuangTaoHuangTao
YARN-9858Optimize RMContext getExclusiveEnforcedPartitionsMajor.Jonathan HungJonathan Hung
HDFS-14492Snapshot memory leakMajorsnapshotsWei-Chiu ChuangWei-Chiu Chuang
HDFS-14418Remove redundant super user priveledge checks from namenode.Major.Ayush SaxenaAyush Saxena
HADOOP-16619Upgrade jackson and jackson-databind to 2.9.10Major.Siyao MengSiyao Meng
HDFS-14637Namenode may not replicate blocks to meet the policy after enabling upgradeDomainMajornamenodeStephen O'DonnellStephen O'Donnell
HDFS-14879Header was wrong in Snapshot web UIMajor.Hemanth BoyinaHemanth Boyina
HDFS-14655[SBN Read] Namenode crashes if one of The JN is downCritical.Harshakiran ReddyAyush Saxena
HDFS-14859Prevent unnecessary evaluation of costly operation getNumLiveDataNodes when dfs.namenode.safemode.min.datanodes is not zeroMajorhdfsSrinivasu MajetiSrinivasu Majeti
YARN-6715Fix documentation about NodeHealthScriptRunnerMajordocumentation, nodemanagerPeter BacskoPeter Bacsko
YARN-9552FairScheduler: NODE_UPDATE can cause NoSuchElementExceptionMajorfairschedulerPeter BacskoPeter Bacsko
HDFS-14754Erasure Coding : The number of Under-Replicated Blocks never reducedCriticalecHemanth BoyinaHemanth Boyina
HDFS-14245Class cast error in GetGroups with ObserverReadProxyProviderMajor.Shen YinjieErik Krogen
HDFS-14373EC : Decoding is failing when block group last incomplete cell fall in to AlignedStripeCriticalec, hdfs-clientSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-14509DN throws InvalidToken due to inequality of password when upgrade NN 2.x to 3.xBlocker.Yuxuan WangYuxuan Wang
HDFS-14886In NameNode Web UI's Startup Progress page, Loading edits always shows 0 secMajor.Hemanth BoyinaHemanth Boyina
YARN-8453Additional Unit tests to verify queue limit and max-limit with multiple resource typesMajorcapacity schedulerSunil GAdam Antal
HDFS-14890Setting permissions on name directory fails on non posix compliant filesystemsBlocker.hirikSiddharth Wagle
HADOOP-16580Disable retry of FailoverOnNetworkExceptionRetry in case of AccessControlExceptionMajorcommonAdam AntalAdam Antal
HDFS-14909DFSNetworkTopology#chooseRandomWithStorageType() should not decrease storage count for excluded node which is already part of excluded scopeMajornamenodeSurendra Singh LilhoreSurendra Singh Lilhore
HADOOP-16662Remove unnecessary InnerNode check in NetworkTopology#add()Minor.Lisheng SunLisheng Sun
HDFS-14847Erasure Coding: Blocks are over-replicated while EC decommissioningCriticalecHui FeiHui Fei
HDFS-14913Correct the value of available count in DFSNetworkTopology#chooseRandomWithStorageType()Major.Ayush SaxenaAyush Saxena
YARN-9915Fix FindBug issue in QueueMetricsMinor.Prabhu JosephPrabhu Joseph
HDFS-12749DN may not send block report to NN after NN restartMajordatanodeTanYuxinXiaoqiao He
HDFS-13901INode access time is ignored because of race between open and renameMajor.JinglunJinglun
HDFS-14910Rename Snapshot with Pre Descendants Fail With IllegalArgumentException.Blocker.Íñigo GoiriWei-Chiu Chuang
HDFS-14308DFSStripedInputStream curStripeBuf is not freed by unbuffer()MajorecJoe McDonnellZhao Yi Ming
HDFS-14931hdfs crypto commands limit column widthMajor.Eric BadgerEric Badger
HADOOP-16669TestRawLocalFileSystemContract.testPermission fails if no native libraryMinorcommon, testSteve LoughranSteve Loughran
HDFS-14920Erasure Coding: Decommission may hang If one or more datanodes are out of service during decommissionMajorecHui FeiHui Fei
HDFS-13736BlockPlacementPolicyDefault can not choose favored nodes when 'dfs.namenode.block-placement-policy.default.prefer-local-node' set to falseMajor.hu xiaodonghu xiaodong
HDFS-14925rename operation should check nest snapshotMajornamenodeJunwang ZhaoJunwang Zhao
YARN-9949Add missing queue configs for root queue in RMWebService#CapacitySchedulerInfoMinorcapacity schedulerPrabhu JosephPrabhu Joseph
HDFS-14945Revise PacketResponder's log.MinordatanodeXudong CaoXudong Cao
HDFS-14946Erasure Coding: Block recovery failed during decommissioningMajor.Hui FeiHui Fei
HDFS-14384When lastLocatedBlock token expire, it will take 1~3s second to refetch it.Majorhdfs-clientSurendra Singh LilhoreSurendra Singh Lilhore
HDFS-14806Bootstrap standby may fail if used in-progress tailingMajornamenodeChen LiangChen Liang
HDFS-14941Potential editlog race condition can cause corrupted fileMajornamenodeChen LiangChen Liang
HDFS-14958TestBalancerWithNodeGroup is not using NetworkTopologyWithNodeGroupMinorhdfsJim BrennanJim Brennan
HDFS-14720DataNode shouldn't report block as bad block if the block length is Long.MAX_VALUE.MajordatanodeSurendra Singh LilhoreHemanth Boyina
HADOOP-16676Backport HADOOP-16152 to branch-3.2MajorcommonDWSiyao Meng
HADOOP-16677Recalculate the remaining timeout millis correctly while throwing an InterupptedException in SocketIOWithTimeout.MinorcommonXudong CaoXudong Cao
HDFS-14884Add sanity check that zone key equals feinfo key while setting XattrsMajorencryption, hdfsMukul Kumar SinghMukul Kumar Singh
HADOOP-15097AbstractContractDeleteTest::testDeleteNonEmptyDirRecursive with misleading pathMinorfs, testzhoutai.ztXieming Li
HADOOP-16710testing_azure.md documentation is misleadingMajorfs/azure, testAndras BokorAndras Bokor
YARN-9984FSPreemptionThread can cause NullPointerException while app is unregistered with containers running on a nodeMajorfairschedulerWilfred SpiegelenburgWilfred Spiegelenburg
YARN-9983Typo in YARN Service overview documentationTrivialdocumentationDenes GerencserDenes Gerencser
HADOOP-16719Remove the disallowed element config within maven-checkstyle-pluginMajor.Wanqiang JiWanqiang Ji
HADOOP-16700RpcQueueTime may be negative when the response has to be sent laterMinor.xuzqxuzq
HADOOP-15686Supress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderrMajorkmsWei-Chiu ChuangWei-Chiu Chuang
HDFS-14940HDFS Balancer : Do not allow to set balancer maximum network bandwidth more than 1TBMinorbalancer & moverSouryakanta DwivedyHemanth Boyina
YARN-9838Fix resource inconsistency for queues when moving app with reserved container to another queueCriticalcapacity schedulerjiulongzhujiulongzhu
YARN-9968Public Localizer is exiting in NodeManager due to NullPointerExceptionMajornodemanagerTarun ParimiTarun Parimi
YARN-9011Race condition during decommissioningMajornodemanagerPeter BacskoPeter Bacsko
HDFS-14973Balancer getBlocks RPC dispersal does not function properlyMajorbalancer & moverErik KrogenErik Krogen
HADOOP-16685FileSystem#listStatusIterator does not check if given path existsMajorfsSahil TakiarSahil Takiar
MAPREDUCE-7240Exception ' Invalid event: TA_TOO_MANY_FETCH_FAILURE at SUCCESS_FINISHING_CONTAINER' cause job errorCritical.luhuachaoluhuachao
MAPREDUCE-7249Invalid event TA_TOO_MANY_FETCH_FAILURE at SUCCESS_CONTAINER_CLEANUP causes job failureCriticalapplicationmaster, mrv2Wilfred SpiegelenburgWilfred Spiegelenburg
YARN-9993Remove incorrectly committed files from YARN-9011MajoryarnWilfred SpiegelenburgWilfred Spiegelenburg
HDFS-15010BlockPoolSlice#addReplicaThreadPool static pool should be initialized by static methodMajordatanodeSurendra Singh LilhoreSurendra Singh Lilhore
HADOOP-16744Fix building instruction to enable zstdMinordocumentationMasatake IwasakiMasatake Iwasaki
YARN-9985Unsupported "transitionToObserver" option displaying for rmadmin commandMinorRM, yarnSouryakanta DwivedyAyush Saxena
HADOOP-16754Fix docker failed to build yetus/hadoopBlockerbuildKevin SuKevin Su
HDFS-15032Balancer crashes when it fails to contact an unavailable NN via ObserverReadProxyProviderMajorbalancer & moverErik KrogenErik Krogen
HDFS-15036Active NameNode should not silently fail the image transferMajornamenodeKonstantin ShvachkoChen Liang
HDFS-14519NameQuota is not update after concat operation, so namequota is wrongMajor.Ranith SardarRanith Sardar
YARN-10055bower install failsBlockerbuild, yarn-ui-v2Akira AjisakaAkira Ajisaka
HDFS-15076Fix tests that hold FSDirectory lock, without holding FSNamesystem lock.MajortestKonstantin ShvachkoKonstantin Shvachko
HDFS-15073Replace curator-shaded guava import with the standard oneMinorhdfs-clientAkira AjisakaChandra Sanivarapu
HADOOP-16042Update the link to HadoopJavaVersionMinordocumentationAkira AjisakaChandra Sanivarapu
HDFS-14934[SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0Major.Takanobu AsanumaAyush Saxena
YARN-10053Placement rules do not use correct group service initMajoryarnWilfred SpiegelenburgWilfred Spiegelenburg
HDFS-15068DataNode could meet deadlock if invoke refreshVolumes when registerMajordatanodeXiaoqiao HeAiphago
MAPREDUCE-7255Fix typo in MapReduce documentaion exampleTrivialdocumentationSergey PogorelovSergey Pogorelov
HDFS-15072HDFS MiniCluster fails to start when run in directory path with a %Minor.Geoffrey JacobyMasatake Iwasaki
HDFS-15077Fix intermittent failure of TestDFSClientRetries#testLeaseRenewSocketTimeoutMinortestMasatake IwasakiMasatake Iwasaki
HDFS-15080Fix the issue in reading persistent memory cached data with an offsetMajorcaching, datanodeFeilong HeFeilong He
YARN-7387org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer fails intermittentlyMajor.Miklos SzegediJim Brennan
YARN-8672TestContainerManager#testLocalingResourceWhileContainerRunning occasionally times outMajornodemanagerJason Darrell LoweChandni Singh
HDFS-14957INodeReference Space Consumed was not same in QuotaUsage and ContentSummaryMajornamenodeHemanth BoyinaHemanth Boyina
MAPREDUCE-7252Handling 0 progress in SimpleExponential task runtime estimatorMinor.Ahmed HusseinAhmed Hussein
HADOOP-16749Configuration parsing of CDATA values are blankMajorconfJonathan Turner EaglesDaryn Sharp
HDFS-15095Fix accidental comment in flaky test TestDecommissioningStatusMajorhdfsAhmed HusseinAhmed Hussein
HDFS-15099[SBN Read] checkOperation(WRITE) should throw ObserverRetryOnActiveException on ObserverNodeMajornamenodeKonstantin ShvachkoChen Liang
HDFS-14578AvailableSpaceBlockPlacementPolicy always prefers local nodeMajorblock placementWei-Chiu ChuangAyush Saxena
HADOOP-16683Disable retry of FailoverOnNetworkExceptionRetry in case of wrapped AccessControlExceptionMajorcommonAdam AntalAdam Antal
MAPREDUCE-7256Fix javadoc error in SimpleExponentialSmoothingMinordocumentationMasatake IwasakiMasatake Iwasaki
YARN-8373RM Received RMFatalEvent of type CRITICAL_THREAD_CRASHMajorfairscheduler, resourcemanagerGirish BhatWilfred Spiegelenburg
MAPREDUCE-7247Modify HistoryServerRest.html content,change The job attempt id‘s datatype from string to intMajordocumentationzhaoshengjiezhaoshengjie
YARN-9970Refactor TestUserGroupMappingPlacementRule#verifyQueueMappingMajor.Manikandan RManikandan R
YARN-8148Update decimal values for queue capacities shown on queue status CLIMajorclientPrabhu JosephPrabhu Joseph
HADOOP-16808Use forkCount and reuseForks parameters instead of forkMode in the config of maven surefire pluginMinorbuildAkira AjisakaXieming Li
HADOOP-16793Remove WARN log when ipc connection interrupted in Client#handleSaslConnectionFailure()Minor.Lisheng SunLisheng Sun
YARN-9462TestResourceTrackerService.testNodeRemovalGracefully fails sporadicallyMinorresourcemanager, testPrabhu JosephPrabhu Joseph
YARN-9790Failed to set default-application-lifetime if maximum-application-lifetime is less than or equal to zeroMajor.kyungwan namkyungwan nam
HDFS-14993checkDiskError doesn't work during datanode startupMajordatanodeYang YunYang Yun
HDFS-13179TestLazyPersistReplicaRecovery#testDnRestartWithSavedReplicas fails intermittentlyCriticalfsGabor BotaAhmed Hussein
MAPREDUCE-7259testSpeculateSuccessfulWithUpdateEvents fails IntermittentlyMinor.Ahmed HusseinAhmed Hussein
HDFS-15146TestBalancerRPCDelay.testBalancerRPCDelay fails intermittentlyMinor.Ahmed HusseinAhmed Hussein
MAPREDUCE-7079JobHistory#ServiceStop implementation is incorrectMajor.Jason Darrell LoweAhmed Hussein
HDFS-15118[SBN Read] Slow clients when Observer reads are enabled but there are no Observers on the cluster.Majorhdfs-clientKonstantin ShvachkoChen Liang
HDFS-7175Client-side SocketTimeoutException during FsckMajornamenodeCarl SteinbachStephen O'Donnell
HDFS-15148dfs.namenode.send.qop.enabled should not apply to primary NN portMajor.Chen LiangChen Liang
HADOOP-16410Hadoop 3.2 azure jars incompatible with alpine 3.9Minorfs/azureJose Luis Pedrosa
HDFS-15115Namenode crash caused by NPE in BlockPlacementPolicyDefault when dynamically change logger to debugMajor.wangzhixiangwangzhixiang
HDFS-15158The number of failed volumes mismatch with volumeFailures of Datanode metricsMinordatanodeYang YunYang Yun
HADOOP-16849start-build-env.sh behaves incorrectly when username is numeric onlyMinorbuildJihyun ChoJihyun Cho
HDFS-15161When evictableMmapped or evictable size is zero, do not throw NoSuchElementException in ShortCircuitCache#close()Major.Lisheng SunLisheng Sun
HDFS-15164Fix TestDelegationTokensWithHAMajor.Ayush SaxenaAyush Saxena
HADOOP-16868ipc.Server readAndProcess threw NullPointerExceptionMajorrpc-serverTsz-wo SzeTsz-wo Sze
HADOOP-16869Upgrade findbugs-maven-plugin to 3.0.5 to fix mvn findbugs:findbugs failureMajorbuildAkira AjisakaAkira Ajisaka
HDFS-15052WebHDFS getTrashRoot leads to OOM due to FileSystem object creationMajorwebhdfsWei-Chiu ChuangMasatake Iwasaki
HDFS-15185StartupProgress reports edits segments until the entire startup completesMajornamenodeKonstantin ShvachkoKonstantin Shvachko
HDFS-15166Remove redundant field fStream in ByteStringLogMajor.Konstantin ShvachkoXieming Li
YARN-10143YARN-10101 broke Yarn logs CLIBlockeryarnAdam AntalAdam Antal
HADOOP-16841The description of hadoop.http.authentication.signature.secret.file contains outdated informationMinordocumentationAkira AjisakaXieming Li
YARN-10156Fix typo 'complaint' which means quite different in Federation.mdMinordocumentation, federationSungpeo KookSungpeo Kook
HDFS-15147LazyPersistTestCase wait logic is error-proneMinor.Ahmed HusseinAhmed Hussein
HDFS-14668Support Fuse with Users from multiple Security RealmsCriticalfuse-dfsSailesh PatelIstván Fajth
HDFS-15111stopStandbyServices() should log which service state it is transitioning from.Majorhdfs, loggingKonstantin ShvachkoXieming Li
HDFS-15199NPE in BlockSenderMajor.Ayush SaxenaAyush Saxena
HADOOP-16891Upgrade jackson-databind to 2.9.10.3Blocker.Siyao MengSiyao Meng
HDFS-15204TestRetryCacheWithHA testRemoveCacheDescriptor fails intermittentlyMajorhdfsAhmed HusseinAhmed Hussein
HADOOP-16840AliyunOSS: getFileStatus throws FileNotFoundException in versioning bucketMajorfs/osswujinhuwujinhu
YARN-9419Log a warning if GPU isolation is enabled but LinuxContainerExecutor is disabledMajor.Szilard NemethAndras Gyori
YARN-9427TestContainerSchedulerQueuing.testKillOnlyRequiredOpportunisticContainers fails sporadicallyMajorscheduler, testPrabhu JosephAhmed Hussein
HDFS-15135EC : ArrayIndexOutOfBoundsException in BlockRecoveryWorker#RecoveryTaskStriped.Majorerasure-codingSurendra Singh LilhoreRavuri Sushma sree
HDFS-14442Disagreement between HAUtil.getAddressOfActive and RpcInvocationHandler.getConnectionIdMajor.Erik KrogenRavuri Sushma sree
HDFS-15216Wrong Use Case of -showprogress in fsckMajor.Ravuri Sushma sreeRavuri Sushma sree
HDFS-15211EC: File write hangs during close in case of Exception during updatePipelineCritical.Ayush SaxenaAyush Saxena
HDFS-15208Suppress bogus AbstractWadlGeneratorGrammarGenerator in KMS stderr in hdfsTrivial.Wei-Chiu ChuangWei-Chiu Chuang
HDFS-15223FSCK fails if one namenode is not availableMajor.Ayush SaxenaAyush Saxena
HDFS-15232Fix libhdfspp test failures with GCC 7Majornative, testAkira AjisakaAkira Ajisaka
HDFS-15219DFS Client will stuck when ResponseProcessor.run throw ErrorMajorhdfs-clientzhengchenyuzhengchenyu
HDFS-15191EOF when reading legacy buffer in BlockTokenIdentifierMajorhdfsSteven RandSteven Rand
YARN-10202Fix documentation about NodeAttributes.MinordocumentationSen ZhaoSen Zhao
HADOOP-16949pylint fails in the build environmentCriticalbuildAkira AjisakaAkira Ajisaka
HADOOP-14836Upgrade maven-clean-plugin to 3.1.0MajorbuildAllen WittenauerAkira Ajisaka
YARN-10207CLOSE_WAIT socket connection leaks during rendering of (corrupted) aggregated logs on the JobHistoryServer Web UIMajoryarnSiddharth AhujaSiddharth Ahuja
HDFS-12862CacheDirective becomes invalid when NN restart or failoverMajorcaching, hdfsWang XLWang XL
MAPREDUCE-7272TaskAttemptListenerImpl excessive log messagesMajor.Ahmed HusseinAhmed Hussein
HDFS-15283Cache pool MAXTTL is not persisted and restored on cluster restartMajornamenodeStephen O'DonnellStephen O'Donnell
HADOOP-16944Use Yetus 0.12.0 in GitHub PRMajorbuildAkira AjisakaAkira Ajisaka
HDFS-15276Concat on INodeRefernce fails with illegal state exceptionCritical.Hemanth BoyinaHemanth Boyina
YARN-10223Duplicate jersey-test-framework-core dependency in yarn-server-commonMinorbuildAkira AjisakaAkira Ajisaka
HDFS-15281ZKFC ignores dfs.namenode.rpc-bind-host and uses dfs.namenode.rpc-address to bind to host addressMajorha, namenodeDhiraj HegdeDhiraj Hegde
HDFS-15297TestNNHandlesBlockReportPerStorage::blockReport_02 fails intermittently in trunkMajordatanode, testMingliang LiuAyush Saxena
HADOOP-17014Upgrade jackson-databind to 2.9.10.4Blocker.Siyao MengSiyao Meng
YARN-9848revert YARN-4946Blockerlog-aggregation, resourcemanagerSteven RandSteven Rand
HDFS-15286Concat on a same files deleting the fileCritical.Hemanth BoyinaHemanth Boyina
YARN-10256Refactor TestContainerSchedulerQueuing.testContainerUpdateExecTypeGuaranteedToOpportunisticMajor.Ahmed HusseinAhmed Hussein
HDFS-15270Account for *env == NULL in hdfsThreadDestructorMajor.Babneet SinghBabneet Singh
YARN-8959TestContainerResizing fails randomlyMinor.Bibin ChundattAhmed Hussein
HDFS-15323StandbyNode fails transition to active due to insufficient transaction tailingMajornamenode, qjmKonstantin ShvachkoKonstantin Shvachko
HADOOP-17025Fix invalid metastore configuration in S3GuardTool testsMinorfs/s3, testMasatake IwasakiMasatake Iwasaki
HDFS-15339TestHDFSCLI fails for user names with the dot/dash characterMajortestYan XiaoleYan Xiaole
HDFS-15250Setting `dfs.client.use.datanode.hostname` to true can crash the system because of unhandled UnresolvedAddressExceptionMajor.CtestCtest
HDFS-14367EC: Parameter maxPoolSize in striped reconstruct thread pool isn't affecting number of threadsMajorecGuo LeiGuo Lei
HADOOP-15565ViewFileSystem.close doesn't close child filesystems and causes FileSystem objects leak.Major.JinglunJinglun
YARN-9444YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resourceMinorapiGergely PollakGergely Pollak
HADOOP-17044Revert "HADOOP-8143. Change distcp to have -pb on by default"Majortools/distcpSteve LoughranSteve Loughran
HDFS-15293Relax the condition for accepting a fsimage when receiving a checkpointCriticalnamenodeChen LiangChen Liang
HADOOP-17024ListStatus on ViewFS root (ls "/") should list the linkFallBack root (configured target root).Majorfs, viewfsUma Maheswara Rao GAbhishek Das
HADOOP-17040Fix intermittent failure of ITestBlockingThreadPoolExecutorServiceMinorfs/s3, testMasatake IwasakiMasatake Iwasaki
HDFS-15363BlockPlacementPolicyWithNodeGroup should validate if it is initialized by NetworkTopologyWithNodeGroupMajor.Hemanth BoyinaHemanth Boyina
MAPREDUCE-7278Speculative execution behavior is observed even when mapreduce.map.speculative and mapreduce.reduce.speculative are falseMajortaskTarun ParimiTarun Parimi
HADOOP-7002Wrong description of copyFromLocal and copyToLocal in documentationMinor.Jingguo YaoAndras Bokor
HADOOP-17052NetUtils.connect() throws unchecked exception (UnresolvedAddressException) causing clients to abortMajornetDhiraj HegdeDhiraj Hegde
HADOOP-17062Fix shelldocs path in JenkinsfileMajorbuildAkira AjisakaAkira Ajisaka
HADOOP-17056shelldoc fails in hadoop-commonMajorbuildAkira AjisakaAkira Ajisaka
YARN-10286PendingContainers bugs in the scheduler outputsCritical.Adam AntalAndras Gyori
HDFS-15396Fix TestViewFileSystemOverloadSchemeHdfsFileSystemContract#testListStatusRootDirMajor.Ayush SaxenaAyush Saxena
HDFS-15386ReplicaNotFoundException keeps happening in DN after removing multiple DN's data directoriesMajor.Toshihiro SuzukiToshihiro Suzuki
YARN-10300appMasterHost not set in RM ApplicationSummary when AM fails before first heartbeatMajor.Eric BadgerEric Badger
HADOOP-17059ArrayIndexOfboundsException in ViewFileSystem#listStatusMajorviewfsHemanth BoyinaHemanth Boyina
YARN-10296Make ContainerPBImpl#getId/setId synchronizedMinor.Benjamin TekeBenjamin Teke
YARN-10295CapacityScheduler NPE can cause apps to get stuck without resourcesMajorcapacityschedulerBenjamin TekeBenjamin Teke
HADOOP-17060listStatus and getFileStatus behave inconsistent in the case of ViewFs implementation for isDirectoryMajorviewfsSrinivasu MajetiUma Maheswara Rao G
YARN-10312Add support for yarn logs -logFile to retain backward compatibilityMajorclientJim BrennanJim Brennan
HDFS-15403NPE in FileIoProvider#transferToSocketFullyMajor.Hemanth BoyinaHemanth Boyina
HADOOP-17029ViewFS does not return correct user/group and ACLMajorfs, viewfsAbhishek DasAbhishek Das
HDFS-15421IBR leak causes standby NN to be stuck in safe modeBlockernamenodeKihwal LeeAkira Ajisaka
YARN-9903Support reservations continue looking for Node LabelsMajor.Tarun ParimiJim Brennan
HADOOP-17032Handle an internal dir in viewfs having multiple children mount points pointing to different filesystemsMajorfs, viewfsAbhishek DasAbhishek Das
HDFS-15446CreateSnapshotOp fails during edit log loading for /.reserved/raw/path with error java.io.FileNotFoundException: Directory does not exist: /.reserved/raw/pathMajorhdfsSrinivasu MajetiStephen O'Donnell
HADOOP-17081MetricsSystem doesn't start the sink adapters on restartMinormetricsMadhusoodanMadhusoodan
HDFS-15451Restarting name node stuck in safe mode when using provided storageMajornamenodeshanyu zhaoshanyu zhao
HADOOP-17120Fix failure of docker image creation due to pip2 install errorMajor.Masatake IwasakiMasatake Iwasaki
YARN-10347Fix double locking in CapacityScheduler#reinitialize in branch-3.1Criticalcapacity schedulerMasatake IwasakiMasatake Iwasaki
YARN-10348Allow RM to always cancel tokens after app completesMajoryarnJim BrennanJim Brennan
MAPREDUCE-7284TestCombineFileInputFormat#testMissingBlocks failsMajortestAkira AjisakaAkira Ajisaka
HDFS-14498LeaseManager can loop forever on the file for which create has failedMajornamenodeSergey ShelukhinStephen O'Donnell
HADOOP-17130Configuration.getValByRegex() shouldn't update the results while fetching.MajorcommonMukund ThakurMukund Thakur
HADOOP-17119Jetty upgrade to 9.4.x causes MR app fail with IOExceptionMajor.Bilwa S TBilwa S T
YARN-4771Some containers can be skipped during log aggregation after NM restartMajornodemanagerJason Darrell LoweJim Brennan
MAPREDUCE-7051Fix typo in MultipleOutputFormatTrivial.ywheelywheel
HDFS-15313Ensure inodes in active filesystem are not deleted during snapshot deleteMajorsnapshotsShashikant BanerjeeShashikant Banerjee
HDFS-14950missing libhdfspp libs in dist-packageMajorbuild, libhdfs++Yuan ZhouYuan Zhou
HADOOP-17184Add --mvn-custom-repos parameter to yetus callsMajorbuildMingliang LiuMingliang Liu
HDFS-15499Clean up httpfs/pom.xml to remove aws-java-sdk-s3 exclusionMajorhttpfsMingliang LiuMingliang Liu
HADOOP-17164UGI loginUserFromKeytab doesn't set the last login timeMajorsecuritySandeep GuggilamSandeep Guggilam
YARN-4575ApplicationResourceUsageReport should return ALL reserved resourceMajor.Bibin ChundattBibin Chundatt
HADOOP-17196Fix C/C++ standard warningsMajorbuildGautham BanasandraGautham Banasandra
HADOOP-17204Fix typo in Hadoop KMS documentTrivialdocumentation, kmsAkira AjisakaXieming Li
HADOOP-17209Erasure Coding: Native library memory leakMajornativeSean ChowSean Chow
HADOOP-16925MetricsConfig incorrectly loads the configuration whose value is String list in the properties fileMajormetricsJiayi LiuJiayi Liu
HDFS-14852Removing from LowRedundancyBlocks does not remove the block from all queuesMajornamenodeHui FeiHui Fei
HDFS-15290NPE in HttpServer during NameNode startupMajornamenodeKonstantin ShvachkoSimbarashe Dzinamarira
YARN-10430Log improvements in NodeStatusUpdaterImplMinornodemanagerBilwa S TBilwa S T
MAPREDUCE-7294Only application master should upload resource to Yarn Shared CacheMajormrv2zhenzhao wangzhenzhao wang
MAPREDUCE-7289Fix wrong comment in LongLong.javaTrivialdocumentation, examplesAkira AjisakaWanqiang Ji
YARN-9809NMs should supply a health status when registering with RMMajor.Eric BadgerEric Badger
YARN-10393MR job live lock caused by completed state container leak in heartbeat between node manager and RMMajornodemanager, yarnzhenzhao wangJim Brennan
YARN-10455TestNMProxy.testNMProxyRPCRetry is not consistentMajor.Ahmed HusseinAhmed Hussein
HADOOP-17223update org.apache.httpcomponents:httpclient to 4.5.13 and httpcore to 4.4.13Blocker.Pranav BhedaPranav Bheda
HDFS-15628HttpFS server throws NPE if a file is a symlinkMajorfs, httpfsAhmed HusseinAhmed Hussein
HDFS-15627Audit log deletes before collecting blocksMajorlogging, namenodeAhmed HusseinAhmed Hussein
HADOOP-17309Javadoc warnings and errors are ignored in the precommit jobsMajorbuild, documentationAkira AjisakaAkira Ajisaka
HDFS-15639[JDK 11] Fix Javadoc errors in hadoop-hdfs-clientMajor.Takanobu AsanumaTakanobu Asanuma
HDFS-15618Improve datanode shutdown latencyMajordatanodeAhmed HusseinAhmed Hussein
HDFS-15622Deleted blocks linger in the replications queueMajorhdfsAhmed HusseinAhmed Hussein
HDFS-15641DataNode could meet deadlock if invoke refreshNameNodeCritical.Hongbing WangHongbing Wang
HDFS-15644Failed volumes can cause DNs to stop block reportingMajorblock placement, datanodeAhmed HusseinAhmed Hussein
YARN-10467ContainerIdPBImpl objects can be leaked in RMNodeImpl.completedContainersMajorresourcemanagerHaibo ChenHaibo Chen
HADOOP-17329mvn site commands fails due to MetricsSystemImpl changesMajor.Xiaoqiao HeXiaoqiao He
YARN-10472Backport YARN-10314 to branch-3.2BlockeryarnSiyao MengSiyao Meng
HADOOP-17340TestLdapGroupsMapping failing -string mismatch in exception validationMajortestSteve LoughranSteve Loughran
HADOOP-17352Update PATCH_NAMING_RULE in the personality fileMinorbuildAkira AjisakaAkira Ajisaka
HADOOP-17096ZStandardCompressor throws java.lang.InternalError: Error (generic)MajorioStephen Jung (Stripe)Stephen Jung (Stripe)
HADOOP-17358Improve excessive reloading of ConfigurationsMajorconfAhmed HusseinAhmed Hussein
HDFS-15538Fix the documentation for dfs.namenode.replication.max-streams in hdfs-default.xmlMajor.Xieming LiXieming Li
HADOOP-17362Doing hadoop ls on Har file triggers too many RPC callsMajorfsAhmed HusseinAhmed Hussein
YARN-10485TimelineConnector swallows InterruptedExceptionMajor.Ahmed HusseinAhmed Hussein
HADOOP-17360Log the remote address for authentication successMinoripcAhmed HusseinAhmed Hussein
YARN-10396Max applications calculation per queue disregards queue level settings in absolute modeMajorcapacity schedulerBenjamin TekeBenjamin Teke
HADOOP-17346Fair call queue is defeated by abusive service principalsMajorcommon, ipcAhmed HusseinAhmed Hussein
YARN-10470When building new web ui with root user, the bower install should support it.Majorbuild, yarn-ui-v2zhuqizhuqi
HADOOP-16080hadoop-aws does not work with hadoop-client-apiMajorfs/s3Keith TurnerChao Sun
HDFS-15707NNTop counts don't add up as expectedMajorhdfs, metrics, namenodeAhmed HusseinAhmed Hussein
HDFS-15709EC: Socket file descriptor leak in StripedBlockChecksumReconstructorMajordatanode, ec, erasure-codingYushi HayasakaYushi Hayasaka
HDFS-15240Erasure Coding: dirty buffer causes reconstruction block errorBlockerdatanode, erasure-codingHuangTaoHuangTao

TESTS:

JIRASummaryPriorityComponentReporterContributor
YARN-10072TestCSAllocateCustomResource failuresMajoryarnJim BrennanJim Brennan
YARN-10161TestRouterWebServicesREST is corrupting STDOUTMinoryarnJim BrennanJim Brennan
HADOOP-14206TestSFTPFileSystem#testFileExists failure: Invalid encoding for signatureMajorfs, testJohn ZhugeJim Brennan
MAPREDUCE-7288Fix TestLongLong#testRightShiftMinor.Wanqiang JiWanqiang Ji
HDFS-15514Remove useless dfs.webhdfs.enabledMinortestHui FeiHui Fei
HADOOP-17205Move personality file from Yetus to Hadoop repositoryMajortest, yetusChao SunChao Sun

SUB-TASKS:

JIRASummaryPriorityComponentReporterContributor
HADOOP-15775[JDK9] Add missing javax.activation-api dependencyCriticaltestAkira AjisakaAkira Ajisaka
HDFS-14096[SPS] : Add Support for Storage Policy Satisfier in ViewFsMajorfederationAyush SaxenaAyush Saxena
HADOOP-15787[JDK11] TestIPC.testRTEDuringConnectionSetup failsMajor.Akira AjisakaZsolt Venczel
HDFS-14262[SBN read] Unclear Log.WARN message in GlobalStateIdContextMajorhdfsShwetaShweta
YARN-7243Moving logging APIs over to slf4j in hadoop-yarn-server-resourcemanagerMajor.Yeliang CangPrabhu Joseph
HDFS-13404RBF: TestRouterWebHDFSContractAppend.testRenameFileBeingAppended failsMajortestTakanobu AsanumaTakanobu Asanuma
HADOOP-16117Update AWS SDK to 1.11.563Majorbuild, fs/s3Steve LoughranSteve Loughran
HDFS-14590[SBN Read] Add the document link to the top pageMajordocumentationTakanobu AsanumaTakanobu Asanuma
YARN-9791Queue Mutation API does not allow to remove a configMajorcapacity schedulerPrabhu JosephPrabhu Joseph
HDFS-14822[SBN read] Revisit GlobalStateIdContext locking when getting server state idMajorhdfsChen LiangChen Liang
HDFS-14785[SBN read] Change client logging to be less aggressiveMajorhdfsChen LiangChen Liang
YARN-9864Format CS Configuration present in Configuration StoreMajorcapacity schedulerPrabhu JosephPrabhu Joseph
YARN-9801SchedConfCli does not work with https modeMajor.Prabhu JosephPrabhu Joseph
HDFS-14858[SBN read] Allow configurably enable/disable AlignmentContext on NameNodeMajorhdfsChen LiangChen Liang
HDFS-12979StandbyNode should upload FsImage to ObserverNode after checkpointing.MajorhdfsKonstantin ShvachkoChen Liang
YARN-9873Mutation API Config Change need to update Version NumberMajor.Prabhu JosephPrabhu Joseph
HDFS-14162Balancer should work with ObserverNodeMajor.Konstantin ShvachkoErik Krogen
YARN-9773Add QueueMetrics for Custom ResourcesMajor.Manikandan RManikandan R
HADOOP-16598Backport "HADOOP-16558 [COMMON+HDFS] use protobuf-maven-plugin to generate protobuf classes" to all active branchesMajorcommonDuo ZhangDuo Zhang
YARN-9950Unset Ordering Policy of Leaf/Parent queue converted from Parent/Leaf queue respectivelyMajorcapacity schedulerPrabhu JosephPrabhu Joseph
YARN-9900Revert to previous state when Invalid Config is applied and Refresh Support in SchedulerConfig FormatMajorcapacity schedulerPrabhu JosephPrabhu Joseph
HADOOP-16610Upgrade to yetus 0.11.1 and use emoji vote on github pre commitMajorbuildDuo ZhangDuo Zhang
YARN-9909Offline format of YarnConfigurationStoreMajorcapacity schedulerPrabhu JosephPrabhu Joseph
YARN-9836General usability improvements in showSimulationTrace.htmlMinorscheduler-load-simulatorAdam AntalAdam Antal
HADOOP-16612Track Azure Blob File System client-perceived latencyMajorfs/azure, hdfs-clientJeetesh MangwaniJeetesh Mangwani
HADOOP-16758Refine testing.md to tell user better how to use auth-keys.xmlMinorfs/s3Mingliang LiuMingliang Liu
HADOOP-16609Add Jenkinsfile for all active branchesMajorbuildDuo ZhangAkira Ajisaka
HADOOP-16785Improve wasb and abfs resilience on double close() callsMajorfs/azureSteve LoughranSteve Loughran
YARN-10026Pull out common code pieces from ATS v1.5 and v2MajorATSv2, yarnAdam AntalAdam Antal
YARN-10028Integrate the new abstract log servlet to the JobHistory serverMajoryarnAdam AntalAdam Antal
YARN-10083Provide utility to ask whether an application is in final statusMinor.Adam AntalAdam Antal
YARN-10109Allow stop and convert from leaf to parent queue in a single Mutation API callMajorcapacity schedulerPrabhu JosephPrabhu Joseph
YARN-10101Support listing of aggregated logs for containers belonging to an application attemptMajorlog-aggregation, yarnAdam AntalAdam Antal
YARN-10022Create RM Rest API to validate a CapacityScheduler ConfigurationMajor.Kinga MartonKinga Marton
HDFS-15173RBF: Delete repeated configuration 'dfs.federation.router.metrics.enable'Minordocumentation, rbfpanlijiepanlijie
YARN-10139ValidateAndGetSchedulerConfiguration API fails when cluster max allocation > default 8GBMajor.Prabhu JosephPrabhu Joseph
HDFS-14731[FGL] Remove redundant locking on NameNode.MajornamenodeKonstantin ShvachkoKonstantin Shvachko
YARN-10194YARN RMWebServices /scheduler-conf/validate leaks ZK ConnectionsBlockercapacityschedulerAkhil PBPrabhu Joseph
HDFS-14353Erasure Coding: metrics xmitsInProgress become to negative.Majordatanode, erasure-codingBaolong MaoBaolong Mao
HDFS-15305Extend ViewFS and provide ViewFSOverloadScheme implementation with scheme configurable.Majorfs, hadoop-client, hdfs-client, viewfsUma Maheswara Rao GUma Maheswara Rao G
HDFS-15306Make mount-table to read from central place ( Let's say from HDFS)Majorconfiguration, hadoop-clientUma Maheswara Rao GUma Maheswara Rao G
HADOOP-16756distcp -update to S3A; abfs, etc always overwrites due to block size mismatchMajorfs/s3, tools/distcpDaisuke KobayashiSteve Loughran
HDFS-15322Make NflyFS to work when ViewFsOverloadScheme's scheme and target uris schemes are same.Majorfs, nflyFs, viewfs, viewfsOverloadSchemeUma Maheswara Rao GUma Maheswara Rao G
HDFS-15321Make DFSAdmin tool to work with ViewFSOverloadSchemeMajordfsadmin, fs, viewfsUma Maheswara Rao GUma Maheswara Rao G
HDFS-15330Document the ViewFSOverloadScheme details in ViewFS guideMajorviewfs, viewfsOverloadSchemeUma Maheswara Rao GUma Maheswara Rao G
HDFS-15389DFSAdmin should close filesystem and dfsadmin -setBalancerBandwidth should work with ViewFSOverloadSchemeMajordfsadmin, viewfsOverloadSchemeAyush SaxenaAyush Saxena
HDFS-15394Add all available fs.viewfs.overload.scheme.target.<scheme>.impl classes in core-default.xml bydefault.Majorconfiguration, viewfs, viewfsOverloadSchemeUma Maheswara Rao GUma Maheswara Rao G
HDFS-15387FSUsage$DF should consider ViewFSOverloadScheme in processPathMinorviewfsUma Maheswara Rao GUma Maheswara Rao G
HDFS-15418ViewFileSystemOverloadScheme should represent mount links as non symlinksMajor.Uma Maheswara Rao GUma Maheswara Rao G
HDFS-15427Merged ListStatus with Fallback target filesystem and InternalDirViewFS.MajorviewfsUma Maheswara Rao GUma Maheswara Rao G
HDFS-15429mkdirs should work when parent dir is internalDir and fallback configured.Major.Uma Maheswara Rao GUma Maheswara Rao G
HDFS-15436Default mount table name used by ViewFileSystem should be configurableMajorviewfs, viewfsOverloadSchemeVirajith JalapartiVirajith Jalaparti
HDFS-15450Fix NN trash emptier to work if ViewFSOveroadScheme enabledMajornamenode, viewfsOverloadSchemeUma Maheswara Rao GUma Maheswara Rao G
HDFS-15462Add fs.viewfs.overload.scheme.target.ofs.impl to core-default.xmlMajorconfiguration, viewfs, viewfsOverloadSchemeSiyao MengSiyao Meng
HDFS-15464ViewFsOverloadScheme should work when -fs option pointing to remote cluster without mount linksMajorviewfsOverloadSchemeUma Maheswara Rao GUma Maheswara Rao G
HADOOP-17101Replace Guava Function with Java8+ FunctionMajor.Ahmed HusseinAhmed Hussein
HADOOP-17099Replace Guava Predicate with Java8+ PredicateMinor.Ahmed HusseinAhmed Hussein
HDFS-15478When Empty mount points, we are assigning fallback link to self. But it should not use full URI for target fs.Major.Uma Maheswara Rao GUma Maheswara Rao G
HADOOP-17100Replace Guava Supplier with Java8+ Supplier in HadoopMajor.Ahmed HusseinAhmed Hussein
HDFS-15515mkdirs on fallback should throw IOE out instead of suppressing and returning falseMajor.Uma Maheswara Rao GUma Maheswara Rao G
HADOOP-17199Backport HADOOP-13230 list/getFileStatus changes for preserved directory markersMajorfs/s3Steve LoughranSteve Loughran
HDFS-8631WebHDFS : Support setQuotaMajor.nijelChao Sun
YARN-10332RESOURCE_UPDATE event was repeatedly registered in DECOMMISSIONING stateMinorresourcemanageryehuanhuanyehuanhuan
HDFS-15459TestBlockTokenWithDFSStriped fails intermittentlyMajorhdfsAhmed HusseinAhmed Hussein
HDFS-15461TestDFSClientRetries#testGetFileChecksum fails intermittentlyMajordfsclient, testAhmed HusseinAhmed Hussein
HDFS-9776TestHAAppend#testMultipleAppendsDuringCatchupTailing is flakyMajor.Vinayakumar BAhmed Hussein
HADOOP-17330Backport HADOOP-16005-"NativeAzureFileSystem does not support setXAttr" to branch-3.2Majorfs/azureSally ZuoSally Zuo
HDFS-15643EC: Fix checksum computation in case of native encodersBlocker.Ahmed HusseinAyush Saxena
HADOOP-17343Upgrade aws-java-sdk to 1.11.901Minorbuild, fs/s3Dongjoon HyunSteve Loughran
HADOOP-17325WASB: Test failuresMajorfs/azure, testSneha VijayarajanSteve Loughran
HDFS-15708TestURLConnectionFactory fails by NoClassDefFoundError in branch-3.3 and branch-3.2BlockertestAkira AjisakaChao Sun

OTHER:

JIRASummaryPriorityComponentReporterContributor
HDFS-14394Add -std=c99 / -std=gnu99 to libhdfs compile flagsMajorhdfs-client, libhdfs, nativeSahil TakiarSahil Takiar
HADOOP-16365Upgrade jackson-databind to 2.9.9MajorbuildShwetaShweta
HADOOP-16491Upgrade jetty version to 9.3.27Major.Hrishikesh GadreHrishikesh Gadre
HADOOP-16542Update commons-beanutils version to 1.9.4Major.Wei-Chiu ChuangKevin Su
YARN-9730Support forcing configured partitions to be exclusive based on app node labelMajor.Jonathan HungJonathan Hung
HADOOP-16675Upgrade jackson-databind to 2.9.10.1BlockersecurityWei-Chiu ChuangLisheng Sun
HDFS-14959[SBNN read] access time should be turned offMajordocumentationWei-Chiu ChuangChao Sun
HADOOP-16784Update the year to 2020Major.Ayush SaxenaAyush Saxena
HADOOP-16803Upgrade jackson-databind to 2.9.10.2BlockersecurityAkira AjisakaMasatake Iwasaki
HADOOP-16871Upgrade Netty version to 4.1.45.Final to handle CVE-2019-20444,CVE-2019-16869Major.Aray Chenchu SukeshAray Chenchu Sukesh
HADOOP-16647Support OpenSSL 1.1.1 LTSCriticalsecurityWei-Chiu ChuangRakesh Radhakrishnan
HADOOP-16982Update Netty to 4.1.48.FinalBlocker.Wei-Chiu ChuangLisheng Sun
HADOOP-16990Update MockserverMajor.Wei-Chiu ChuangAttila Doroszlai
YARN-10540Node page is broken in YARN UI1 and UI2 including RMWebService api for nodesCriticalwebappSunil GJim Brennan
HADOOP-17445Update the year to 2021Major.Xiaoqiao HeXiaoqiao He