hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DevelopYourOwnDevicePlugin.md
A device plugin is loaded into the framework when
starting NM. Your plugin class only needs to consider two interfaces provided
by the framework. The DevicePlugin is a must to implement and the
DevicePluginScheduler is optional.
/**
* A must interface for vendor plugin to implement.
* */
public interface DevicePlugin {
/**
* Called first when device plugin framework wants to register.
* @return DeviceRegisterRequest {@link DeviceRegisterRequest}
* @throws Exception
* */
DeviceRegisterRequest getRegisterRequestInfo()
throws Exception;
/**
* Called when update node resource.
* @return a set of {@link Device}, {@link java.util.TreeSet} recommended
* @throws Exception
* */
Set<Device> getDevices() throws Exception;
/**
* Asking how these devices should be prepared/used
* before/when container launch. A plugin can do some tasks in its own or
* define it in DeviceRuntimeSpec to let the framework do it.
* For instance, define {@code VolumeSpec} to let the
* framework to create volume before running container.
*
* @param allocatedDevices A set of allocated {@link Device}.
* @param yarnRuntime Indicate which runtime YARN will use
* Could be {@code RUNTIME_DEFAULT} or {@code RUNTIME_DOCKER}
* in {@link DeviceRuntimeSpec} constants. The default means YARN's
* non-docker container runtime is used. The docker means YARN's
* docker container runtime is used.
* @return a {@link DeviceRuntimeSpec} description about environment,
* {@link VolumeSpec}, {@link MountVolumeSpec}. etc
* @throws Exception
* */
DeviceRuntimeSpec onDevicesAllocated(Set<Device>; allocatedDevices,
YarnRuntimeType yarnRuntime) throws Exception;
/**
* Called after device released.
* @param releasedDevices A set of released devices
* @throws Exception
* */
void onDevicesReleased(Set<Device> releasedDevices)
throws Exception;
}
The above code shows the DevicePlugin interface you need to implement.
Let’s go through the methods that a your plugin should implement.
The getRegisterRequestInfo interface is used for the plugin to advertise a new resource type name and then the ResourceManager. The “DeviceRegisterRequest” returned by the method consists a plugin version and a resource type name like “nvidia.com/gpu”.
The getDevices interface is used to get latest vendor device list in this NM
node.
The resource count pre-defined in node-resources.xml will be overridden.
And it’s recommended that the vendor plugin manages allowed devices reported
to YARN in its own configuration. YARN can only have a blacklist
configuration devices.denied-numbers in container-executor.cfg.
In this method, you may invoke shell command or invoke RESTful/RPC to remote
service to get the devices at your convenience.
Please note that the Device object can describe a fake device. If the major
device number, minor device number and device path is left unset, the
framework won't do isolation for it. This provide feasibility for user to
define a fake device without real hardware.
The onDevicesAllocated interface is invoked to tell the framework how to use these devices.
The NM invoke this interface to let the plugin do some preparation work like create volume before container launch
and give hints on how to expose the devices to container when launch it. The
DeviceRuntimeSpec is the structure of the hints. For instance,
DeviceRuntimeSpec can describes the container launch requirements like
environment variables, device and volume mounts, Docker runtime type.etc.
The onDeviceReleased interface is used for the plugin to do some cleanup work after container finish.
/**
* An optional interface to implement if custom device scheduling is needed.
* If this is not implemented, the device framework will do scheduling.
* */
public interface DevicePluginScheduler {
/**
* Called when allocating devices. The framework will do all device book
* keeping and fail recovery. So this hook could be stateless and only do
* scheduling based on available devices passed in. It could be
* invoked multiple times by the framework. The hint in environment variables
* passed in could be potentially used in making better scheduling decision.
* For instance, GPU scheduling might support different kind of policy. The
* container can set it through environment variables.
* @param availableDevices Devices allowed to be chosen from.
* @param count Number of device to be allocated.
* @param env Environment variables of the container.
* @return A set of {@link Device} allocated
* */
Set<Device> allocateDevices(Set<Device> availableDevices, int count,
Map<String, String> env);
}
The above code shows the DevicePluginScheduler interface that you might
needed if you want to arm the plugin with a more efficient scheduler.
This allocateDevices method is invoked by YARN each time when asking the
plugin's recommendation devices for one container.
This interface is optional because YARN will provide a very basic scheduler.
You can refer to NvidiaGPUPluginForRuntimeV2 plugin for a plugin customized
scheduler. Its scheduler is targeting for Nvidia GPU topology aware
scheduling and can get considerable performance boost for the container.
When developing the plugin, you need to add below dependency property into
your projects's pom.xml. For instance,
<dependencies>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-server-nodemanager</artifactId>
<version>3.3.0</version>
<scope>provided</scope>
</dependency>
</dependencies>
And after this, you can implement the above interfaces based on classes
provided in org.apache.hadoop.yarn.server.nodemanager.api.deviceplugin.
Please note that the plugin project is coupled with the Hadoop YARN NM version.
Once you build your project and package a jar which contains your plugin class and want to give it a try in your Hadoop cluster.
Firstly, put the jar file under a directory in Hadooop classpath. (recommend $HADOOP_COMMAND_HOME/share/hadoop/yarn). Secondly, follow the configurations described in Pluggable Device Framework and restart YARN.