Quickstart for Android

This page shows you how to build an Android app with TensorFlow Lite to analyze a live camera feed and identify objects. This machine learning use case is called object detection. The example app uses the TensorFlow Lite Task library for vision via Google Play services to enable execution of the object detection machine learning model, which is the recommended approach for building an ML application with TensorFlow Lite.

<aside class="note"> <b>Terms:</b> By accessing or using TensorFlow Lite in Google Play services APIs, you agree to the <a href="./play_services#tos">Terms of Service</a>. Please read and understand all applicable terms and policies before accessing the APIs. </aside>

{: .attempt-right width="250px"}

Setup and run the example

For the first part of this exercise, download the example code from GitHub and run it using Android Studio. The following sections of this document explore the relevant sections of the code example, so you can apply them to your own Android apps. You need the following versions of these tools installed:

Android Studio 4.2 or higher
Android SDK version 21 or higher

Note: This example uses the camera, so you should run it on a physical Android device.

Get the example code

Create a local copy of the example code so you can build and run it.

To clone and setup the example code:

Clone the git repository <pre class="devsite-click-to-copy"> git clone https://github.com/tensorflow/examples.git </pre>
Configure your git instance to use sparse checkout, so you have only the files for the object detection example app: <pre class="devsite-click-to-copy"> cd examples git sparse-checkout init --cone git sparse-checkout set lite/examples/object_detection/android_play_services </pre>

Import and run the project

Use Android Studio to create a project from the downloaded example code, build the project, and run it.

To import and build the example code project:

Start Android Studio.
From the Android Studio Welcome page, choose Import Project, or select File > New > Import Project.
Navigate to the example code directory containing the build.gradle file (...examples/lite/examples/object_detection/android_play_services/build.gradle) and select that directory.

After you select this directory, Android Studio creates a new project and builds it. When the build completes, the Android Studio displays a BUILD SUCCESSFUL message in the Build Output status panel.

To run the project:

From Android Studio, run the project by selecting Run > Run… and MainActivity.
Select an attached Android device with a camera to test the app.

How the example app works

The example app uses pre-trained object detection model, such as mobilenetv1.tflite, in TensorFlow Lite format look for objects in a live video stream from an Android device's camera. The code for this feature is primarily in these files:

ObjectDetectorHelper.kt - Initializes the runtime environment, enables hardware acceleration, and runs the object detection ML model.
CameraFragment.kt - Builds the camera image data stream, prepares data for the model, and displays the object detection results.

Note: This example app uses the TensorFlow Lite Task Library, which provides easy-to-use, task-specific APIs for performing common machine learning operations. For apps with more specific needs and customized ML functions, consider using the Interpreter API.

The next sections show you the key components of these code files, so you can modify an Android app to add this functionality.

Build the app {:#build_app}

The following sections explain the key steps to build your own Android app and run the model shown in the example app. These instructions use the example app shown earlier as a reference point.

Note: To follow along with these instructions and build your own app, create a basic Android project using Android Studio.

Add project dependencies {:#add_dependencies}

In your basic Android app, add the project dependencies for running TensorFlow Lite machine learning models and accessing ML data utility functions. These utility functions convert data such as images into a tensor data format that can be processed by a model.

The example app uses the TensorFlow Lite Task library for vision from Google Play services to enable execution of the object detection machine learning model. The following instructions explain how to add the required library dependencies to your own Android app project.

To add module dependencies:

In the Android app module that uses TensorFlow Lite, update the module's build.gradle file to include the following dependencies. In the example code, this file is located here: ...examples/lite/examples/object_detection/android_play_services/app/build.gradle
```
...
dependencies {
...
    // Tensorflow Lite dependencies
    implementation 'org.tensorflow:tensorflow-lite-task-vision-play-services:0.4.2'
    implementation 'com.google.android.gms:play-services-tflite-gpu:16.1.0'
...
}
```
In Android Studio, sync the project dependencies by selecting: File > Sync Project with Gradle Files.

Initialize Google Play services

When you use Google Play services to run TensorFlow Lite models, you must initialize the service before you can use it. If you want to use hardware acceleration support with the service, such as GPU acceleration, you also enable that support as part of this initialization.

To initialize TensorFlow Lite with Google Play services:

Create a TfLiteInitializationOptions object and modify it to enable GPU support:

val options = TfLiteInitializationOptions.builder()
    .setEnableGpuDelegateSupport(true)
    .build()

Use the TfLiteVision.initialize() method to enable use of the Play services runtime, and set a listener to verify that it loaded successfully:

TfLiteVision.initialize(context, options).addOnSuccessListener {
    objectDetectorListener.onInitialized()
}.addOnFailureListener {
    // Called if the GPU Delegate is not supported on the device
    TfLiteVision.initialize(context).addOnSuccessListener {
        objectDetectorListener.onInitialized()
    }.addOnFailureListener{
        objectDetectorListener.onError("TfLiteVision failed to initialize: "
                + it.message)
    }
}

Initialize the ML model interpreter

Initialize the TensorFlow Lite machine learning model interpreter by loading the model file and setting model parameters. A TensorFlow Lite model includes a .tflite file containing the model code. You should store your models in the src/main/assets directory of your development project, for example:

.../src/main/assets/mobilenetv1.tflite`

Tip: Task library interpreter code automatically looks for models in the src/main/assets directory if you do not specify a file path.

To initialize the model:

Add a .tflite model file to the src/main/assets directory of your development project, such as ssd_mobilenet_v1.
Set the modelName variable to specify your ML model's file name:
```
val modelName = "mobilenetv1.tflite"
```

Set the options for model, such as the prediction threshold and results set size:

val optionsBuilder =
    ObjectDetector.ObjectDetectorOptions.builder()
        .setScoreThreshold(threshold)
        .setMaxResults(maxResults)

Enable GPU acceleration with the options and allow the code to fail gracefully if acceleration is not supported on the device:

try {
    optionsBuilder.useGpu()
} catch(e: Exception) {
    objectDetectorListener.onError("GPU is not supported on this device")
}

Use the settings from this object to construct a TensorFlow Lite ObjectDetector object that contains the model:

objectDetector =
    ObjectDetector.createFromFileAndOptions(
        context, modelName, optionsBuilder.build())

For more information about using hardware acceleration delegates with TensorFlow Lite, see TensorFlow Lite Delegates.

Prepare data for the model

You prepare data for interpretation by the model by transforming existing data such as images into the Tensor data format, so it can be processed by your model. The data in a Tensor must have specific dimensions, or shape, that matches the format of data used to train the model. Depending on the model you use, you may need to transform the data to fit what the model expects. The example app uses an ImageAnalysis object to extract image frames from the camera subsystem.

To prepare data for processing by the model:

Build an ImageAnalysis object to extract images in the required format:

imageAnalyzer =
    ImageAnalysis.Builder()
        .setTargetAspectRatio(AspectRatio.RATIO_4_3)
        .setTargetRotation(fragmentCameraBinding.viewFinder.display.rotation)
        .setBackpressureStrategy(ImageAnalysis.STRATEGY_KEEP_ONLY_LATEST)
        .setOutputImageFormat(OUTPUT_IMAGE_FORMAT_RGBA_8888)
        .build()
        ...

Connect the analyzer to the camera subsystem and create a bitmap buffer to contain the data received from the camera:

        .also {
        it.setAnalyzer(cameraExecutor) { image ->
            if (!::bitmapBuffer.isInitialized) {
                bitmapBuffer = Bitmap.createBitmap(
                    image.width,
                    image.height,
                    Bitmap.Config.ARGB_8888
                )
            }
            detectObjects(image)
        }
    }

Extract the specific image data needed by the model, and pass the image rotation information:

private fun detectObjects(image: ImageProxy) {
    // Copy out RGB bits to the shared bitmap buffer
    image.use { bitmapBuffer.copyPixelsFromBuffer(image.planes[0].buffer) }
    val imageRotation = image.imageInfo.rotationDegrees
    objectDetectorHelper.detect(bitmapBuffer, imageRotation)
}

Complete any final data transformations and add the image data to a TensorImage object, as shown in the ObjectDetectorHelper.detect() method of the example app:

val imageProcessor = ImageProcessor.Builder().add(Rot90Op(-imageRotation / 90)).build()

// Preprocess the image and convert it into a TensorImage for detection.
val tensorImage = imageProcessor.process(TensorImage.fromBitmap(image))

Run predictions

Once you create a TensorImage object with image data in the correct format, you can run the model against that data to produce a prediction, or inference. In the example app, this code is contained in the ObjectDetectorHelper.detect() method.

To run a the model and generate predictions from image data:

Run the prediction by passing the image data to your predict function:
```
val results = objectDetector?.detect(tensorImage)
```

Handle model output

After you run image data against the object detection model, it produces a list of prediction results which your app code must handle by executing additional business logic, displaying results to the user, or taking other actions. The object detection model in the example app produces a list of predictions and bounding boxes for the detected objects. In the example app, the prediction results are passed to a listener object for further processing and display to the user.

To handle model prediction results:

Use a listener pattern to pass results to your app code or user interface objects. The example app uses this pattern to pass detection results from the ObjectDetectorHelper object to the CameraFragment object:
```
objectDetectorListener.onResults( // instance of CameraFragment
    results,
    inferenceTime,
    tensorImage.height,
    tensorImage.width)
```

Act on the results, such as displaying the prediction to the user. The example app draws an overlay on the CameraPreview object to show the result:

override fun onResults(
  results: MutableList<Detection>?,
  inferenceTime: Long,
  imageHeight: Int,
  imageWidth: Int
) {
    activity?.runOnUiThread {
        fragmentCameraBinding.bottomSheetLayout.inferenceTimeVal.text =
            String.format("%d ms", inferenceTime)

        // Pass necessary information to OverlayView for drawing on the canvas
        fragmentCameraBinding.overlay.setResults(
            results ?: LinkedList<Detection>(),
            imageHeight,
            imageWidth
        )

        // Force a redraw
        fragmentCameraBinding.overlay.invalidate()
    }
}

Next steps

Learn more about the Task Library APIs
Learn more about the Interpreter APIs.
Explore the uses of TensorFlow Lite in the examples.
Learn more about using and building machine learning models with TensorFlow Lite in the Models section.
Learn more about implementing machine learning in your mobile application in the TensorFlow Lite Developer Guide.