AudioClassifier

public final class AudioClassifier

Performs classification on audio waveforms.

The API expects a TFLite model with TFLite Model Metadata..

The API supports models with one audio input tensor and one classification output tensor. To be more specific, here are the requirements.

• Input audio tensor (kTfLiteFloat32)

  • input audio buffer of size [batch x samples].
  • batch inference is not supported (batch is required to be 1).

• Output score tensor (kTfLiteFloat32)

• with N classes of either 2 or 4 dimensions, such as [1 x N] or [1 x 1 x 1 x N]

• the label file is required to be packed to the metadata. See the example of creating metadata for an image classifier. If no label files are packed, it will use index as label in the result.

See an example of such model, and a CLI demo tool for easily trying out this API.

Nested Classes

| class | AudioClassifier.AudioClassifierOptions | Options for setting up an AudioClassifier. |

Public Methods

| List<Classifications> | classify(TensorAudio tensor) Performs actual classification on the provided audio tensor.

| | AudioRecord | createAudioRecord() Creates an AudioRecord instance to record audio stream.

| | static AudioClassifier | createFromBuffer(ByteBuffer modelBuffer) Creates an AudioClassifier instance with a model buffer and the default AudioClassifier.AudioClassifierOptions.

| | static AudioClassifier | createFromBufferAndOptions(ByteBuffer modelBuffer, AudioClassifier.AudioClassifierOptions options) Creates an AudioClassifier instance with a model buffer and AudioClassifier.AudioClassifierOptions.

| | static AudioClassifier | createFromFile(Context context, String modelPath) Creates an AudioClassifier instance from the default AudioClassifier.AudioClassifierOptions.

| | static AudioClassifier | createFromFile(File modelFile) Creates an AudioClassifier instance from the default AudioClassifier.AudioClassifierOptions.

| | static AudioClassifier | createFromFileAndOptions(Context context, String modelPath, AudioClassifier.AudioClassifierOptions options) Creates an AudioClassifier instance from AudioClassifier.AudioClassifierOptions.

| | static AudioClassifier | createFromFileAndOptions(File modelFile, AudioClassifier.AudioClassifierOptions options) Creates an AudioClassifier instance.

| | TensorAudio | createInputTensorAudio() Creates a TensorAudio instance to store input audio samples.

| | long | getRequiredInputBufferSize() Returns the required input buffer size in number of float elements.

| | TensorAudio.TensorAudioFormat | getRequiredTensorAudioFormat() Returns the TensorAudio.TensorAudioFormat required by the model.

|

Inherited Methods

From class org.tensorflow.lite.task.core.BaseTaskApi

| synchronized void | close() Release the memory allocated from C++ and deregister the library from the static holder.

| | long | getNativeHandle() | | boolean | isClosed() |

From class java.lang.Object

| boolean | equals(Object arg0) | | final Class<?> | getClass() | | int | hashCode() | | final void | notify() | | final void | notifyAll() | | String | toString() | | final void | wait(long arg0, int arg1) | | final void | wait(long arg0) | | final void | wait() |

From interface java.io.Closeable

| abstract void | close() |

From interface java.lang.AutoCloseable

| abstract void | close() |

Public Methods

public List<Classifications> classify (TensorAudio tensor)

Performs actual classification on the provided audio tensor.

Parameters

| tensor | a TensorAudio containing the input audio clip in float with values between [-1, 1). The tensor argument should have the same flat size as the TFLite model's input tensor. It's recommended to create tensor using createInputTensorAudio method. |

Throws

| IllegalArgumentException | if an argument is invalid | | IllegalStateException | if error occurs when classifying the audio clip from the native code |

public AudioRecord createAudioRecord ()

Creates an AudioRecord instance to record audio stream. The returned AudioRecord instance is initialized and client needs to call AudioRecord.startRecordingnull method to start recording.

Returns

• an AudioRecord instance in AudioRecord.STATE_INITIALIZED

Throws

| IllegalArgumentException | if the model required channel count is unsupported | | IllegalStateException | if AudioRecord instance failed to initialize |

public static AudioClassifier createFromBuffer (ByteBuffer modelBuffer)

Creates an AudioClassifier instance with a model buffer and the default AudioClassifier.AudioClassifierOptions.

Parameters

| modelBuffer | a direct ByteBuffer or a MappedByteBuffer of the classification model |

Throws

| IllegalStateException | if there is an internal error | | RuntimeException | if there is an otherwise unspecified error | | IllegalArgumentException | if the model buffer is not a direct ByteBuffer or a MappedByteBuffer |

public static AudioClassifier createFromBufferAndOptions (ByteBuffer modelBuffer, AudioClassifier.AudioClassifierOptions options)

Creates an AudioClassifier instance with a model buffer and AudioClassifier.AudioClassifierOptions.

Parameters

| modelBuffer | a direct ByteBuffer or a MappedByteBuffer of the classification model | | options | |

Throws

| IllegalStateException | if there is an internal error | | RuntimeException | if there is an otherwise unspecified error | | IllegalArgumentException | if the model buffer is not a direct ByteBuffer or a MappedByteBuffer |

public static AudioClassifier createFromFile (Context context, String modelPath)

Creates an AudioClassifier instance from the default AudioClassifier.AudioClassifierOptions.

Parameters

| context | | | modelPath | path of the classification model with metadata in the assets |

Throws

| IOException | if an I/O error occurs when loading the tflite model | | IllegalArgumentException | if an argument is invalid | | IllegalStateException | if there is an internal error | | RuntimeException | if there is an otherwise unspecified error |

public static AudioClassifier createFromFile (File modelFile)

Creates an AudioClassifier instance from the default AudioClassifier.AudioClassifierOptions.

Parameters

| modelFile | the classification model File instance |

Throws

| IOException | if an I/O error occurs when loading the tflite model | | IllegalArgumentException | if an argument is invalid | | IllegalStateException | if there is an internal error | | RuntimeException | if there is an otherwise unspecified error |

public static AudioClassifier createFromFileAndOptions (Context context, String modelPath, AudioClassifier.AudioClassifierOptions options)

Creates an AudioClassifier instance from AudioClassifier.AudioClassifierOptions.

Parameters

| context | | | modelPath | path of the classification model with metadata in the assets | | options | |

Throws

| IOException | if an I/O error occurs when loading the tflite model | | IllegalArgumentException | if an argument is invalid | | IllegalStateException | if there is an internal error | | RuntimeException | if there is an otherwise unspecified error |

public static AudioClassifier createFromFileAndOptions (File modelFile, AudioClassifier.AudioClassifierOptions options)

Creates an AudioClassifier instance.

Parameters

| modelFile | the classification model File instance | | options | |

Throws

| IOException | if an I/O error occurs when loading the tflite model | | IllegalArgumentException | if an argument is invalid | | IllegalStateException | if there is an internal error | | RuntimeException | if there is an otherwise unspecified error |

public TensorAudio createInputTensorAudio ()

Creates a TensorAudio instance to store input audio samples.

Returns

• a TensorAudio with the same size as model input tensor

Throws

| IllegalArgumentException | if the model is not compatible |

public long getRequiredInputBufferSize ()

Returns the required input buffer size in number of float elements.

public TensorAudio.TensorAudioFormat getRequiredTensorAudioFormat ()

Returns the TensorAudio.TensorAudioFormat required by the model.