.. _deploy-cli:

CLI

MLC Chat CLI is the command line tool to run MLC-compiled LLMs out of the box interactively.

.. contents:: Table of Contents :local: :depth: 2

Install MLC-LLM Package

Chat CLI is a part of the MLC-LLM package. To use the chat CLI, first install MLC LLM by following the instructions :ref:here <install-mlc-packages>. Once you have install the MLC-LLM package, you can run the following command to check if the installation was successful:

.. code:: bash

mlc_llm chat --help

You should see serve help message if the installation was successful.

Quick Start

This section provides a quick start guide to work with MLC-LLM chat CLI. To launch the CLI session, run the following command:

.. code:: bash

mlc_llm chat MODEL [--model-lib PATH-TO-MODEL-LIB]

where MODEL is the model folder after compiling with :ref:MLC-LLM build process <compile-model-libraries>. Information about other arguments can be found in the next section.

Once the chat CLI is ready, you can enter the prompt to interact with the model.

.. code::

You can use the following special commands: /help print the special commands /exit quit the cli /stats print out stats of last request (token/sec) /metrics print out full engine metrics /reset restart a fresh chat /set [overrides] override settings in the generation config. For example, /set temperature=0.5;top_p=0.8;seed=23;max_tokens=100;stop=str1,str2 Note: Separate stop words in the stop option with commas (,). Multi-line input: Use escape+enter to start a new line.

What's the meaning of life? The meaning of life is a philosophical and metaphysical question related to the purpose or significance of life or existence in general...

Run CLI with Multi-GPU

If you want to enable tensor parallelism to run LLMs on multiple GPUs, please specify argument --overrides "tensor_parallel_shards=$NGPU". For example,

.. code:: shell

mlc_llm chat HF://mlc-ai/Llama-3-8B-Instruct-q4f16_1-MLC --overrides "tensor_parallel_shards=2"

The mlc_llm chat Command

We provide the list of chat CLI interface for reference.

.. code:: bash

mlc_llm chat MODEL [--model-lib PATH-TO-MODEL-LIB] [--device DEVICE] [--overrides OVERRIDES]

MODEL The model folder after compiling with MLC-LLM build process. The parameter can either be the model name with its quantization scheme (e.g. Llama-2-7b-chat-hf-q4f16_1), or a full path to the model folder. In the former case, we will use the provided name to search for the model folder over possible paths.

--model-lib A field to specify the full path to the model library file to use (e.g. a .so file). --device The description of the device to run on. User should provide a string in the form of device_name:device_id or device_name, where device_name is one of cuda, metal, vulkan, rocm, opencl, auto (automatically detect the local device), and device_id is the device id to run on. The default value is auto, with the device id set to 0 for default. --overrides Model configuration override. Supports overriding context_window_size, prefill_chunk_size, sliding_window_size, attention_sink_size, and tensor_parallel_shards. The overrides could be explicitly specified via details knobs, e.g. --overrides context_window_size=1024;prefill_chunk_size=128.