Models

geniex infer ai-hub-models/Qwen3-4B

geniex infer <org>/<repo>-GGUF

geniex infer unsloth/Phi-4-mini-instruct-GGUF

1. `GENIEX_HFTOKEN` environment variable
2. `HF_TOKEN` environment variable
3. `~/.cache/huggingface/token` file

**Option A — environment variable (recommended):**

<CodeGroup>
```powershell windows
# Standard HF variable (works with other HF tools too)
$env:HF_TOKEN = "hf_..."

# Or GenieX-specific (takes priority over HF_TOKEN)
$env:GENIEX_HFTOKEN = "hf_..."
```

```bash linux
# Standard HF variable (works with other HF tools too)
export HF_TOKEN="hf_..."

# Or GenieX-specific (takes priority over HF_TOKEN)
export GENIEX_HFTOKEN="hf_..."
```
</CodeGroup>

**Option B — Hugging Face CLI login (persists to disk):**

```bash
pip install huggingface_hub
huggingface-cli login
```

This writes the token to `~/.cache/huggingface/token`, which GenieX reads automatically.

**Self-converted from Hugging Face**

Make sure the [Hugging Face CLI](https://huggingface.co/docs/huggingface_hub/en/guides/cli) is installed, then download a bundle and pull it:

<CodeGroup>
```powershell windows
# Set model name (must match the HF subfolder name exactly)
$model = "llama_v3_2_3b_instruct"

# Download the model from Hugging Face
hf download yichqian/geniex-qairt-models `
  --local-dir $model `
  --include "$model/**"

# Pull using the downloaded folder (absolute path)
geniex pull local/$model --local-path (Resolve-Path "$model\$model").Path

# Run inference
geniex infer local/$model
```

```bash linux
# Set model name (must match the HF subfolder name exactly)
model="llama_v3_2_3b_instruct"

# Download the model from Hugging Face
hf download yichqian/geniex-qairt-models \
  --local-dir "$model" \
  --include "$model/**"

# Pull using the downloaded folder (absolute path)
geniex pull "local/$model" --local-path "$(realpath "$model/$model")"

# Run inference
geniex infer "local/$model"
```
</CodeGroup>

**Already on disk**

Point `--local-path` at the extracted bundle directory (containing `.bin` shards and `metadata.json`):

<CodeGroup>
```powershell windows
geniex pull local/my-bundle --local-path C:\models\my-bundle
geniex infer local/my-bundle
```

```bash linux
geniex pull local/my-bundle --local-path /home/user/models/my-bundle
geniex infer local/my-bundle
```
</CodeGroup>

You can also pull directly from an AI Hub `.zip` archive without extracting first:

<CodeGroup>
```powershell windows
geniex pull local/my-bundle --local-path C:\downloads\model.zip
```

```bash linux
geniex pull local/my-bundle --local-path /home/user/downloads/model.zip
```
</CodeGroup>

<Note>`geniex pull` copies model files into its local cache. After a successful pull, you can safely delete the original download to reclaim disk space. Use `geniex list` to confirm the model is cached.</Note>

```python
from geniex import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    r"C:\<your_path>\Qwen3-4B-Instruct-2507",   # any .bin file or the bundle folder
    model_name="qwen3_4b_instruct_2507",        # matches the Qualcomm AI Hub model id
    device_map="qairt",                         # required for Qualcomm AI Engine Direct
)
```

See the [Python API reference](/en/run/python/api-reference) for all parameters.

```kotlin
ModelManagerWrapper.pullFlow(
    ModelPullInput(
        model_name = "local/qwen3-4b-2507",
        hub        = HubSource.LOCALFS,
        local_path = "/data/local/tmp/Qwen3-4B-Instruct-2507",  // extracted dir or .zip
    )
).collect { /* Progress / Completed / Error */ }
```

Then load with `runtime_id = "qairt"` — identical to the downloaded-model flow:

```kotlin
val paths = ModelManagerWrapper.getPaths("local/qwen3-4b-2507")
    ?: error("Model not imported")

val llm = LlmWrapper.builder()
    .llmCreateInput(
        LlmCreateInput(
            model_name = paths.model_name,
            model_path = paths.model_path,
            config     = ModelConfig(max_tokens = 2048, enable_thinking = false),
            runtime_id  = "qairt",
            compute_unit  = null,   // qairt is NPU-only
        )
    )
    .build()
    .getOrThrow()
```

<Note>
The `model_name` is just the cache key — any `org/repo`-style string works; a `local/...` prefix is a common convention. You don't author a `geniex.json` for a local import — the model manager generates one during import. A QAIRT folder is recognized by `metadata.json` + `.bin` shards (or a `.zip`); a directory of loose `.bin` files with no `metadata.json` won't be detected. When importing models of mixed types, prefer `paths.runtime_id` (authoritative) over hard-coding `runtime_id`.
</Note>

<CodeGroup>
```powershell windows
geniex pull local/my-model --local-path C:\models\my-model
geniex infer local/my-model
```

```bash linux
geniex pull local/my-model --local-path /home/user/models/my-model
geniex infer local/my-model
```
</CodeGroup>

```python
from geniex import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    r"C:\models\my-model\model.gguf",   # path to a local .gguf file
    device_map="auto",                  # auto -> npu for llama_cpp
)
```

See the [Python API reference](/en/run/python/api-reference) for all parameters.

```kotlin
ModelManagerWrapper.pullFlow(
    ModelPullInput(
        model_name = "local/qwen3-0.6b",
        hub        = HubSource.LOCALFS,
        local_path = "/data/local/tmp/qwen3-0.6b",   // dir containing *.gguf
    )
).collect { /* Progress / Completed / Error */ }
```

Then load with `runtime_id = "llama_cpp"` — identical to the downloaded-model flow:

```kotlin
val paths = ModelManagerWrapper.getPaths("local/qwen3-0.6b")
    ?: error("Model not imported")

val llm = LlmWrapper.builder()
    .llmCreateInput(
        LlmCreateInput(
            model_name = paths.model_name,
            model_path = paths.model_path,
            config     = ModelConfig(nCtx = 4096),
            runtime_id  = "llama_cpp",
            compute_unit  = null,   // null → NPU on Snapdragon (recommended)
        )
    )
    .build()
    .getOrThrow()
```

Choose a precision version to download
> Q4_0       [1.2 GiB] (default)
  Q8_0       [2.0 GiB]
  F16        [3.8 GiB]