Object Store Configuration

Lance supports object stores such as AWS S3 (and compatible stores), Azure Blob Store, and Google Cloud Storage. Which object store to use is determined by the URI scheme of the dataset path. For example, s3://bucket/path will use S3, az://bucket/path will use Azure, and gs://bucket/path will use GCS.

These object stores take additional configuration objects. There are two ways to specify these configurations: by setting environment variables or by passing them to the storage_options parameter of lance.dataset and lance.write_dataset. So for example, to globally set a higher timeout, you would run in your shell:

export TIMEOUT=60s

If you only want to set the timeout for a single dataset, you can pass it as a storage option:

import lance
ds = lance.dataset("s3://path", storage_options={"timeout": "60s"})

General Configuration

These options apply to all object stores.

S3 Configuration

S3 (and S3-compatible stores) have additional configuration options that configure authorization and S3-specific features (such as server-side encryption).

AWS credentials can be set in the environment variables AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, and AWS_SESSION_TOKEN. Alternatively, they can be passed as parameters to the storage_options parameter:

import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "access_key_id": "my-access-key",
        "secret_access_key": "my-secret-key",
        "session_token": "my-session-token",
    }
)

If you are using AWS SSO, you can specify the AWS_PROFILE environment variable. It cannot be specified in the storage_options parameter.

The following keys can be used as both environment variables or keys in the storage_options parameter:

S3-compatible stores

Lance can also connect to S3-compatible stores, such as MinIO. To do so, you must specify both region and endpoint:

import lance
ds = lance.dataset(
    "s3://bucket/path",
    storage_options={
        "region": "us-east-1",
        "endpoint": "http://minio:9000",
    }
)

This can also be done with the AWS_ENDPOINT and AWS_DEFAULT_REGION environment variables.

S3 Express (Directory Bucket)

Lance supports S3 Express One Zone buckets, a.k.a. S3 directory buckets. S3 Express buckets only support connecting from an EC2 instance within the same region. By default, Lance automatically recognize the --x-s3 suffix of an express bucket, there is no special configuration needed.

In case of an access point or private link that hides the bucket name, you can configure express bucket access explicitly through storage option s3_express.

import lance
ds = lance.dataset(
    "s3://my-bucket--use1-az4--x-s3/path/imagenet.lance",
    storage_options={
        "region": "us-east-1",
        "s3_express": "true",
    }
)

Google Cloud Storage Configuration

GCS credentials are configured by setting the GOOGLE_SERVICE_ACCOUNT environment variable to the path of a JSON file containing the service account credentials. Alternatively, you can pass the path to the JSON file in the storage_options

import lance
ds = lance.dataset(
    "gs://my-bucket/my-dataset",
    storage_options={
        "service_account": "path/to/service-account.json",
    }
)

!!! note

By default, GCS uses HTTP/1 for communication, as opposed to HTTP/2. This improves
maximum throughput significantly. However, if you wish to use HTTP/2 for some reason,
you can set the environment variable `HTTP1_ONLY` to `false`.

The following keys can be used as both environment variables or keys in the storage_options parameter:

Azure Blob Storage Configuration

Azure Blob Storage credentials can be configured by setting the AZURE_STORAGE_ACCOUNT_NAME and AZURE_STORAGE_ACCOUNT_KEY environment variables. Alternatively, you can pass the account name and key in the storage_options parameter:

import lance
ds = lance.dataset(
    "az://my-container/my-dataset",
    storage_options={
        "account_name": "some-account",
        "account_key": "some-key",
    }
)

These keys can be used as both environment variables or keys in the storage_options parameter:

AliCloud Object Storage Service Configuration

OSS credentials can be set in the environment variables OSS_ACCESS_KEY_ID, OSS_ACCESS_KEY_SECRET, OSS_REGION, and OSS_SECURITY_TOKEN. Alternatively, they can be passed as parameters to the storage_options parameter:

import lance
ds = lance.dataset(
    "oss://bucket/path",
    storage_options={
        "oss_region": "oss-region",
        "oss_endpoint": "oss-endpoint",
        "oss_access_key_id": "my-access-key",
        "oss_secret_access_key": "my-secret-key",
        "oss_security_token": "my-session-token",
    }
)