Refget, sequence digests, stores, and BEDbase

Verified on 2026-07-23 against Python gtars==0.9.2, gtars-refget==0.9.1, and gtars-cli==0.9.0.

Digest terminology

Current Python functions:

from gtars.refget import (
    compute_fai,
    digest_fasta,
    digest_sequence,
    load_fasta,
    md5_digest,
    sha512t24u_digest,
)

digest = sha512t24u_digest("ACGT")
assert digest == "aKF498dAxcJAqme6QYQ7EZ07-fiw8Kw2"
assert md5_digest("ACGT") == "f1f8f4bf413b16ad135722aa4591043e"

Gtars returns the 32-character unprefixed sha512t24u value. The preferred GA4GH refget sequence identifier adds SQ.:

SQ.aKF498dAxcJAqme6QYQ7EZ07-fiw8Kw2

MD5 is retained for legacy lookup and is not a collision-resistant provenance hash. Keep an independent SHA-256 for artifact integrity.

digest_sequence(data: bytes, name=None, description=None) uppercases sequence bytes and returns a SequenceRecord containing metadata and data. digest_fasta(path) computes a SequenceCollection; load_fasta(path) loads sequence data into memory. compute_fai(path) is for uncompressed FASTA.

GA4GH refget normalizes sequence content before computing its sequence identifier. Do not hash raw FASTA bytes and call that a sequence digest: headers, line wrapping, compression, and sequence normalization are different layers.

Current Python store class

The runtime class is RefgetStore, not the older tutorial name GlobalRefgetStore:

from gtars.refget import RefgetStore

Verified constructors:

RefgetStore.in_memory()
RefgetStore.on_disk(cache_path)
RefgetStore.open_local(path)
RefgetStore.open_remote(cache_path, remote_url)
RefgetStore.store_exists(path)

Local and in-memory modes

from gtars.refget import RefgetStore

store = RefgetStore.in_memory()
metadata, was_new = store.add_sequence_collection_from_fasta(
    "reviewed-reference.fa",
    force=False,
    namespaces=["refseq"],
)

store.write_store_to_dir("approved-store")
reopened = RefgetStore.open_local("approved-store")

Side effects:

• in_memory() does not write.
• on_disk(path) opens an existing store or creates a new disk-backed store.
• open_local(path) reads store metadata/indexes and lazy-loads local sequence data.
• write_store_to_dir and enable_persistence create/write files.
• force=True can replace existing collection/sequence entries.

Use store_exists(path) before choosing create versus open. Refuse symlinks, unexpected files, output collisions, and unapproved stores.

Batch import:

results = store.add_sequence_collections_from_fastas(
    ["ref-a.fa.gz", "ref-b.fa.gz"],
    file_list=None,
    jobs=1,
    force=False,
    namespaces=["refseq"],
)

fastas can also accept globs or directories and jobs=0 means automatic concurrency. For controlled runs, enumerate reviewed files explicitly and set a positive bounded job count.

Metadata-only and sequence access

Metadata methods avoid loading full sequence content:

collections_page = store.list_collections(page=0, page_size=100)
sequence_metadata = store.list_sequences()
one_metadata = store.get_sequence_metadata(sequence_digest)
collection_metadata = store.get_collection_metadata(collection_digest)

Data methods:

record = store.get_sequence(sequence_digest)
by_name = store.get_sequence_by_name(collection_digest, "chr1")
piece = store.get_substring(sequence_digest, 100, 200)
pieces = store.get_substrings(sequence_digest, [(100, 200), (500, 550)])

for chunk in store.stream_sequence(
    sequence_digest,
    start=0,
    end=1_000_000,
    chunk_size=65_536,
):
    consume_bounded(chunk)

Substring ranges are 0-based half-open. Validate 0 <= start <= end <= length. stream_sequence bounds peak result memory, but downstream accumulation can still defeat streaming.

The 0.9.2 runtime also exposes load_sequence, load_collection, load_all_sequences, and load_all_collections. These materialize more data; do not call an all-load method without an explicit byte/RAM budget.

Remote stores require an approval gate

# Network + cache side effects: do not call before approval.
remote = RefgetStore.open_remote(
    approved_cache_path,
    approved_https_base_url,
)

open_remote takes only a cache path and base URL. It fetches remote metadata, creates/uses local cache state, and enables persistence by default. In 0.9.2:

• get_substring can issue remote byte-range reads without downloading the whole sequence;
• stream_sequence can stream remote ranges;
• load_sequence is the whole-sequence path and can persist it.

Calling disable_persistence() after opening does not undo metadata/cache work already performed. There is no constructor parameter for revision, endpoint allowlist, checksum manifest, byte quota, or offline mode.

Before open_remote:

1. obtain explicit approval for exact HTTPS host/path and cache directory;
2. reject credentials in URLs and unreviewed redirects;
3. pin an immutable server/store revision or content-addressed identifier;
4. record expected collection/sequence digests and independent SHA-256 where applicable;
5. cap metadata, per-range, total transfer, sequence length, cache, retries, concurrency, and time;
6. disclose that request coordinates/digests and network metadata leave the environment;
7. validate returned lengths and digests before scientific use.

Custom/patient-specific assemblies and requested ranges can be sensitive even when a public reference genome is not.

Read-only concurrent stores

into_readonly() converts a mutable store into ReadonlyRefgetStore for concurrent reads. It consumes/replaces the mutable store and cannot lazy-load collections that were not prepared. Load only the bounded data required before conversion; do not use load_all_* reflexively.

CLI store build

The only current refget CLI subcommand is local store construction:

gtars refget build reference.fa reference-alt.fa.gz \
  --output approved-store \
  --jobs 1

Options:

• --file-list/-f PATH: file of paths/globs/directories;
• --output/-o DIR: required;
• --jobs/-j N: concurrent FASTA files, default 0 (auto);
• --raw: raw instead of default encoded 2-bit storage;
• --force: overwrite existing entries.

There is no current gtars refget digest or verify CLI matching the old skill. Use the Python digest functions, direct Rust API, or a local store build after preflight.

Offline metadata/digest plan

The bundled helper accepts a conservative local policy manifest:

{
  "schema_version": "1.0",
  "assembly": "GRCh38.p14",
  "coordinate_system": "0-based-half-open",
  "collection_digest": "<32-char sha512t24u-like value>",
  "sequences": [
    {
      "name": "chr1",
      "length": 248956422,
      "sha512t24u": "<32-char digest>",
      "md5": "<32 lowercase hex>"
    }
  ]
}

This is a skill policy manifest, not an upstream refget wire schema. Validate it and optionally recompute sequence-level digests from local FASTA:

python3 -B scripts/refget_digest_plan.py \
  --metadata refget-metadata.json \
  --fasta reference.fa.gz \
  --assembly GRCh38.p14

The helper does not recompute the sequence-collection digest; it validates its shape and verifies each listed local sequence's length, sha512t24u, and MD5. Use Gtars' pinned collection implementation for final seqcol verification.

BEDbase and bbcache

BEDbase caching is separate from refget. In CLI 0.9.0:

gtars bbcache cache-bed
gtars bbcache cache-bedset
gtars bbcache seek
gtars bbcache inspect-bedfiles
gtars bbcache inspect-bedsets
gtars bbcache rm

Defaults from tagged source:

• API: BEDBASE_API, otherwise https://api.bedbase.org;
• cache: BBCLIENT_CACHE, otherwise $HOME/.bbcache/ (or /tmp/.bbcache/);
• cached BEDs: bedfiles/<first>/<second>/<id>.bed.gz;
• BED sets: bedsets/<first>/<second>/<id>.txt;
• cache metadata includes SQLite state through the cache dependency.

Constructing BBClient creates the root and BED/BED-set subdirectories, even for inspection/seek. cache-bed accepts a local file/directory, URL, or intended BEDbase identifier; cache-bedset accepts a local directory/list or intended BEDbase ID. rm deletes files and cache records and can remove member BEDs for a BED set.

Important source finding: the v0.9.0 BBClient.load_bed(id) delegates a bare ID to RegionSet::try_from, while the core source has the bare-BEDbase-ID fallback commented out. Therefore, ID-only cache-bed/BED-set downloads may fail in this release even though the public docs claim support. Do not work around this with guessed URLs. Verify the installed help/behavior on a non-sensitive approved test, or resolve an explicit official file URL through reviewed BEDbase metadata.

The bbcache source does not expose an expected SHA-256/revision parameter. Treat downloads as untrusted:

• approve and allowlist api.bedbase.org and any exact data host separately;
• record BEDbase record ID, API response revision/time, explicit file URL, expected identifier/digest, and SHA-256;
• use an explicit quota-limited cache folder;
• validate BED assembly/bounds/content before indexing;
• never rely on a successful cache write as integrity proof.

Rust pin

For the latest direct refget component:

[dependencies]
gtars-refget = "=0.9.1"

For the 0.9.0 wrapper release set:

[dependencies]
gtars = { version = "=0.9.0", default-features = false, features = ["refget"] }

Do not assume these expose identical refget patch behavior.

Official sources (accessed 2026-07-23)

• Python refget 0.9.2 stubs
• Python 0.9.2 release
• gtars-refget 0.9.1 crate
• Gtars refget module guide
• Gtars Python refget API
• CLI refget parser
• BEDbase cache source
• BEDbase caching guide
• GA4GH refget sequences v2
• GA4GH refget sequence collections