Indexer Data and Integration

Building a custom indexer on Sui lets you take full control of data ingestion, storage, and processing. You can choose from multiple checkpoint data sources such as remote store, local files, or direct RPC API access, depending on whether you're indexing Mainnet data, testing against known checkpoints, or working on a local network.

For storage, you can either use the built-in IndexerCluster with PostgreSQL or implement your own Store and Connection traits to integrate a different database or storage backend. After connected, you can wire up a manual indexer, add your custom pipelines, and handle watermark coordination to keep data in sync.

Finally, you need to deserialize Move events from raw BCS bytes into Rust structs, using bcs and serde, so that your pipelines can work with strongly-typed data. This gives you a reproducible, end-to-end setup that you can tune for performance, reliability, and custom analytics.

Checkpoint data sources {#checkpoint-data-sources}

The sui-indexer-alt-framework supports multiple data sources for accessing Sui blockchain data.You configure these sources through command-line arguments. They fall into two categories: push-based and polling-based.

Push-based data sources

Push-based sources deliver real-time checkpoint data as it becomes available, offering lower latency than polling.

Recommended: gRPC streaming

gRPC streaming delivers real-time checkpoint data pushed from full nodes for the latest checkpoints. Because it only streams latest data, you must configure a polling-based fallback source (remote store, local path, or full node RPC) to retrieve historical checkpoints and ensure reliability:

$ cargo run -- --remote-store-url https://checkpoints.testnet.sui.io --streaming-url https://fullnode.testnet.sui.io:443

Endpoint format: https://fullnode.NETWORK.sui.io:443 where NETWORK is one of the available networks:

• testnet
• devnet
• mainnet

When to use gRPC streaming:

• Production indexers that require minimal latency
• Real-time data processing pipelines
• Applications that need immediate checkpoint updates

:::info

• Always configure a polling-based fallback source (like --remote-store-url) to ensure reliability and access to historical data.
• The streaming connection automatically falls back to the polling source when streaming is unavailable or when historical checkpoints are needed.

:::

Polling-based data sources

Polling-based sources periodically check for new checkpoints using the polling mechanism provided by the indexing framework.

Remote store

The most direct stream source is a subscription to a remote checkpoint store. Sui provides the following endpoints:

• Testnet: https://checkpoints.testnet.sui.io
• Mainnet: https://checkpoints.mainnet.sui.io

$ cargo run -- --remote-store-url https://checkpoints.testnet.sui.io

Local path

Local checkpoint files are useful for development and testing scenarios:

$ cargo run -- --local-ingestion-path /path/to/checkpoints

Common use cases:

• Unit and integration testing with known checkpoint data
• Development workflows with reproducible datasets

Full node RPC

Full node RPC queries use gRPC:

$ cargo run -- --rpc-api-url https://fullnode.testnet.sui.io:443

Endpoint format: https://fullnode.NETWORK.sui.io:443 where NETWORK is one of the available networks:

• testnet
• devnet
• mainnet

When to use RPC API:

• Networks without official remote store (devnet, localnet, custom networks)
• Development against local Sui networks