eden/mononoke/docs/4.5-microwave.md
This document explains Mononoke's microwave cache warming system—a mechanism for preloading caches with repository data before they are needed.
Microwave (features/microwave/) is a cache warming system that addresses the cold start problem in Mononoke deployments. When a Mononoke server starts with empty caches, initial requests experience higher latency as data is fetched from backend storage and loaded into cache. Microwave preloads caches with repository state, reducing this initial latency.
The system consists of two components:
Builder (features/microwave/builder/) - Creates snapshots of repository state that can be stored and later used for cache warming. The builder binary runs as an offline job.
Preloader (features/microwave/src/lib.rs) - Loads stored snapshots and populates server caches during startup or on-demand.
Microwave integrates with the broader cache warmup system (features/cache_warmup/) which coordinates various cache preloading activities.
Microwave operates in two phases: snapshot creation and snapshot loading.
The microwave builder creates a snapshot of repository state by:
main)The snapshot format is defined in features/microwave/if/microwave.thrift and includes:
Snapshots are versioned using a code version constant, allowing the format to evolve while maintaining backward compatibility.
Snapshots can be stored in two locations:
Blobstore - Snapshots are written to the mutable repository blobstore using a versioned key (microwave_snapshot_v{version}). This is the primary storage location for production deployments.
Local filesystem - Snapshots can be written to a shared filesystem path for development and testing scenarios.
The storage location is specified when running the builder binary.
When a server starts or when cache warming is triggered, microwave loads the stored snapshot and populates caches:
This preloading happens before the server begins serving client requests, reducing the number of cache misses during initial operation.
Microwave focuses on specific types of data that are frequently accessed and expensive to derive:
The primary data type warmed by microwave is filenodes—Mercurial file metadata that maps file paths and filenode hashes to their parent relationships and linknode information. Filenodes are required for:
Populating the filenodes cache avoids expensive database queries during initial requests.
While microwave itself focuses on filenodes, it operates as part of the broader cache warmup process that also:
The complete cache warmup process integrates microwave preloading with manifest warming, commit graph segment loading, and derived data computation.
The cache warmup process that microwave participates in also loads manifest entries from the blobstore, populating both cachelib (in-memory cache) and memcache (shared cache) with:
This reduces the number of blobstore fetches needed during initial operation.
Cache warming occurs in specific scenarios determined by configuration and operational needs.
Mononoke servers can be configured to perform cache warmup at startup. The configuration is specified per-repository in the repository configuration file using CacheWarmupParams:
main or another important branch)When microwave_preload is enabled, the server loads the snapshot before performing other cache warming activities.
The microwave builder runs as a scheduled job to create updated snapshots. This job:
The builder uses the same cache warmup infrastructure to ensure derived data is available before creating the snapshot.
Cache warming can also be triggered programmatically through the cache_warmup function in features/cache_warmup/src/lib.rs. This is used by:
Microwave is one component of Mononoke's multi-phase cache warming strategy:
When cache warmup runs with microwave enabled:
These phases run concurrently where possible to reduce total warmup time.
The cache warmup system distinguishes between two scenarios:
MicrowaveBuilder - Used when creating snapshots. Speed is less critical as this is an offline job. All necessary derived data is computed.
MononokeServer - Used during server startup. Optimized for speed to minimize the time before the server can handle traffic. Some optimizations (like skipping blobstore warmup) may be enabled via configuration.
Cache warmup configuration is specified in repository configuration files using CacheWarmupParams. The warmup runs at server startup for repositories assigned to each server instance through repository sharding.
Warmup can be disabled by omitting the cache_warmup configuration section, leaving caches cold at startup.
Several characteristics of microwave affect its operational behavior:
Snapshot Staleness - Snapshots reflect repository state at the time of builder execution. If the configured bookmark moves forward significantly between builder runs, the snapshot becomes less effective as it does not include recent commits.
Builder Coupling - The builder must run periodically to maintain useful snapshots. If the builder fails or does not run, snapshots become stale and eventually may reference derived data that is no longer available.
Storage Overhead - Snapshots consume blobstore space. The size depends on the number of files in manifests being snapshotted and the depth of history captured.
Warmup Time - Loading a snapshot is faster than deriving data from scratch, but still takes measurable time during server startup. The warmup process must complete before the server is ready to serve traffic.
Versioning - Snapshot format changes require updating the CODEVER constant in the Thrift definition. Old snapshots become invalid when the version changes, requiring the builder to run with the new version before servers can use updated snapshots.
Microwave interacts with several other Mononoke components:
Filenodes (filenodes/ and repo_attributes/filenodes/) - The primary data structure that microwave preloads. The filenodes implementation provides a prime_cache method specifically for microwave.
Cache Warmup (features/cache_warmup/) - The orchestration layer that coordinates microwave with other warmup activities.
Warm Bookmarks Cache (repo_attributes/bookmarks/warm_bookmarks_cache/) - A complementary caching mechanism that maintains up-to-date derived data for important bookmarks through continuous background derivation.
Derived Data - Microwave depends on derived data being available for the target bookmark. The FilenodesOnlyPublic derived data type is computed during the builder process.
Mutable Blobstore (repo_attributes/mutable_blobstore/) - Used to store snapshots in the blobstore when not using local filesystem storage.
Key implementation files:
features/microwave/src/lib.rs - Core preloading logicfeatures/microwave/builder/main.rs - Builder binaryfeatures/microwave/if/microwave.thrift - Snapshot data formatfeatures/cache_warmup/src/lib.rs - Integration point for cache warmupThe microwave builder is a standard Mononoke binary using the mononoke_app framework. It can be built and run using:
buck2 build @mode/opt fbcode//eden/mononoke/features/microwave/builder:builder
buck2 run @mode/opt fbcode//eden/mononoke/features/microwave/builder:builder -- blobstore
The builder supports two subcommands: local-path (for filesystem storage) and blobstore (for blobstore storage).
Component-specific implementation details are located in features/microwave/ and features/cache_warmup/.