eden/mononoke/docs/3.1-servers-and-services.md
This document describes Mononoke's servers and microservices, their responsibilities, and how they interact with the repository layer.
Mononoke is deployed as a distributed service architecture with multiple server types. Frontend services handle external client requests, while internal microservices manage expensive operations and provide coordination. All services are stateless, with persistent state stored in external storage (blobstore and metadata database).
The separation of frontend services from microservices allows independent scaling and workload isolation. Frontend services can scale horizontally to handle client load, while microservices can be scaled based on the characteristics of specific operations (CPU-intensive derivation, serialized landing operations, etc.).
Frontend services implement various protocols to serve external clients. Each service is stateless and can be deployed as multiple instances behind a load balancer.
Location: servers/slapi/slapi_server/ Binary:
fbcode//eden/mononoke:mononoke Clients: Sapling CLI, EdenFS
The Mononoke server is the primary frontend service for Sapling and EdenFS clients. It implements the EdenAPI (also called SLAPI - Sapling Remote API) protocol over HTTP, providing source control operations for Sapling-based workflows.
The server handles clone, pull, and push operations from Sapling clients, as
well as on-demand file fetches from EdenFS instances. Request handling is
implemented in the repo_listener/ subdirectory, which manages TCP connections,
TLS termination, and routing to protocol handlers. The actual protocol
implementation resides in servers/slapi/slapi_service/, which translates HTTP
requests into operations on the repository layer.
The server supports repository sharding, allowing repositories to be distributed across multiple server instances based on external shard assignment. Cache warmup is performed at startup for repositories assigned to each instance, preloading derived data and warming caches for optimal performance.
Location: scs/scs_server/ Binary:
fbcode//eden/mononoke/servers/scs/scs_server:scs_server Clients: SCMQuery,
internal tools, automation
The Source Control Service provides a Thrift API for programmatic repository access. While the Mononoke server implements VCS protocols (Git, Sapling), SCS exposes a high-level, language-agnostic API for tools and services that need to query or manipulate repositories.
The Thrift interface (scs/if/source_control.thrift) defines approximately 80
methods for repository operations: commit lookups, history traversal, blame
annotation, file content retrieval, directory listings, and diff generation.
Write operations include commit creation and bookmark manipulation. The service
supports multiple commit identity schemes (Bonsai, Git, Mercurial, Globalrev,
SVN), translating between formats transparently using VCS mapping facets.
SCS is used by internal services for repository introspection, code review systems for commit information, and automation tools for repository management. The service handles authentication via mTLS and performs authorization checks through the repository permission checker facet.
Location: servers/git/git_server/ Binary:
fbcode//eden/mononoke/servers/git/git_server:git_server Clients: Standard
Git clients
The Git server implements the Git protocol over HTTP, allowing standard Git clients to interact with Mononoke repositories. It handles Git clone, fetch, and push operations using Git's pack protocol and object formats.
The server translates between Git's data model (commits, trees, blobs) and
Mononoke's Bonsai format using the Git-specific facets and derived data types.
Git commits are mapped to Bonsai changesets via bonsai_git_mapping, while Git
trees are derived from Bonsai file changes. Push operations create Bonsai
changesets from incoming Git commits and can land them using pushrebase if
invoked from the git pushrebase client.
The server supports Git's smart HTTP protocol, including upload-pack (fetch) and receive-pack (push) operations. It implements packfile generation for efficient data transfer and handles Git references (branches and tags) through symbolic refs facets.
Location: servers/lfs/lfs_server/ Binary:
fbcode//eden/mononoke/servers/lfs/lfs_server:lfs_server Clients: Git
clients, Sapling clients, EdenFS
The LFS server implements the Git LFS (Large File Storage) protocol over HTTP. It handles upload and download of large files referenced by pointer files in Git and Sapling repositories.
The server stores LFS objects in the blobstore using the filestore facet, which provides chunking for large files and content-addressed storage. Upload operations accept file content via HTTP POST and return object IDs (content hashes). Download operations serve file content by object ID, streaming data to avoid memory pressure for large files.
The server can operate standalone or with an upstream LFS server for federation. Batch API endpoints allow clients to request upload or download URLs for multiple objects in a single request, reducing round trips during operations involving many large files.
Internal microservices handle specialized operations that are offloaded from frontend servers. These services provide coordination, serialization, and dedicated scaling for specific workloads.
Location: servers/land_service/ Interface:
servers/land_service/if/land_service.thrift Clients: Frontend servers
(Mononoke, SCS), automation
The Land Service provides serialized commit landing to public bookmarks. It ensures that commits are landed atomically via pushrebase without race conditions from concurrent push operations.
When multiple clients attempt to push to the same bookmark simultaneously, frontend servers forward requests to the Land Service rather than performing pushrebase locally. The service queues landing requests per bookmark and processes them serially, preventing wasted work from failed pushrebases where servers compete to move the same bookmark.
The service executes the full pushrebase workflow: running pre-commit hooks, rebasing commits onto the current bookmark position, and moving the bookmark atomically. It returns the mapping of original commit IDs to rebased commit IDs, along with information about conflicts or hook failures.
Location: derived_data/remote/ (Thrift interface),
facebook/derived_data_service/ (server implementation) Interface:
derived_data/remote/if/derived_data_service.thrift Clients: Frontend
servers, derivation workers
The Derived Data Service coordinates asynchronous derivation of derived data types. Rather than each server deriving data independently, this service deduplicates derivation requests and manages a pool of derivation workers.
Frontend servers submit derivation requests to the service, which returns a token for polling completion status. Requests are placed in a dependency-aware queue that ensures derived data types are computed in the correct order (e.g., blame requires unodes, which require fsnodes). Background workers poll the queue and execute derivation using the derived data manager.
This architecture reduces redundant computation (data derived once is available to all servers), provides centralized control over derivation resources, and allows horizontal scaling of derivation capacity by adding workers.
Bookmark Service (Facebook-internal: facebook/bookmark_service/):
Maintains a warm cache of public bookmarks with all derived data types computed,
used by frontend servers to quickly serve queries for important branches.
Diff Service (Facebook-internal: facebook/diff_service/): Provides diff
computation as a service, offloading expensive diff generation from frontend
servers.
Load Limiter Service (Facebook-internal: facebook/load_limiter/):
Implements rate limiting and load shedding across multiple frontend instances,
coordinating QPS limits.
These services follow the same pattern: Thrift interfaces, stateless operation, and integration with the repository layer through facets.
All servers and services are built using the same internal architecture
described in Architecture Overview. They use the
cmdlib/mononoke_app/ framework for initialization and access repositories
through facets.
Servers obtain repository capabilities through facets rather than accessing a monolithic repository object. A server declares its requirements via trait bounds, specifying which facets it needs:
async fn handle_request(repo: &impl RepoBlobstore + Bookmarks + RepoDerivedData) {
// Access storage
let blobstore = repo.repo_blobstore();
// Query bookmarks
let bookmark = repo.bookmarks().get(...).await?;
// Access derived data
let derived_data = repo.repo_derived_data();
}
This pattern makes dependencies explicit and allows testing with minimal facet implementations rather than full repository instances.
Servers use features to implement high-level operations. Features are stateless functions that compose multiple facets. For example:
Pushrebase (in features/pushrebase/): The Mononoke server and SCS server
both use pushrebase when landing commits. The pushrebase feature requires
several facets: bookmarks (to read and update target bookmark), commit graph
(for ancestry queries), derived data (to compute manifests), and hook manager
(to run validation).
Cross-repo sync (in features/cross_repo_sync/): Used by servers handling
cross-repository operations, this feature composes VCS mapping facets, commit
transformation logic, and blobstore access.
By using features, servers avoid reimplementing complex operations and benefit from shared, well-tested code.
Frontend servers typically use the mononoke_api/ layer, which provides
high-level abstractions over facets. This layer exposes Repo, Changeset, and
File objects that encapsulate common operations:
let repo = mononoke_api.repo(...)?;
let changeset = repo.changeset(commit_id).await?;
let file = changeset.path_with_content(path).await?;
The API layer hides internal representation details (which manifest type, which commit graph implementation) and provides a stable interface as implementations evolve.
Mononoke servers use HTTP and Thrift for communication.
EdenAPI (SLAPI): The Mononoke server implements EdenAPI over HTTP/2. Requests and responses use CBOR (Compact Binary Object Representation) encoding for efficient serialization. The protocol is designed around streaming and incremental operations - rather than transferring entire bundles, clients request specific files, trees, and commits as needed.
The protocol supports content negotiation, compression, and multiplexing multiple requests over a single connection. Streaming responses allow serving large results without buffering entire datasets in memory.
Git Protocol: The Git server uses Git's smart HTTP protocol, implementing the upload-pack and receive-pack services. Requests use Git's pkt-line format, and responses include packfiles containing Git objects (commits, trees, blobs).
LFS Protocol: The LFS server follows the Git LFS specification, providing JSON-based batch API endpoints and binary upload/download endpoints. Upload URLs are generated by the batch endpoint and used by clients to PUT file content. Download URLs similarly provide GET access to stored objects.
SCS: Uses Thrift RPC over HTTP with mTLS for authentication. The interface supports both synchronous methods (simple queries) and streaming methods (for operations returning large result sets).
Land Service: Provides a single Thrift method (land_changesets) that
accepts commit stacks and returns pushrebase outcomes or error details
(conflicts, hook rejections).
Derived Data Service: Asynchronous Thrift API where clients submit derivation requests and poll for completion. The service returns tokens that encode request state for later status queries.
All servers are stateless. They do not maintain per-client session state or cache data specific to individual requests. State resides in:
This design allows horizontal scaling - adding more server instances increases capacity without coordination overhead. Load balancers can route requests to any instance, and failed instances can be replaced without state migration.
Mononoke supports two sharding modes:
Shallow sharding: All repositories are loaded at startup. A load balancer routes requests for specific repositories to designated server instances. This mode is suitable for smaller deployments with tens or hundreds of repositories.
Deep sharding: Repositories are loaded on-demand based on external shard assignment. A shard manager (external to Mononoke) determines which repositories each server instance should handle. Servers dynamically add and remove repositories as assignments change. This mode scales to thousands of repositories.
Sharding configuration is specified in repository configs and command-line
arguments (--sharded-service-name). Each service type can use different
sharding strategies.
Servers share common configuration through cmdlib/mononoke_app/:
metaconfig/)Service-specific configuration includes:
Configuration is loaded at startup, with some settings (rate limits, feature flags) supporting dynamic updates via cached config handles.
All servers export metrics and logs through standard infrastructure:
Metrics: ODS counters and histograms track QPS, latency (P50, P90, P99), error rates, and resource usage per endpoint/method.
Logging: Scuba logging captures request details: repository, operation type, client identity, duration, outcome. Sampling rates can be configured per service.
Tracing: Distributed tracing propagates request context through service calls, allowing end-to-end latency analysis across frontend servers and microservices.
Health checks: Servers expose health endpoints and ready flags used by load balancers and orchestration systems.
Monitoring infrastructure is initialized by the mononoke_app framework and
integrated into each server's request handling middleware.
Mononoke's service architecture separates concerns between frontend protocol servers and internal microservices. Frontend servers (Mononoke, SCS, Git, LFS) handle external clients and implement various protocols, while microservices (Land, Derived Data) manage expensive operations and provide coordination.
All services use the same internal architecture: stateless operation, facet-based repository access, and shared library code. This consistency reduces maintenance burden and allows new services to be built by composing existing facets and features.
The stateless design enables horizontal scaling and operational flexibility - capacity can be added by deploying more instances, and servers can be updated or replaced without state migration. Protocol diversity (HTTP, Thrift) supports different client types while maintaining a single canonical repository backend.
For operational details about specific servers, refer to component directories:
servers/slapi/, scs/, servers/git/, servers/lfs/,
servers/land_service/, and derived_data/remote/.