eden/mononoke/docs/3.4-libraries-and-frameworks.md
This document describes the shared libraries and frameworks used across Mononoke applications. These components provide common functionality for building servers, tools, and jobs, and form the foundation layer of Mononoke's architecture as described in the Architecture Overview.
Mononoke applications are built using a consistent set of libraries that handle common concerns like command-line parsing, configuration loading, monitoring setup, asynchronous operations, SQL database access, and logging. This shared foundation ensures that all Mononoke binaries have consistent behavior and operational characteristics.
The libraries described in this document fall into several categories:
cmdlib/mononoke_app/) - Standard framework for building Mononoke binariescmdlib/) - Argument handling, environment setup, configurationcommon/) - Async helpers, SQL utilities, logging, and specialized toolsmononoke_types/) - Fundamental data structurestests/utils/) - Shared testing toolsThe cmdlib/mononoke_app/ directory contains the standard framework for building Mononoke applications. All Mononoke binaries - servers, tools, and jobs - use this framework to ensure consistent initialization, configuration, and operational behavior.
The framework handles:
Applications built with mononoke_app follow a standard pattern:
MononokeAppBuilder with FacebookInitThe framework provides two main types:
MononokeAppBuilder - Builder for constructing applicationsMononokeApp - Runtime application object passed to the main functionTo create a new Mononoke binary:
Define the main function:
use anyhow::Result;
use fbinit::FacebookInit;
use mononoke_app::{MononokeApp, MononokeAppBuilder};
#[fbinit::main]
fn main(fb: FacebookInit) -> Result<()> {
let app = MononokeAppBuilder::new(fb).build()?;
app.run_with_monitoring_and_logging(async_main, "binary_name", AliveService)
}
async fn async_main(app: MononokeApp) -> Result<()> {
// Application logic here
Ok(())
}
Access repositories:
Applications can open repositories using the repository arguments provided by the framework:
use mononoke_app::args::RepoArgs;
async fn async_main(app: MononokeApp) -> Result<()> {
let repo_args: RepoArgs = app.args()?;
let repo = app.open_repo(&repo_args).await?;
// Use repository facets
Ok(())
}
The framework automatically handles argument parsing for repository selection, configuration paths, caching options, logging levels, and other common parameters.
Applications can register extensions to add custom argument groups or initialization logic:
let app = MononokeAppBuilder::new(fb)
.with_app_extension(CustomExtension::new())
.build()?;
Extensions provide additional arguments and can perform initialization when the application starts. Standard extensions include MonitoringAppExtension for monitoring setup and ScrubAppExtension for blobstore scrubbing capabilities.
For tools with multiple subcommands (like the admin CLI), the framework provides a macro to define subcommands:
use mononoke_app::subcommands;
subcommands! {
mod first_command;
mod second_command;
}
Each subcommand module defines a CommandArgs struct and a run function. The framework handles dispatching to the appropriate subcommand based on command-line arguments.
The framework automatically provides several categories of command-line arguments:
Configuration:
--config-path - Path to repository configurations--config-tier - Configuration tier to useRepository Selection:
--repo-name - Repository name--repo-id - Repository IDCaching:
--cache-mode - Caching behavior (local-only, shared, disabled)--cachelib-size - In-process cache sizeStorage:
--readonly-storage - Enable read-only mode--blobstore-options - Blobstore configuration overridesLogging:
--log-level - Logging verbosity--scuba-dataset - Scuba logging datasetMonitoring:
--monitoring-port - Port for monitoring endpointsApplications can add their own arguments alongside these standard options.
The cmdlib/ directory contains libraries used by mononoke_app and available to applications for specific needs. These are organized by functionality:
config_args/ - Configuration path parsing and loading
commit_id/ - Commit identifier argument parsing (supports Bonsai IDs, Git hashes, Mercurial hashes, bookmark names)
sharding/ - Shard selection and filtering arguments
base_app/ - Base application framework (used internally by mononoke_app)
environment/ - Mononoke environment setup (database pools, caching, configuration store)
logging/ - Logging initialization and configuration
log/ - Structured logging utilities
caching/ - Cachelib initialization and configuration
scrubbing/ - Blobstore scrubbing support for validation
cross_repo/ - Cross-repository operation utilities
displaying/ - Output formatting and display utilities
extensions/ - Extension framework for adding custom functionality
zk_leader_election/ - ZooKeeper-based leader election for distributed jobs
Applications typically do not use these libraries directly - mononoke_app provides access to their functionality through the application object and standard arguments.
The common/ directory contains specialized utility libraries used throughout Mononoke. These provide building blocks for asynchronous programming, database access, logging, and other common needs.
futures_watchdog/ - Monitors futures and logs warnings when polling takes too long. Helps identify performance problems where futures block for extended periods.
async_limiter/ - Limits concurrent execution of asynchronous operations. Provides backpressure and resource control.
assembly_line/ - Sequential processing pipeline for async operations.
running/ - Utilities for running tasks until termination signals.
yield_stream/ - Stream utilities that yield control periodically.
sql_construct/ - Construction utilities for SQL database managers. Provides traits for building database managers from configuration, with support for both MySQL and SQLite backends.
rust/sql_ext/ (in common/rust/sql_ext/) - SQL connection utilities, query builders, and Facebook-specific MySQL features like connection pooling and failover.
scuba_ext/ - Extensions for Scuba structured logging. Provides MononokeScubaSampleBuilder for consistent logging across Mononoke.
logger_ext/ - Extensions to the standard logger for Mononoke-specific needs.
ods_counters/ - ODS metrics counters for operational monitoring.
session_id/ - Session ID tracking for request correlation.
scribe_ext/ - Scribe logging integration.
dedupmap/ - Hash map that deduplicates values with the same key.
uniqueheap/ - Heap data structure that ensures uniqueness of elements.
type_map/ - Type-indexed map for storing values of different types.
topo_sort/ - Topological sorting for directed acyclic graphs.
time_measuring/ - Utilities for measuring operation duration.
reloader/ - Periodic reloading of configuration or data.
path_hash/ - Path hashing utilities.
copy_utils/ - Utilities for copying data structures.
rendezvous/ - Batching of SQL queries with deduplication.
wait_for_replication/ - Waits for database replication to catch up.
metadata/ - System metadata utilities.
memory/ - Memory usage tracking and reporting.
connection_security_checker/ - Connection security validation.
iterhelpers/ - Iterator utility functions and extensions.
thrift_convert/ - Conversion utilities between Mononoke types and Thrift types.
In addition to Mononoke-specific utilities in common/, Mononoke uses utilities from common/rust/shed/, which provides general-purpose Rust libraries used across multiple Meta projects.
Key libraries from shed include:
bounded_traversal/ - Traversal of trees and DAGs with bounded concurrency. Provides functions for processing large graphs in parallel while limiting the number of concurrent operations. Used extensively in Mononoke for walking commit graphs and file trees.
facet/ - Facet container pattern for trait-based composition. The foundation for Mononoke's repository facet system.
fbinit/ - Facebook initialization. Required for all Mononoke binaries - provides the FacebookInit token that ensures proper initialization of Facebook infrastructure.
cached_config/ - Configuration system that periodically reloads configuration from external sources. Used for repository configuration and feature flags.
cloned/ - Macro for cloning variables into closures without verbose syntax.
futures_stats/ - Statistics collection for futures execution.
stats/ - Metrics and statistics framework.
scuba_sample/ - Scuba logging sample builder.
These libraries are located in common/rust/shed/ rather than eden/mononoke/common/ because they are shared across multiple projects at Meta.
The mononoke_types/ directory defines the fundamental data structures used throughout Mononoke:
Bonsai Changesets - The canonical representation of commits in Mononoke. Includes parent relationships, author information, timestamp, message, and file changes.
Content Identifiers - Content-addressed identifiers using Blake2b hashing. Includes ContentId for file contents and ChangesetId for commits.
File Changes - Representation of file additions, modifications, and deletions.
Paths - Type-safe path representations with validation (MPath, NonRootMPath).
VCS Types - Git and Mercurial-specific identifiers and structures for interoperability.
Derived Data Types - Type definitions for various derived data formats.
These types are used across all layers of Mononoke - from storage through the API layer to protocol servers. They provide a stable interface and ensure type safety throughout the system.
The types are designed for:
serde and ThriftContentId where a ChangesetId is expected)The tests/utils/ directory provides utilities for writing tests:
drawdag - A domain-specific language for constructing commit graphs in tests. Allows compact representation of test repositories:
// Example usage pattern (simplified)
let commits = drawdag::extend_from_dag(
&repo,
r##"
A-B-C
\
D-E
"##,
)?;
This creates a commit graph with branches without verbose setup code.
random - Random data generation for testing. Provides functions for creating random file contents, paths, and other test data.
Library functions - Various helper functions for common test setup tasks like creating repositories, initializing storage, and setting up test environments.
These utilities are used extensively in both unit tests and integration tests to reduce boilerplate and make tests more readable.
Most Mononoke development involves either:
mononoke_appmononoke_app frameworkcommon/ for specific needsThe framework handles initialization and provides access to repositories, so application code can focus on the specific operations needed.
Common utilities are available as dependencies in BUCK files:
rust_binary(
name = "my_tool",
srcs = glob(["src/**/*.rs"]),
deps = [
"//eden/mononoke/cmdlib/mononoke_app:mononoke_app",
"//eden/mononoke/common/futures_watchdog:futures_watchdog",
"//eden/mononoke/common/sql_construct:sql_construct",
"//common/rust/shed/bounded_traversal:bounded_traversal",
],
)
Applications access repository facets through the repository object:
let blobstore = repo.repo_blobstore();
let commit_graph = repo.commit_graph();
let derived_data = repo.repo_derived_data();
The mononoke_app framework handles repository initialization based on configuration, so applications receive fully initialized repositories.
Mononoke's libraries and frameworks provide:
mononoke_appmononoke_typesWhen building new functionality:
mononoke_app for new binariescommon/ or shed/ as neededmononoke_types for data structurestests/utils for testingThe framework and utilities handle initialization, configuration, monitoring, and other cross-cutting concerns, allowing application code to focus on source control operations.
For component-specific details: