doc/design/multiprocess.md
Guide to the design and architecture of the Bitcoin Core multiprocess feature
This document describes the design of the multiprocess feature. For usage information, see the top-level multiprocess.md file.
The Bitcoin Core software has historically employed a monolithic architecture. The existing design has integrated functionality like P2P network operations, wallet management, and a GUI into a single executable. While effective, it has limitations in flexibility, security, and scalability. This project introduces changes that transition Bitcoin Core to a more modular architecture. It aims to enhance security, improve usability, and facilitate maintenance and development of the software in the long run.
The current system features two primary executables: bitcoind and bitcoin-qt. bitcoind combines a Bitcoin P2P node with an integrated JSON-RPC server, wallet, and indexes. bitcoin-qt extends this by incorporating a Qt-based GUI. This monolithic structure, although robust, presents challenges such as limited operational flexibility and increased security risks due to the tight integration of components.
The new architecture divides the existing code into three specialized executables:
bitcoin-node: Manages the P2P node, indexes, and JSON-RPC server.bitcoin-wallet: Handles all wallet functionality.bitcoin-gui: Provides a standalone Qt-based GUI.This modular approach is designed to enhance security through component isolation and improve usability by allowing independent operation of each module. This allows for new use-cases, such as running the node on a dedicated machine and operating wallets and GUIs on separate machines with the flexibility to start and stop them as needed.
This subdivision could be extended in the future. For example, indexes could be removed from the bitcoin-node executable and run in separate executables. And JSON-RPC servers could be added to wallet and index executables, so they can listen and respond to RPC requests on their own ports, without needing to forward RPC requests through bitcoin-node.
flowchart LR
node[bitcoin-node] -- listens on --> socket["<datadir>/node.sock"]
wallet[bitcoin-wallet] -- connects to --> socket
gui[bitcoin-gui] -- connects to --> socket
This section describes the major components of the Inter-Process Communication (IPC) framework covering the relevant source files, generated files, tools, and libraries.
src/interfaces/src/interfaces/ directory. These classes define pure virtual methods that code in src/node/, src/wallet/, and src/qt/ directories call to interact with each other.src/ipc/capnp/.capnp files within the src/ipc/capnp/ directory. These files are used as input to the mpgen tool (described below) to generate C++ code.mpgen Code Generation Toolmpgen tool which is part of the libmultiprocess project. This tool takes the .capnp files as input and generates C++ code.src/interfaces/. These subclasses are the workhorses of the IPC mechanism.mpgen tool wrap client classes generated by Cap'n Proto, and use them to send IPC requests. The Cap'n Proto client classes are low-level, with non-blocking methods that use asynchronous I/O and pass request and response objects, while mpgen client subclasses provide normal C++ methods that block while executing and convert between request/response objects and arguments/return values.src/interfaces/ objects, and creating the IPC response.mpgen tool inherit from server classes generated by Cap'n Proto, and use them to process IPC requests.libmultiprocess Runtime Librarylibmultiprocess runtime library's primary function is to instantiate the generated client and server classes as needed.interfaces::Init interface (defined in src/interfaces/init.h) to a UNIX socket. This initial interface has methods returning other interfaces that different Bitcoin Core modules use to communicate after the bootstrapping phase.src/ipc/capnp/*-types.hsrc/ipc/capnp/*-types.h, function overloads of libmultiprocess C++ functions, mp::CustomReadField, mp::CustomBuildField, mp::CustomReadMessage and mp::CustomBuildMessage, are defined. These overloads are used for customizing the conversion of specific C++ types to and from Cap’n Proto types.mpgen tool and libmultiprocess library can convert most C++ types to and from Cap’n Proto types automatically, including interface types, primitive C++ types, standard C++ types like std::vector, std::set, std::map, std::tuple, and std::function, as well as simple C++ structs that consist of aforementioned types and whose fields correspond 1:1 with Cap’n Proto struct fields. For other types, *-types.h files provide custom code to convert between C++ and Cap’n Proto data representations.src/ipc/src/ipc/capnp/, the code in the src/ipc/ directory is protocol-agnostic. This enables potential support for other protocols, such as gRPC or a custom protocol in the future.ipc::Exception class which is thrown from the generated C++ client class methods when there is an unexpected IPC error, such as a disconnection.flowchart TD
capnpFile[ipc/capnp/chain.capnp] -->|Input to| mpgenTool([mpgen Tool])
mpgenTool -->|Generates| proxyTypesH[ipc/capnp/chain.capnp.proxy-types.h]
mpgenTool --> proxyClientCpp[ipc/capnp/chain.capnp.proxy-client.c++]
mpgenTool --> proxyServerCpp[ipc/capnp/chain.capnp.proxy-server.c++]
proxyTypesH -.->|Includes| interfaces/chain.h
proxyClientCpp -.-> interfaces/chain.h
proxyServerCpp -.-> interfaces/chain.h
The choice to use Cap’n Proto for IPC was primarily influenced by its support for passing object references and managing object lifetimes, which would have to be implemented manually with a framework that only supported plain requests and responses like gRPC. The support is especially helpful for passing callback objects like std::function and enabling bidirectional calls between processes.
The choice to use an RPC framework at all instead of a custom protocol was necessitated by the size of Bitcoin Core internal interfaces which consist of around 150 methods that pass complex data structures and are called in complicated ways (in parallel, and from callbacks that can be nested and stored). Writing a custom protocol to wrap these complicated interfaces would be a lot more work, akin to writing a new RPC framework.
The IPC mechanism is deliberately isolated from the rest of the codebase so less code has to be concerned with IPC.
Building Bitcoin Core with IPC support is optional, and node, wallet, and GUI code can be compiled to either run in the same process or separate processes. The build system also ensures Cap’n Proto library headers can only be used within the src/ipc/capnp/ directory, not in other parts of the codebase.
The libmultiprocess runtime is designed to place as few constraints as possible on IPC interfaces and to make IPC calls act like normal function calls. Method arguments, return values, and exceptions are automatically serialized and sent between processes. Object references and std::function arguments are tracked to allow invoked code to call back into invoking code at any time. And there is a 1:1 threading model where every client thread has a corresponding server thread responsible for executing incoming calls from that thread (there can be multiple calls from the same thread due to callbacks) without blocking, and holding the same thread-local variables and locks so behavior is the same whether IPC is used or not.
The choice to maintain interface definitions and C++ type mappings as .capnp files in the src/ipc/capnp/ was mostly done for convenience, and is probably something that could be improved in the future.
In the current design, class names, method names, and parameter names are duplicated between C++ interfaces in src/interfaces/ and Cap’n Proto files in src/ipc/capnp/. While this keeps C++ interface headers simple and free of references to IPC, it is a maintenance burden because it means inconsistencies between C++ declarations and Cap’n Proto declarations will result in compile errors. (Static type checking ensures these are not runtime errors.)
An alternate approach could use custom C++ Attributes embedded in interface declarations to automatically generate .capnp files from C++ headers. This has not been pursued because parsing C++ headers is more complicated than parsing Cap’n Proto interface definitions, especially portably on multiple platforms.
In the meantime, the developer guide Internal interface guidelines can provide guidance on keeping interfaces consistent and functional and avoiding compile errors.
The currently defined IPC interfaces are unstable, and can change freely with no backwards compatibility. The decision to allow this stems from the recognition that our current interfaces are still evolving and not yet ideal for external use. As these interfaces mature and become more refined, there may be an opportunity to declare them stable and use Cap’n Proto's support for protocol evolution (Cap'n Proto - Evolving Your Protocol) to allow them to be extended while remaining backwards compatible. This could allow different versions of node, GUI, and wallet binaries to interoperate, and potentially open doors for external tools to utilize these interfaces, such as creating custom indexes through a stable indexing interface. However, for now, the priority is to improve the interfaces internally. Given their current state and the advantages of using JSON-RPC for most common tasks, it's more practical to focus on internal development rather than external applicability.
The integration of Cap’n Proto and libmultiprocess into the Bitcoin Core architecture increases its potential attack surface. Cap’n Proto, being a complex and substantial new dependency, introduces potential sources of vulnerability, particularly through the creation of new UNIX sockets. The inclusion of libmultiprocess, while a smaller external dependency, also contributes to this risk. However, plans are underway to incorporate libmultiprocess as a git subtree, aligning it more closely with the project's well-reviewed internal libraries. While adopting these multiprocess features does introduce some risk, it's worth noting that they can be disabled, allowing builds without these new dependencies. This flexibility ensures that users can balance functionality with security considerations as needed.
Let’s walk through an example where the bitcoin-wallet process requests the hash of a block at a specific height from the bitcoin-node process. This example demonstrates the practical application of the IPC mechanism, specifically the interplay between C++ method calls and Cap’n Proto-generated RPC calls.
sequenceDiagram
box "bitcoin-wallet process"
participant WalletCode as Wallet code
participant ChainClient as Generated Chain client class
ProxyClient<messages::Chain>
end
box "bitcoin-node process"
participant ChainServer as Generated Chain server class
ProxyServer<messages::Chain>
participant LocalChain as Chain object
node::ChainImpl
end
WalletCode->>ChainClient: getBlockHash(height)
ChainClient->>ChainServer: Send RPC getBlockHash request
ChainServer->>LocalChain: getBlockHash(height)
LocalChain->>ChainServer: Return block hash
ChainServer->>ChainClient: Send response with block hash
ChainClient->>WalletCode: Return block hash
Initiation in bitcoin-wallet
getBlockHash method on a Chain object. This method is defined as a virtual method in src/interfaces/chain.h.Translation to Cap’n Proto RPC
Chain::getBlockHash virtual method is overridden by the Chain client subclass to translate the method call into a Cap’n Proto RPC call.mpgen tool from the chain.capnp file in src/ipc/capnp/.Request Preparation and Dispatch
getBlockHash method of the generated Chain client subclass in bitcoin-wallet populates a Cap’n Proto request with the height parameter, sends it to bitcoin-node process, and waits for a response.Handling in bitcoin-node
bitcoin-node process calls the getBlockHash method of the Chain server class.mpgen tool from the chain.capnp file in src/ipc/capnp/.getBlockHash method of the generated Chain server subclass in bitcoin-wallet receives a Cap’n Proto request object with the height parameter, and calls the getBlockHash method on its local Chain object with the provided height.bitcoin-wallet process.Response and Return
getBlockHash method of the generated Chain client subclass in bitcoin-wallet which sent the request now receives the response.Further improvements are possible such as:
bitcoin-node, and running indexing code in separate processes (see indexes: Stop using node internal types #24230)..capnp files from C++ interface definitions (see Interface Definition Maintenance).This modularization represents an advancement in Bitcoin Core's architecture, offering enhanced security, flexibility, and maintainability. The project invites collaboration and feedback from the community.
abstract class: A class in C++ that consists of virtual functions. In the Bitcoin Core project, they define interfaces for inter-component communication.
asynchronous I/O: A form of input/output processing that allows a program to continue other operations while a transmission is in progress.
Cap’n Proto: A high-performance data serialization and RPC library, chosen for its support for object references and bidirectional communication.
Cap’n Proto interface: A set of methods defined in Cap’n Proto to facilitate structured communication between different software components.
Cap’n Proto struct: A structured data format used in Cap’n Proto, similar to structs in C++, for organizing and transporting data across different processes.
client class (in generated code): A C++ class generated from a Cap’n Proto interface which inherits from a Bitcoin Core abstract class, and implements each virtual method to send IPC requests to another process. (see also components section)
IPC (inter-process communication): Mechanisms that enable processes to exchange requests and data.
ipc::Exception class: A class within Bitcoin Core's protocol-agnostic IPC code that is thrown by client class methods when there is an IPC error.
libmultiprocess: A custom library and code generation tool used for creating IPC interfaces and managing IPC connections.
marshalling: Transforming an object’s memory representation for transmission.
mpgen tool: A tool within the libmultiprocess suite that generates C++ code from Cap’n Proto files, facilitating IPC.
protocol-agnostic code: Generic IPC code in src/ipc/ that does not rely on Cap’n Proto and could be used with other protocols. Distinct from code in src/ipc/capnp/ which relies on Cap’n Proto.
RPC (remote procedure call): A protocol that enables a program to request a service from another program in a different address space or network. Bitcoin Core uses JSON-RPC for RPC.
server class (in generated code): A C++ class generated from a Cap’n Proto interface which handles requests sent by a client class in another process. The request handled by calling a local Bitcoin Core interface method, and the return values (if any) are sent back in a response. (see also: components section)
unix socket: Communication endpoint which is a filesystem path, used for exchanging data between processes running on the same host.
virtual method: A function or method whose behavior can be overridden within an inheriting class by a function with the same signature.
This design doc was written by @ryanofsky, who is grateful to all the reviewers who gave feedback and tested multiprocess PRs, and everyone else who's helped with this project. Particular thanks to @ariard who deeply reviewed IPC code and improved the design of the IPC library and initialization process. @jnewbery who championed the early refactoring PRs and helped guide them through development and review. @sjors who has reviewed and repeatedly tested multiprocess code, reporting many issues and helping debug them. @hebasto, @fanquake, and @maflcko who made significant improvements to the build system and fixed countless build issues. @vasild and @jamesob who were brave contributors to the libmultiprocess library. And Chaincode Labs for making this work possible. Also thanks to ChatGPT, who actually wrote most of this document (not @ryanofsky).