BulletproofTxManager Architecture Overview

Diagrams

Finite state machine

evm.txes.state

unstarted | | v in_progress (only one per key) |
|
v v fatal_error unconfirmed | ^ | | v | confirmed

eth_tx_attempts.state

in_progress | ^ | | v | broadcast

Data structures

Key:

⚫️ - has never been broadcast to the network

🟠 - may or may not have been broadcast to the network

🔵 - has definitely been broadcast to the network

EB - EthBroadcaster

EC - EthConfirmer

evm.txes has five possible states:

• EB ⚫️ unstarted
• EB 🟠 in_progress
• EB/EC ⚫️ fatal_error
• EB/EC 🔵 unconfirmed
• EB/EC 🔵 confirmed

eth_tx_attempts has two possible states:

• EB/EC 🟠 in_progress
• EB/EC 🔵 broadcast

An attempt may have 0 or more eth_receipts indicating that the transaction has been mined into a block. This block may or may not exist as part of the canonical longest chain.

Components

BulletproofTxManager is split into three components, each of which has a clearly delineated set of responsibilities.

EthTx

Conceptually, EthTx defines the transaction.

EthTx is responsible for generating the transaction criteria and inserting the initial unstarted row into the evm.txes table.

EthTx guarantees that the transaction is defined with the following criteria:

• From address
• To address
• Encoded payload
• Value (eth)
• Gas limit

Only one transaction may be created per EthTx task.

EthTx should wait until it's transaction confirms before marking the task as completed.

EthBroadcaster

Conceptually, EthBroadcaster assigns a nonce to a transaction and ensures that it is valid. It alone maintains the next usable sequence for a transaction.

EthBroadcaster monitors evm.txes for transactions that need to be broadcast, assigns nonces and ensures that at least one eth node somewhere has placed the transaction into its mempool.

It does not guarantee eventual confirmation!

A whole host of other things can subsequently go wrong such as transactions being evicted from the mempool, eth nodes crashing, netsplits between eth nodes, chain re-orgs etc. Responsibility for ensuring eventual inclusion into the longest chain falls on the shoulders of EthConfirmer.

EthBroadcaster makes the following guarantees:

• A gapless, monotonically increasing sequence of nonces for evm.txes (scoped to key).
• Transition of evm.txes from unstarted to either fatal_error or unconfirmed.
• If final state is fatal_error then the nonce is unassigned, and it is impossible that this transaction could ever be mined into a block.
• If final state is unconfirmed then a saved eth_transaction_attempt exists.
• If final state is unconfirmed then an eth node somewhere has accepted this transaction into its mempool at least once.

EthConfirmer must serialize access on a per-key basis since nonce assignment needs to be tightly controlled. Multiple keys can however be processed in parallel. Serialization is enforced with an advisory lock scoped to the key.

EthConfirmer

Conceptually, EthConfirmer adjusts the gas price as necessary to get a transaction mined into a block on the longest chain.

EthConfirmer listens to new heads and performs four separate tasks in sequence every time we become aware of a longer chain.

1. Mark "broadcast before"

When we receive a block we can be sure that any currently unconfirmed transactions were broadcast before this block was received, so we set broadcast_before_block_num on all transaction attempts made since we saw the last block.

It is important to know how long a transaction has been waiting for inclusion, so we can know for how many blocks a transaction has been waiting for inclusion in order to decide if we need to bump gas.

2. Check for receipts

Find all unconfirmed transactions and ask the eth node for a receipt. If there is a receipt, we save it and move this transaction into confirmed state.

3. Bump gas if necessary

Find all unconfirmed transactions where all attempts have remained unconfirmed for more than ETH_GAS_BUMP_THRESHOLD blocks. Create a new eth_transaction_attempt for each, with a higher gas price.

4. Re-org protection

Find all transactions confirmed within the past ETH_FINALITY_DEPTH blocks and verify that they have at least one receipt in the current longest chain. If any do not, then rebroadcast those transactions.

EthConfirmer makes the following guarantees:

• All transactions will eventually be confirmed on the canonical longest chain, unless a reorg occurs that is deeper than ETH_FINALITY_DEPTH blocks.
• In the case that an external wallet used the nonce, we will ensure that a transaction exists at this nonce up to a depth of ETH_FINALITY_DEPTH blocks but it most likely will not be the transaction in our database.

Note that since checking for inclusion in the longest chain can now be done cheaply, without any calls to the eth node, ETH_FINALITY_DEPTH can be set to something quite large without penalty (e.g. 50 or 100).

EthBroadcaster runs are designed to be serialized. Running it concurrently with itself probably can't get the data into an inconsistent state, but it might hit database conflicts or double-send transactions. Serialization is enforced with an advisory lock.

Head Tracker limitations

The design of EthConfirmer relies on an unbroken chain of heads in our database. If there is a break in the chain of heads, our re-org protection is limited to this break.

For example if we have heads at heights:

1

2

4

Then a reorg that happened at block height 3 or above will not be detected and any transactions mined in those blocks may be left erroneously marked as confirmed.

Currently, the design of the head tracker opens us up to gaps in the head sequence. This can occur in several scenarios:

1. CL Node goes offline for more than one or two blocks
2. Eth node is behind a load balancer and gets switched out for one that has different block timing
3. Websocket connection is broken and resubscribe does not occur right away

For this reason, I propose that follow-up work should be undertaken to ensure that the head tracker has some facility for backfilling heads up toETH_FINALITY_DEPTH.