Headers Downloader And Stage

Explanation of how the Reth Headers Downloader and Stage works

• We started off with sketching out a generic header downloader interface, so that we can support multiple downloading strategies implementing the interface. See reth#58 and reth#118.
• First, we implemented the reverse linear download. It received the current chain tip and local head as arguments and requested blocks in batches starting from the tip, and retried on request failure. See reth#58 and reth#119.
• The first complete implementation of the headers stage was introduced in reth#126. The stage looked up the local head & queried the consensus for the chain tip and queried the downloader passing them as arguments. After the download finished, the stage would proceed to insert headers in the ascending order by appending the entries to the corresponding tables.
• The original downloader was refactored in reth#249 to return a Future which would resolve when either the download is completed or the error occurred during polling. This future kept a pointer to a current request at any time, allowing to retry the request in case of failure. The insert logic of the headers stage remained unchanged.
  • NOTE: Up to this point the headers stage awaited the full range of blocks (from local head to tip) to be downloaded before proceeding to insert.
• reth#296 introduced the Stream implementation of the download as well as the commit threshold for the headers stage. The Stream implementation yields headers as soon as they are received and validated. It dispatches the request for the next header batch until the head is reached. The headers stage now has a configurable commit threshold which allows configuring the insert batch size. With this change, the headers stage no longer waits for the download to be complete, but rather collects the headers from the stream up to the commit threshold parameter. After collecting, the stage proceeds to insert the batch. The process is repeated until the stream is drained. At this point, we populated all tables except for HeadersTD since it has to be computed in a linear ascending order. The stage starts walking through the populated headers table and computes & inserts new total difficulty values.
• This header implementation is unique because it is implemented as a Stream, it yields headers as soon as they become available (contrary to waiting for download to complete), and it keeps only one header in buffer (required to form the next header request) .