testing/github-12069/steps.md
As this PR introduces the change that moves the default disk buffer implementation from disk v1 to
disk v2, we similarly add the logic to automatically migrate users' disk buffers from v1 to v2 when
Vector starts.
We do so to ensure that users do not upgrade Vector and have their old buffers fall by the wayside unbeknownst to them, as it is likely they may unintentionally upgrade such that they do not see any warning/error logs that we emit to indicate that they must first drain their old buffer before upgrading, and so on.
Additionally, we did not change the on-disk format for disk_v2 between 0.20.0 and now, so we
should also be able to create a disk_v2 buffer with 0.20.0 and continue using it as a disk
buffer with a binary built from this PR.
Start by grabbing a Vector binary for 0.20.0, and build one from the PR branch. We'll use a simple configuration that will read records from stdin and attempt to send them to an HTTP sink.
For the HTTP sink, we'll have one configuration that uses an invalid port and another, an identical
version, which has the correct port. Since Vector will retry "connection refused" errors, and retry
them infinitely, the messages will never be acknowledged, which ensures they remain in the buffer.
For the purposes of verifying that the same data that went in is still present after migrating the
disk v1 buffer, etc, we can use dummyhttp (a Rust project for serving up a simplistic HTTP
endpoint that can be configured to respond a certain way) to listen on the port and inspect the HTTP
request made by the sink.
For generating logs so we can fill up our disk buffers to their limit, we'll use flog
which is a handy tool for generating fake logs, and a lot of them very quickly.
Ensure that records written to a v1 disk buffer from a version of Vector without this change can be read back after being migrated:
Start dummyhttp listening on the relevant port.
Run the 0.20.0 binary, using the "wrong" configuration, with a clean data directory. The
five-lines-first file should be piped to STDIN.
Stop Vector.
Run the PR binary, using the "wrong" configuration. The five-lines-second file should be
piped to STDIN.
Observe that the old buffer has been migrated, and that five records have been migrated: the
entries from five-lines-first.
Stop Vector.
Run the PR binary, using the "right" configuration, and ensure that it reads the records from
the buffer and sends them to the HTTP sink. The five-lines-second file should be piped to
STDIN.
Overall, there should be fifteen records: all five from the run using Vector 0.20.0, five from the run of the PR binary with the wrong configuration, and five from this run.
Ensure that records written to a v2 disk buffer from a version of Vector without this change can be read back after being migrated:
Start dummyhttp listening on the relevant port.
Run the 0.20.0 binary, using the "wrong disk v2" configuration, with a clean data directory.
The five-lines-first file should be piped to STDIN.
Stop Vector.
Run the PR binary, using the "wrong" configuration. The five-lines-second file should be
piped to STDIN.
Stop Vector.
Run the PR binary, using the "right" configuration, and ensure that it reads the records from
the buffer and sends them to the HTTP sink. The five-lines-second file should be piped to
STDIN.
Overall, there should be fifteen records: all five from the run using Vector 0.20.0, five from the run of the PR binary with the wrong configuration, and five from this run.
Ensure that records written to a v1 disk buffer from a version of Vector without this change can be migrated over when the new v2 buffer ends up exceeding the configured maximum buffer size:
flog should be piped to STDIN.