I/O Tester utility

The I/O Tester utility, io_tester generates a user-defined I/O pattern spanning one of multiple shards that is designed to simulate the I/O behavior of a complex Seastar application.

Running I/O tester:

I/O tester takes the same options as Seastar, and those options may be used to test the behavior of I/O under the circumnstances established by those options. For instance, one may adjust the --task-quota-ms option to see if that affects higher percentile latencies.

Aside from the usual seastar options, I/O tester accepts the following options:

duration: for how long to run the evaluation,
storage: a directory or a block device where to execute the test (it must be on XFS),
conf: the path to a YAML file describing the evaluation,
keep-files: a flag that indicates keeping test files - next run may re-use them.

Describing the evaluation

The evaluation is described in a YAML file that contains multiple classes. Each class spans jobs of similar characteristics in different shards and (for now) all jobs run concurrently.

The YAML file contains a list of maps where each element of the list describes a class. A class has some properties that are common to all elements of the class, and a nested map that contain properties of a job (class instance in a shard)

For example:

- name: big_writes
  type: seqread
  shards: all
  shard_info:
    parallelism: 10
    reqsize: 256kB
    shares: 10
    think_time: 0

- name: vectorized_writes
  type: randwrite
  shards: all
  shard_info:
    parallelism: 10
    reqsize: 64kB
    vectorized: true
    iov_count: 4        # 4 segments of 16kB each
    shares: 10

name: mandatory property, a string that identifies jobs of this class
type: mandatory property, one of seqread, seqwrite, randread, randwrite, append, cpu, unlink
shards: mandatory property, either the string "all" or a list of shards where this class should place jobs.
data_size: optional property, used to divide the available disk space between workloads. Each shard inside the workload uses its portion of the assigned space. If not specified 1GB is used.
extent_allocation_size_hint: optional property, allows setting the hint for allocation of extents for files. If not specified, then the size of file is used as hint.
files_count: optional property, relevant only for unlink job class - in such case it is required. Describes the number of files that need to be created during startup to be unlinked during evaluation. Describes files count per shard.

NOTE: the actual file size is always aligned to 1MB. NOTE: if not properly aligned, then the extent allocation size hint is aligned to 128kB by seastar.

The properties under shard_info represent properties of the job that will be replicated to each shard. All properties under shard_info are optional, and in case not specified, defaults are used.

parallelism: the amount of parallel requests this job will generate in a specific shard. Requests can be either active or thinking (see think_time)
limit: the maximum number of requests to send in this job. If not set, job sends requests throughout the whole duration
rps: the requests-per-second rate to apply to sending fibers. If unset, each fiber sends new request as soon as previous one completes
batch: the number of requests sent at a time when the rps mode is used. It's 1 by default
reqsize : (I/O loads only) the size of requests generated by this job
shares : how many shares requests in this job will have in the scheduler
class: name of the job to share the sched class with (used if shares is not set)
think_time: how long to wait before submitting another request in this job once one finishes.
execution_time: (cpu loads only) for how long to execute a CPU loop
vectorized: (I/O loads only) if true, use scatter-gather I/O with multiple iovec segments per request
iov_count: (I/O loads only) number of iovec segments when vectorized is true. The reqsize is split into iov_count uniform segments. Each segment must be at least 4096 bytes and properly aligned.

Example output

      Creating initial files...
      Starting evaluation...
      Shard  0
       Class 0(big_writes: 10 shares, 262144-byte SEQ WRITE, 10 concurrent requests, NO think time)
        Throughput         :   436556 KB/s
        Lat average        :     5847 usec
        Lat quantile=  0.5 :     2678 usec
        Lat quantile= 0.95 :    13029 usec
        Lat quantile= 0.99 :    20835 usec
        Lat quantile=0.999 :   246090 usec
        Lat max            :   450785 usec

Future

Some ideas for extending I/O tester:

allow properties like think time, request size, etc, to be specified as distributions instead of a fixed number
allow classes to have class-wide properties. For instance, we could define a class with parallelism of 100, and distribute those 100 requests over all shards in which this class is placed
allow some jobs to be executed sequentially in relationship to others, so we can have preparation jobs.
support other types, like sync, etc.
provide functionality similar to diskplorer.