Back to Grype

dbtest - Database Testing Utilities

internal/dbtest/README.md

0.112.015.0 KB
Original Source

dbtest - Database Testing Utilities

This package provides utilities for building and testing grype vulnerability databases from fixture data, along with fluent assertion helpers that decouple test code from internal API shapes.

Quick Start

Using Fixtures in Tests

go
func TestVulnerabilityMatching(t *testing.T) {
    // use a fixture from your package's testdata directory
    dbtest.DBs(t, "my-fixture").Run(func(t *testing.T, db *dbtest.DB) {
        // db implements vulnerability.Provider
        vulns, err := db.FindVulnerabilities(...)
    })
}

func TestWithSharedFixture(t *testing.T) {
    // use a fixture from internal/dbtest/testdata/shared/
    dbtest.SharedDBs(t, "all").Run(func(t *testing.T, db *dbtest.DB) {
        // db.Match returns a FindingsAssertion for fluent assertions
        db.Match(t, matcher, pkg).
            SelectMatch("CVE-2024-1234").
            SelectDetailByType(match.ExactDirectMatch).
            AsDistroSearch()
    })
}

Filtering Fixtures at Test Time

go
// only include specific CVEs from a larger fixture
dbtest.SharedDBs(t, "all").
    SelectOnly("CVE-2024-1234", "CVE-2024-5678").
    Run(func(t *testing.T, db *dbtest.DB) {
        // db only contains the selected CVEs
    })

Motivation

Testing grype's vulnerability matching requires realistic database fixtures. However, real vunnel caches can contain millions of records, making them impractical for test fixtures. This package solves several problems:

  1. Focused fixtures: Extract only the specific CVEs or namespaces needed for a test scenario
  2. Deterministic tests: Create reproducible fixtures from real vunnel data
  3. Easy maintenance: Append new records to existing fixtures as test cases evolve
  4. Cross-package sharing: Share fixtures between test packages via SharedDBs()
  5. Fixture provenance: Track how fixtures were created and regenerate them when needed
  6. API-agnostic assertions: Decouple test assertions from internal data structures to survive refactors

Go API

Building Test Databases

The Builder type provides a fluent API for building test databases from fixtures:

go
// DBs looks for testdata/<name> relative to the calling test file
func DBs(t *testing.T, fixtureName string) *Builder

// SharedDBs looks for fixtures in internal/dbtest/testdata/shared/<name>
func SharedDBs(t *testing.T, fixtureName string) *Builder

// SelectOnly filters which records are included (patterns combined with OR)
func (b *Builder) SelectOnly(patterns ...string) *Builder

// Run executes a test function for each schema version
func (b *Builder) Run(fn func(t *testing.T, db *DB))

// Build returns databases without running them through t.Run
func (b *Builder) Build(schemas ...int) []*DB

The DB Type

DB wraps a vulnerability.Provider with test metadata:

go
type DB struct {
    Name          string  // e.g., "v6"
    SchemaVersion int
    Path          string
}

// implements vulnerability.Provider
func (db *DB) FindVulnerabilities(criteria ...vulnerability.Criteria) ([]vulnerability.Vulnerability, error)
func (db *DB) VulnerabilityMetadata(ref vulnerability.Reference) (*vulnerability.Metadata, error)
func (db *DB) PackageSearchNames(p grypePkg.Package) []string

// runs matcher and returns fluent assertion chain
func (db *DB) Match(t *testing.T, matcher Matcher, p grypePkg.Package) *FindingsAssertion

Fluent Assertions

The assertion helpers provide a string-based, API-agnostic way to validate match results. This design has several benefits:

  • Survives refactors: Tests don't break when internal struct shapes change (e.g., grype v1 API changes)
  • Removes boilerplate: No need to manually iterate matches/details or check types
  • Completeness checking: By default, tests fail if any matches or details are not asserted
  • Readable tests: Fluent chains clearly express what's being validated

This approach is similar to syft's pkgtest helpers - abstracting test assertions from implementation details.

Basic Usage

go
// match and assert in one fluent chain
db.Match(t, &matcher, pkg).
    SelectMatch("CVE-2024-1234").
    SelectDetailByType(match.ExactDirectMatch).
    AsDistroSearch("< 1.0.0")  // validates constraint

// or use AssertFindings directly with pre-existing matches
dbtest.AssertFindings(t, matches, pkg).
    HasCount(2).
    OnlyHasVulnerabilities("CVE-2024-1234", "CVE-2024-5678")

Completeness Checking

By default, assertions require that all matches and details are asserted. This catches cases where a matcher returns unexpected results:

go
// this will fail if there are matches other than CVE-2024-1234
db.Match(t, &matcher, pkg).
    SelectMatch("CVE-2024-1234").
    SelectDetailByType().
    AsDistroSearch()

// use SkipCompleteness() when you only care about specific matches
db.Match(t, &matcher, pkg).SkipCompleteness().
    ContainsVulnerabilities("CVE-2024-1234")  // other matches OK

Detail Assertions

After selecting a match, drill into details by type or search parameters:

go
// select by match type
findings.SelectMatch("CVE-2024-1234").
    SelectDetailByType(match.ExactDirectMatch).
    AsDistroSearch()

// select by distro (when multiple distro details exist)
findings.SelectMatch("CVE-2024-1234").
    SelectDetailByDistro("debian", "11", "< 1.0.0")

// select by CPE
findings.SelectMatch("CVE-2024-1234").
    SelectDetailByCPE("cpe:2.3:a:vendor:product:1.0:*:*:*:*:*:*:*").
    FoundCPEs("cpe:2.3:a:vendor:product:*:*:*:*:*:*:*:*")

// select by ecosystem/language
findings.SelectMatch("CVE-2024-1234").
    SelectDetailByEcosystem("python", "< 2.0.0")

Available Assertions

MethodDescription
HasCount(n)Assert exactly n matches
IsEmpty()Assert no matches
ContainsVulnerabilities(ids...)Assert these CVEs are present (others may exist)
OnlyHasVulnerabilities(ids...)Assert exactly these CVEs and no others
DoesNotHaveAnyVulnerabilities(ids...)Assert these CVEs are not present
SelectMatch(id)Select a specific match for detail assertions
HasMatchType(type)Assert at least one detail has this match type
HasOnlyMatchTypes(types...)Assert all details have one of these types
SkipCompleteness()Disable completeness checking

Extracting Fixtures from Vunnel Cache

Use FixtureExtractor to create fixtures from a vunnel data directory:

go
extractor := dbtest.NewFixtureExtractor("/path/to/vunnel/data")

// extract to new fixture
err := extractor.
    From("debian").
    Select("CVE-2024-1234", "debian:10").
    WriteTo("internal/dbtest/testdata/shared/my-fixture")

// append to existing fixture
err := extractor.
    From("rhel").
    Select("RHSA-2024:").
    AppendTo("internal/dbtest/testdata/shared/my-fixture")

// multi-provider extraction
err := extractor.
    FromMultiple().
    Provider("debian", "CVE-2024-1234").
    Provider("nvd", "CVE-2024-1234").
    WriteTo("internal/dbtest/testdata/shared/multi-provider")

CLI Tool

The manager CLI provides commands for creating and maintaining fixtures.

Extract Command

Create fixtures from vunnel SQLite caches:

bash
# extract specific CVEs
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider debian \
    --select "CVE-2024-1234" \
    --output internal/dbtest/testdata/shared/my-fixture

# extract by namespace (all CVEs in debian:11)
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider debian \
    --select "debian:11" \
    --output internal/dbtest/testdata/shared/debian11-vulns

# extract with multiple patterns (OR logic)
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider debian \
    --select "CVE-2024-1234" --select "CVE-2024-5678" \
    --output internal/dbtest/testdata/shared/specific-cves

# append to existing fixture
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider rhel \
    --select "RHSA-2024:" \
    --append internal/dbtest/testdata/shared/my-fixture

# extract from multiple providers
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider debian --provider nvd \
    --select "CVE-2024-1234" \
    --output internal/dbtest/testdata/shared/multi-provider
FlagDescription
--vunnel-dataPath to vunnel data directory (required)
--providerProvider name to extract from (repeatable)
--selectPattern for record selection (repeatable)
--outputPath for new fixture directory
--appendPath to existing fixture to append to

Status Command

Check the status of all fixtures:

bash
# show status of all fixtures
go run ./internal/dbtest/cmd/manager status

# show status of a specific fixture
go run ./internal/dbtest/cmd/manager status --fixture internal/dbtest/testdata/shared/my-fixture

Example output:

internal/dbtest/testdata/shared/all             OK (automatic, synced)
grype/matcher/dpkg/testdata/eol-debian8         OK (automatic, synced)
internal/dbtest/testdata/shared/manual-vuln     OK (manual)
internal/dbtest/testdata/shared/modified-one    CONTENT DRIFT (lock: a1b2c3d4, actual: e5f6g7h8)

Regenerate Command

Regenerate fixtures from their db.yaml configs:

bash
# dry run to see what would be regenerated
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --dry-run

# regenerate all fixtures
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data

# regenerate a specific fixture
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --fixture path/to/fixture

# force regeneration even if fixture has been modified
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --force
FlagDescription
--vunnel-dataPath to vunnel data directory (required)
--fixturePath to a specific fixture to regenerate
--search-rootRoot directory to search for fixtures (repeatable)
--forceRegenerate even if fixture has been modified
--dry-runShow what would be regenerated without making changes

Pattern Matching

Patterns use SQL LIKE matching with automatic wildcards. All patterns are wrapped with % for partial matching.

PatternMatchesDescription
CVE-2024-1234debian:10/CVE-2024-1234, ubuntu:20.04/CVE-2024-1234Matches CVE in any namespace
debian:10debian:10/CVE-2024-1234, debian:10/CVE-2024-5678Matches all CVEs in namespace
debiandebian:10/..., debian:11/...Matches all debian namespaces
RHSA-2024:rhel:8/RHSA-2024:0001, rhel:9/RHSA-2024:0002Matches all 2024 RHSAs
debian:11/CVE-2024-1234debian:11/CVE-2024-1234Exact match

Multiple --select patterns are combined with OR logic.

Fixture Structure

Extracted fixtures follow the vunnel workspace format:

fixture/
├── db.yaml               # extraction config (auto-generate flag, patterns)
├── db-lock.json          # state tracking (content hash, timestamps)
├── provider-name/
│   ├── metadata.json     # provider state (store: "flat-file")
│   └── results/
│       ├── debian@11_CVE-2024-1234.json
│       ├── debian@11_CVE-2024-5678.json
│       └── listing.xxh64
└── another-provider/
    ├── metadata.json
    └── results/
        └── ...

Note: : in identifiers is replaced with @ and / with _ in filenames.

Tracking and Regenerating Fixtures

When you extract a fixture, the manager creates two metadata files:

  • db.yaml - Intent: What extractions to perform (human-editable)
  • db-lock.json - State: What was done (machine-generated)

db.yaml

yaml
auto-generate: true
extractions:
  debian:
    - CVE-2024-1234
    - debian:10
  nvd:
    - CVE-2024-1234
FieldDescription
auto-generateIf true, fixture can be regenerated from vunnel cache
extractionsMap of provider name to list of patterns

db-lock.json

json
{
  "content_hash": "a1b2c3d4e5f6g7h8",
  "created_at": "2024-01-15T12:00:00Z",
  "regenerated_at": "2024-03-12T10:30:00Z",
  "providers": {
    "debian": {
      "vunnel_version": "[email protected]",
      "timestamp": "2024-03-11T16:25:19Z"
    }
  }
}
FieldDescription
content_hashxxh64 hash of fixture content (excludes db.yaml and db-lock.json)
created_atWhen fixture was first created
regenerated_atWhen last regenerated (omitted if never regenerated)
providers.<name>.vunnel_versionVunnel version from provider's metadata
providers.<name>.timestampProvider's data timestamp

Fixture Status

StatusDescription
OK (automatic)auto-generate: true and content hash matches lock
OK (manual)auto-generate: false - manually created, never auto-regenerated
content_driftauto-generate: true but files have changed since last generation
config_aheadauto-generate: true but config has providers not in lock
no_lockauto-generate: true but db-lock.json is missing
no_configNo db.yaml file present

Workflows

Creating a New Fixture

bash
# 1. extract (db.yaml and db-lock.json created automatically)
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider debian --select "CVE-2024-1234" \
    --output internal/dbtest/testdata/shared/my-fixture

# 2. optionally append more providers/CVEs
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider nvd --select "CVE-2024-1234" \
    --append internal/dbtest/testdata/shared/my-fixture

# 3. commit
git add internal/dbtest/testdata/shared/my-fixture/
git commit -m "add my-fixture test fixture"

Creating a Manual Fixture

Manual fixtures are never automatically regenerated, useful when you need custom modifications:

bash
# 1. extract base
go run ./internal/dbtest/cmd/manager extract \
    --vunnel-data ~/vunnel/data \
    --provider debian --select "CVE-2024-1234" \
    --output internal/dbtest/testdata/shared/manual-fixture

# 2. manually modify files as needed
vim internal/dbtest/testdata/shared/manual-fixture/debian/results/...

# 3. edit db.yaml to set auto-generate: false
# (prevents automatic regeneration)

# 4. commit
git add internal/dbtest/testdata/shared/manual-fixture/
git commit -m "add manual-fixture with custom modifications"

Regenerating Fixtures

When vunnel data is updated, regenerate fixtures to get fresh vulnerability data:

bash
# 1. check current status
go run ./internal/dbtest/cmd/manager status

# 2. dry run to see what would happen
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data --dry-run

# 3. regenerate (skips manual and modified fixtures)
go run ./internal/dbtest/cmd/manager regenerate --vunnel-data ~/vunnel/data

# 4. run tests to verify
go test ./...