ContextQMD
Back to Goose

Testing Recipe Schema Tracking Automation

documentation/automation/recipe-schema-tracking/TESTING.md

1.33.16.1 KB
Original Source

Testing Recipe Schema Tracking Automation

This guide covers how to test the recipe schema tracking automation both locally and via GitHub Actions.

Local Testing

Prerequisites

  • goose CLI installed
  • jq installed (for JSON processing)
  • Git repository with goose source code

Manual Pipeline Execution

Test the complete pipeline locally:

bash
cd documentation/automation/recipe-schema-tracking

# Test with no changes expected
./scripts/run-pipeline.sh v1.14.0 v1.15.0

# Test with changes expected
./scripts/run-pipeline.sh v1.9.0 v1.15.0

Individual Script Testing

Test each script independently:

bash
# Extract schema from a version
./scripts/extract-schema.sh v1.15.0 > output/test-schema.json

# Extract validation structure
./scripts/extract-validation-structure.sh v1.15.0 > output/test-validation.json

# Compare two validation structures
./scripts/diff-validation-structures.sh output/old.json output/new.json > output/test-changes.json

Recipe Testing

Test the AI recipes:

bash
# Generate change documentation
cd output
goose run --recipe ../recipes/synthesize-validation-changes.yaml

# Update recipe-reference.md
export RECIPE_REF_PATH=/path/to/recipe-reference.md
goose run --recipe ../recipes/update-recipe-reference.yaml

GitHub Actions Testing

Test in Your Fork

The workflow can be tested in your fork without affecting the upstream repository.

Step 1: Push Branch to Fork

bash
git push origin your-branch-name

Step 2: Enable GitHub Actions

  1. Go to your fork on GitHub
  2. Click "Actions" tab
  3. Enable workflows if prompted

Step 3: Run Workflow Manually

  1. Click "Update Recipe Documentation" workflow (docs-update-recipe-ref.yml)
  2. Click "Run workflow" button
  3. Select your branch
  4. Configure inputs (see test scenarios below)
  5. Click "Run workflow"

Test Scenarios

Scenario 1: Dry-Run with No Changes

Purpose: Verify the workflow runs successfully when no changes are detected.

Inputs:

  • old_version: v1.14.0
  • new_version: v1.15.0
  • dry_run: true

Expected Results:

  • ✅ Workflow completes successfully
  • ✅ "No changes detected" message in summary
  • ✅ Artifacts uploaded with extraction results
  • ✅ No PR created

Scenario 2: Dry-Run with Changes

Purpose: Test change detection and documentation generation without creating a PR.

Inputs:

  • old_version: v1.9.0
  • new_version: v1.15.0
  • dry_run: true

Expected Results:

  • ✅ Workflow detects changes (4 validation rules, 1 field removal)
  • ✅ Generates validation-changes.md with documentation
  • ✅ Updates recipe-reference.md
  • ✅ Artifacts uploaded with all generated files
  • ✅ No PR created (dry-run mode)

Review Artifacts:

  1. Download artifact zip from workflow run
  2. Check validation-changes.md - should document all changes
  3. Check update-summary.md - should show what was updated
  4. Compare updated recipe-reference.md with original

Scenario 3: Full Run with PR Creation

Purpose: Test end-to-end including PR creation.

Inputs:

  • old_version: v1.9.0
  • new_version: v1.15.0
  • dry_run: false

Expected Results:

  • ✅ Workflow runs successfully
  • ✅ Creates PR: docs/recipe-reference-v1.15.0
  • ✅ PR contains updated recipe-reference.md
  • ✅ PR description includes change summary and checklist

Review PR:

  1. Check only recipe-reference.md was modified
  2. Verify changes match dry-run artifacts
  3. Confirm no unintended modifications
  4. Test documentation renders correctly

Scenario 4: Auto-Detection

Purpose: Test automatic version detection (simulates production mode).

Inputs:

  • old_version: (leave empty)
  • new_version: (leave empty)
  • dry_run: true

Expected Results:

  • ✅ Auto-detects two most recent releases
  • ✅ Compares them automatically
  • ✅ Uploads artifacts

Reviewing Workflow Results

Check Workflow Summary

Each workflow run provides a summary with:

  • Version comparison performed
  • Whether changes were detected
  • Dry-run mode status
  • Links to artifacts

Download and Review Artifacts

Artifacts include:

  • old-validation-structure.json - Extracted from old version
  • new-validation-structure.json - Extracted from new version
  • validation-changes.json - Structured diff
  • validation-changes.md - Human-readable changes
  • update-summary.md - Documentation update summary
  • pipeline.log - Full pipeline execution log

Check Workflow Logs

For detailed debugging:

  1. Click on the workflow run
  2. Click on the "Update Recipe Documentation" job
  3. Expand each step to see detailed logs
  4. Look for error messages or unexpected behavior

Troubleshooting

Workflow doesn't appear in Actions tab

  • Verify workflow file is in .github/workflows/
  • Check file has .yml or .yaml extension
  • Ensure GitHub Actions is enabled in fork

"No changes detected" when expecting changes

  • Check artifact validation-changes.json to see what was compared
  • Verify versions exist: git tag | grep v1.15.0
  • Review extraction script logs

goose CLI installation fails

  • Workflow installs from current repository
  • Ensure crates/goose-cli builds successfully
  • Check Rust toolchain installation

PR creation fails

  • Verify workflow has required permissions
  • Check branch name doesn't already exist
  • Review workflow logs for error messages

Production Deployment

Once testing is complete:

  1. Merge automation to main: Create PR for documentation/automation/recipe-schema-tracking/
  2. Merge baseline docs: Create PR for revised recipe-reference.md
  3. Merge workflow: Create PR for .github/workflows/docs-update-recipe-ref.yml
  4. Enable release trigger: Uncomment release: section in workflow

After deployment, the workflow will automatically:

  • Trigger on new releases
  • Compare with previous release
  • Create PR if changes detected
  • Notify team for review