ContextQMD
Back to Codexbar

CodexBar Live QA

.agents/skills/qa-test/SKILL.md

0.32.24.5 KB
Original Source

CodexBar Live QA

Use for live provider testing, release smoke tests, menu verification, or debugging “provider works/fails” reports.

Rules

  • Work from the CodexBar repo checkout.
  • Use the packaged CLI first: CodexBar.app/Contents/Helpers/CodexBarCLI.
  • Do not use CodexBar.app/Contents/MacOS/codexbar; that is the app binary and may appear to hang as a CLI.
  • Never run broad env, set, or secret regex dumps.
  • Use $one-password for secrets: all op commands inside one persistent tmux session, service account first, no raw secret output.
  • Treat browser-cookie/keychain flows as prompt-risky. Prefer CLI/API-token checks and KeychainNoUIQuery-safe tests unless the user explicitly requested live UI.
  • For current API behavior, browse official provider docs only.

CLI Matrix

Run the bundled script:

bash
.agents/skills/qa-test/scripts/live_provider_matrix.sh --enabled

Useful modes:

bash
.agents/skills/qa-test/scripts/live_provider_matrix.sh --provider all
.agents/skills/qa-test/scripts/live_provider_matrix.sh --providers openai,zai,deepseek
.agents/skills/qa-test/scripts/live_provider_matrix.sh --default

Interpretation:

  • --enabled asks CodexBarCLI config providers for enabled providers, honoring CODEXBAR_CONFIG and default toggles.
  • --default runs the app-facing default command with no provider override.
  • --provider all forces every registered provider and is expected to fail for providers without sessions/keys.
  • A green app config needs --enabled and --default clean; --provider all is a discovery/triage tool.

Config QA

Validate config:

bash
CodexBar.app/Contents/Helpers/CodexBarCLI config validate
stat -f '%Lp %N' "$HOME/.codexbar/config.json"

Redact config shape:

bash
jq '(.providers // []) |= map(.apiKey = (if .apiKey then "<redacted>" else .apiKey end) |
  .secretKey = (if .secretKey then "<redacted>" else .secretKey end) |
  .cookieHeader = (if .cookieHeader then "<redacted>" else .cookieHeader end) |
  (if .id == "stepfun" and has("region") then .region = "<redacted>" else . end) |
  .tokenAccounts = (if .tokenAccounts then (.tokenAccounts | .accounts = (.accounts | map(.token = "<redacted>"))) else .tokenAccounts end))' \
  "$HOME/.codexbar/config.json"

Before editing config, make a backup:

bash
cp "$HOME/.codexbar/config.json" "$HOME/.codexbar/config.pre-qa-$(date +%Y%m%d%H%M%S).json"
chmod 600 "$HOME/.codexbar"/config.pre-qa-*.json

Live Menu QA

Use Peekaboo after CLI checks:

bash
pkill -x CodexBar || pkill -f 'CodexBar.app/Contents/MacOS/CodexBar' || true
open -n "$PWD/CodexBar.app"
peekaboo menu list-all --json | rg -i 'codexbar'
peekaboo menu click-extra --title codexbar-merged --json
screencapture -x /tmp/codexbar-live-menu.png

Crop top-right menu if needed:

bash
sips --cropToHeightWidth 900 340 --cropOffset 20 2650 /tmp/codexbar-live-menu.png \
  --out /tmp/codexbar-live-menu-crop.png >/dev/null

Verify visually with view_image. Confirm provider tabs/rows match enabled config and no failing provider dominates the first screen.

Browser Use

Use $browser-use only when a logged-in dashboard, API key page, or provider docs need browser/profile state.

Existing Chrome path:

bash
mcporter call chrome-devtools.list_pages --args '{}' --output text
mcporter call chrome-devtools.navigate_page --args '{"url":"https://provider.example"}' --output text
mcporter call chrome-devtools.take_snapshot --args '{}' --output text

If Browser Use is unavailable, say so and use web search for public official docs; do not substitute isolated Playwright for login/profile-dependent pages.

Fix Triage

  • Missing auth/session: configure key/session if available; otherwise leave provider disabled or report blocked auth.
  • Wrong provider API/spec: inspect official docs, then patch fetcher/settings/tests.
  • Provider key exists but live API rejects it: keep key stored if useful, disable provider if the menu would show a persistent error.
  • User-facing behavior changes need CHANGELOG.md.
  • Code fixes need focused tests, make check, $autoreview, and live CLI proof before landing.

Known CodexBar QA Notes

  • OpenAI Admin API key is the useful usage provider key. Project OPENAI_API_KEY values can fail legacy credit-balance fallback with 403.
  • Deepgram usage requires a key/project with Management API permissions; transcription-only keys can return 403.
  • Groq usage uses the Prometheus metrics API, not ordinary inference endpoints.
  • MiniMax pay-as-you-go API keys and Token Plan/Coding Plan keys are different; wrong key kind can leave usage unavailable.