Back to Promptfoo

PR 1: Foundation + List UI (Revised Plan)

docs/plans/pr1-foundation-with-list-ui.md

0.121.920.2 KB
Original Source

PR 1: Foundation + List UI (Revised Plan)

Executive Summary

Original PR 1: Infrastructure-only, no user-facing value Revised PR 1: Infrastructure + working promptfoo list command with Ink UI

MetricOriginalRevisedDelta
Source Lines~1,070~2,050+980
Test Lines~294~470+176
Total Lines~1,364~2,520+1,156
User ValueNoneInteractive list UI+++
Risk LevelLowLow=
Review Time1-2 hours2-3 hours+1 hour

User Experience After PR 1

Before (Current Main Branch)

bash
$ promptfoo list

┌──────────────────────────────────┬───────────────────────────┬────────────┬──────────────┐
│ eval id                          │ description               │ prompts    │ vars         │
├──────────────────────────────────┼───────────────────────────┼────────────┼──────────────┤
│ eval-2024-01-15T10:30:00         │ API testing               │ a1b2c3     │ query, model │
│ eval-2024-01-14T15:45:00         │ Prompt comparison         │ d4e5f6     │ input        │
└──────────────────────────────────┴───────────────────────────┴────────────┴──────────────┘

Run promptfoo show eval <id> to see details of a specific evaluation.

After PR 1 (Interactive UI)

bash
$ promptfoo list

┌─────────────────────────────────────────────────────────────────────────────────┐
│ Evaluations (42 items)                                                          │
├─────────────────────────────────────────────────────────────────────────────────┤
│ Search: (press / to search)                                                     │
├─────────────────────────────────────────────────────────────────────────────────┤
│ ▶ eval-2024-01-15T10:30  API testing for new endpoints      today               │
│   eval-2024-01-14T15:45  Prompt comparison study            yesterday           │
│   eval-2024-01-13T09:00  Redteam security scan              2d ago      redteam │
│   eval-2024-01-12T14:20  Model benchmarking                 3d ago              │
│   eval-2024-01-11T11:00  Customer support prompts           4d ago              │
│   ...                                                                           │
├─────────────────────────────────────────────────────────────────────────────────┤
│ Showing 1-10 of 42 (more available)                                             │
├─────────────────────────────────────────────────────────────────────────────────┤
│ ↑↓/jk: navigate | Enter: select | /: search | r: refresh | q: quit             │
└─────────────────────────────────────────────────────────────────────────────────┘

Key Features

FeatureDescription
Opt-InEnable via PROMPTFOO_ENABLE_INTERACTIVE_UI=true
Keyboard NavigationArrow keys, j/k (vim), Page Up/Down, g/G
SearchPress / to filter evals by description
PaginationAuto-loads more items when scrolling
Visual IndicatorsHighlighted selection, relative dates, redteam badges
Graceful FallbackNon-TTY environments fall back to table output

File Inventory

Foundation Files (Original PR 1)

FileLinesPurpose
src/ui/constants.ts270UI configuration constants
src/ui/interactiveCheck.ts96TTY/CI detection
src/ui/render.ts216Lazy Ink rendering
src/ui/index.ts116Module barrel exports
src/ui/noninteractive/index.ts14Non-interactive exports
src/ui/noninteractive/progress.ts176Plain-text progress
src/ui/noninteractive/textOutput.ts151Structured text output
src/cliState.ts31CLI state tracking
Subtotal1,070

List UI Files (New for Option 2)

FileLinesPurpose
src/ui/list/ListApp.tsx485Main List React component
src/ui/list/listRunner.tsx115Runner with dynamic imports
src/ui/list/index.ts24Module exports
src/ui/init/components/shared/TextInput.tsx184Shared text input component
Subtotal808

Command Integration

FileLines ChangedPurpose
src/commands/list.ts+170Ink UI integration
Subtotal170

Test Files

FileLinesPurpose
test/ui/render.test.ts112Render utility tests
test/ui/noninteractive.test.ts182Non-interactive tests
test/ui/list/listRunner.test.ts173List runner tests
Subtotal467

Config Changes

FileChange
src/envars.tsAdd 3 environment variables
tsconfig.jsonAdd "jsx": "react-jsx"
vitest.config.tsAdd .tsx file support
package.jsonAdd Ink/React dependencies
package-lock.jsonDependency tree

Complete File List for PR 1

text
# Foundation (Core Infrastructure)
src/ui/constants.ts
src/ui/interactiveCheck.ts
src/ui/render.ts
src/ui/index.ts
src/ui/noninteractive/index.ts
src/ui/noninteractive/progress.ts
src/ui/noninteractive/textOutput.ts
src/cliState.ts

# List UI (User-Facing Feature)
src/ui/list/ListApp.tsx
src/ui/list/listRunner.tsx
src/ui/list/index.ts
src/ui/init/components/shared/TextInput.tsx

# Command Integration
src/commands/list.ts

# Tests
test/ui/render.test.ts
test/ui/noninteractive.test.ts
test/ui/list/listRunner.test.ts

# Config
src/envars.ts
tsconfig.json
vitest.config.ts
package.json
package-lock.json

# Total: 18 files, ~2,520 lines

Architecture

Dependency Flow

text
┌─────────────────────────────────────────────────────────────────┐
│                    src/commands/list.ts                         │
│                           │                                     │
│                           ▼                                     │
│              ┌────────────────────────┐                         │
│              │   src/ui/list/         │                         │
│              │   ├── listRunner.tsx   │ ← Dynamic import entry  │
│              │   ├── ListApp.tsx      │ ← React component       │
│              │   └── index.ts         │                         │
│              └────────────┬───────────┘                         │
│                           │                                     │
│              ┌────────────┴───────────┐                         │
│              ▼                        ▼                         │
│  ┌─────────────────────┐  ┌──────────────────────────┐          │
│  │ src/ui/render.ts    │  │ src/ui/init/components/  │          │
│  │ (lazy Ink loading)  │  │ shared/TextInput.tsx     │          │
│  └─────────┬───────────┘  └──────────────────────────┘          │
│            │                                                    │
│            ▼                                                    │
│  ┌─────────────────────────────────┐                            │
│  │ src/ui/interactiveCheck.ts      │                            │
│  │ (TTY/CI detection)              │                            │
│  └─────────────────────────────────┘                            │
│                                                                 │
│  ┌─────────────────────────────────┐                            │
│  │ src/ui/constants.ts             │ ← Used by all components   │
│  └─────────────────────────────────┘                            │
└─────────────────────────────────────────────────────────────────┘

Dynamic Import Pattern

The key pattern that prevents bundle bloat:

typescript
// src/ui/list/listRunner.tsx
export async function runInkList(options: ListRunnerOptions): Promise<ListResult> {
  // Dynamic imports - only loaded when called
  const [React, { renderInteractive }, { ListApp }] = await Promise.all([
    import('react'),
    import('../render'),
    import('./ListApp'),
  ]);

  // ... render the component
}

Command Integration Pattern

typescript
// src/commands/list.ts
import { runInkList, shouldUseInkList } from '../ui/list';

.action(async (cmdObj) => {
  // Check if Ink UI should be used
  if (cmdObj.interactive && shouldUseInkList()) {
    // Transform data for Ink UI
    const items: EvalItem[] = evals.map(evl => ({ ... }));

    // Run interactive UI
    const result = await runInkList({
      resourceType: 'evals',
      items,
    });

    // Handle selection
    if (result.selectedItem) {
      logger.info(`Selected: ${result.selectedItem.id}`);
    }
    return;
  }

  // Fall back to table output
  logger.info(wrapTable(tableData, columnWidths));
});

Environment Variable

Interactive UI is opt-in. Users must explicitly enable it:

VariableDefaultPurpose
PROMPTFOO_ENABLE_INTERACTIVE_UIfalseEnable Ink-based interactive UI

How It Works

  1. If PROMPTFOO_ENABLE_INTERACTIVE_UI=true → Check TTY
  2. If stdout is a TTY → Use interactive UI
  3. Otherwise → Use table output (non-interactive fallback)
bash
# Enable interactive UI
PROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list evals

# Default behavior (table output)
promptfoo list evals

Testing Strategy

Unit Tests

TestFileCoverage
TTY detectionrender.test.tscanUseInteractiveUI()
Opt-in checkrender.test.tsisInteractiveUIEnabled()
Combined checkrender.test.tsshouldUseInkUI()
List runnerlistRunner.test.tsshouldUseInkList(), runInkList()

Manual Testing Matrix

ScenarioCommandExpected
Default (no env var)promptfoo list evalsTable output
Interactive enabled in TTYPROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list evalsInteractive UI
Interactive enabled, pipedPROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list | catTable output (no TTY)
List prompts (enabled)PROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list promptsInteractive UI (prompts view)
List datasets (enabled)PROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list datasetsInteractive UI (datasets)

Terminal Compatibility

TerminalStatus
iTerm2 (macOS)Required
Terminal.app (macOS)Required
VS Code integrated terminalRequired
Windows TerminalRecommended
SSH sessionShould fallback
tmux/screenShould work

Acceptance Criteria

Infrastructure

  • shouldUseInkUI() returns false by default (opt-in)
  • shouldUseInkUI() returns true when PROMPTFOO_ENABLE_INTERACTIVE_UI=true + TTY
  • renderInteractive() dynamically imports Ink
  • Signal handlers set correct exit codes (130, 143)

List UI

  • promptfoo list evals shows table by default
  • promptfoo list evals with env var shows interactive UI in TTY
  • promptfoo list prompts with env var shows interactive prompt list
  • promptfoo list datasets with env var shows interactive dataset list
  • Arrow keys / j/k navigate the list
  • Enter selects an item and shows ID
  • / activates search mode
  • q or Escape exits
  • r refreshes the list
  • Piped output falls back to table (even with env var)

Build & Test

  • npm run build succeeds
  • npm run lint passes
  • npm test passes
  • No bundle size regression for library users

Git Commands to Create PR 1

bash
# Start from the feature branch
git checkout ink-ui
git pull origin ink-ui

# Create PR 1 branch
git checkout -b ink-ui/foundation-with-list

# If using cherry-pick approach, cherry-pick relevant commits
# Or use git reset/checkout to include only PR 1 files

# Verify files match the inventory
git diff --name-only origin/main...HEAD

# Run tests
npm run build && npm test

# Push branch
git push -u origin ink-ui/foundation-with-list

# Create PR
gh pr create --title "feat(ui): Add Ink-based interactive list UI" \
  --body "## Summary
- Add foundation for Ink-based CLI UI (opt-in)
- Implement interactive \`promptfoo list\` command
- Support keyboard navigation, search, pagination
- Graceful fallback for non-TTY environments

## Usage
\`\`\`bash
# Enable interactive UI (opt-in)
PROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list evals

# Default behavior (table output)
promptfoo list evals
\`\`\`

## Environment Variable
- \`PROMPTFOO_ENABLE_INTERACTIVE_UI=true\` - Enable Ink-based interactive UI

## Test Plan
- [ ] Test with env var in iTerm2
- [ ] Test with env var in VS Code terminal
- [ ] Test piped output falls back to table
- [ ] Test default behavior (table output)
"

Rollback Procedures

Quick Disable (No Code Change)

Since interactive UI is opt-in, simply don't set the environment variable. Default behavior is table output (non-interactive).

Code Revert

bash
# Revert the entire PR
git revert <pr1-merge-commit>

# Or revert just the list.ts changes to restore old behavior
git checkout main -- src/commands/list.ts

Risk Assessment

RiskLikelihoodImpactMitigation
Ink crashes in edge caseLowLowFalls back to table
TTY detection wrongLowLowOpt-in only, default is table output
Performance issueVery LowMediumNon-interactive fallback
Bundle size regressionLowMediumDynamic imports

PR Description Template

markdown
## Summary

Add foundation for Ink-based interactive CLI UI with a working `promptfoo list` command.

### What's Included

**Infrastructure:**

- Environment detection (TTY check)
- Opt-in via `PROMPTFOO_ENABLE_INTERACTIVE_UI=true`
- Lazy Ink/React loading (no bundle impact for library users)
- Non-interactive fallback (table output)
- Signal handling with proper exit codes

**User-Facing Feature:**

- Interactive `promptfoo list` command with keyboard navigation
- Search functionality (press `/`)
- Pagination with auto-loading
- Support for evals, prompts, and datasets views

### Usage

```bash
# Enable interactive mode (opt-in)
PROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list evals
PROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list prompts
PROMPTFOO_ENABLE_INTERACTIVE_UI=true promptfoo list datasets

# Default behavior (table output)
promptfoo list evals
```

Screenshots

[Add screenshot of interactive list UI here]

Test Plan

  • Unit tests for all utilities
  • Integration tests for list runner
  • Manual test with env var in iTerm2
  • Manual test with env var in VS Code terminal
  • Manual test with piped output (should fall back to table)
  • Manual test default behavior (table output)

Environment Variable

VariableDefaultPurpose
PROMPTFOO_ENABLE_INTERACTIVE_UIfalseEnable Ink-based interactive UI

---

## What's Left for Future PRs

After PR 1 merges, remaining PRs can build on this foundation:

| PR | What It Adds | Dependencies |
|----|-------------|--------------|
| PR 2 | Hooks, Utils, Shared Components | PR 1 |
| PR 3 | Eval UI (core feature) | PR 1, PR 2 |
| PR 4 | Auth, Cache, Menu, Share UIs | PR 1, PR 2 |
| PR 5 | Init Wizard | PR 1, PR 2 |
| PR 6 | Redteam Generate UI | PR 1, PR 2 |

**Note:** TextInput.tsx is placed in `src/ui/init/components/shared/` in the current branch. After PR 1 merges, PR 5 (Init Wizard) will use this component as-is. No duplication needed.