ContextQMD
Back to Karakeep

Model Comparison Tool

tools/compare-models/README.md

0.31.07.1 KB
Original Source

Model Comparison Tool

A standalone CLI tool to compare the tagging performance of AI models using your existing Karakeep bookmarks.

Features

  • Two comparison modes:
    • Model vs Model: Compare two AI models against each other
    • Model vs Existing: Compare a new model against existing AI-generated tags on your bookmarks
  • Fetches existing bookmarks from your Karakeep instance
  • Runs tagging inference with AI models
  • Random shuffling: Models/tags are randomly assigned to "Model A" or "Model B" for each bookmark to eliminate bias
  • Blind comparison: Model names are hidden during voting (only shown as "Model A" and "Model B")
  • Interactive voting interface
  • Shows final results with winner

Setup

Environment Variables

Required environment variables:

bash
# Karakeep API configuration
KARAKEEP_API_KEY=your_api_key_here
KARAKEEP_SERVER_ADDR=https://your-karakeep-instance.com

# Comparison mode (default: model-vs-model)
# - "model-vs-model": Compare two models against each other
# - "model-vs-existing": Compare a model against existing AI tags
COMPARISON_MODE=model-vs-model

# Models to compare
# MODEL1_NAME: The new model to test (always required)
# MODEL2_NAME: The second model to compare against (required only for model-vs-model mode)
MODEL1_NAME=gpt-4o-mini
MODEL2_NAME=claude-3-5-sonnet

# OpenAI/OpenRouter API configuration (for running inference)
OPENAI_API_KEY=your_openai_or_openrouter_key
OPENAI_BASE_URL=https://openrouter.ai/api/v1  # Optional, defaults to OpenAI

# Optional: Number of bookmarks to test (default: 10)
COMPARE_LIMIT=10

Using OpenRouter

For OpenRouter, set:

bash
OPENAI_BASE_URL=https://openrouter.ai/api/v1
OPENAI_API_KEY=your_openrouter_key

Using OpenAI Directly

For OpenAI directly:

bash
OPENAI_API_KEY=your_openai_key
# OPENAI_BASE_URL can be omitted for direct OpenAI

Usage

Run with pnpm (Recommended)

bash
cd tools/compare-models
pnpm install
pnpm run

Run with environment file

Create a .env file:

env
KARAKEEP_API_KEY=your_api_key
KARAKEEP_SERVER_ADDR=https://your-karakeep-instance.com
MODEL1_NAME=gpt-4o-mini
MODEL2_NAME=claude-3-5-sonnet
OPENAI_API_KEY=your_openai_key
COMPARE_LIMIT=10

Then run:

bash
pnpm run

Using directly with node

If you prefer to run the compiled JavaScript directly:

bash
pnpm build
export KARAKEEP_API_KEY=your_api_key
export KARAKEEP_SERVER_ADDR=https://your-karakeep-instance.com
export MODEL1_NAME=gpt-4o-mini
export MODEL2_NAME=claude-3-5-sonnet
export OPENAI_API_KEY=your_openai_key
node dist/index.js

Comparison Modes

Model vs Model Mode

Compare two different AI models against each other:

bash
COMPARISON_MODE=model-vs-model
MODEL1_NAME=gpt-4o-mini
MODEL2_NAME=claude-3-5-sonnet

This mode runs inference with both models on each bookmark and lets you choose which tags are better.

Model vs Existing Mode

Compare a new model against existing AI-generated tags on your bookmarks:

bash
COMPARISON_MODE=model-vs-existing
MODEL1_NAME=gpt-4o-mini
# MODEL2_NAME is not required in this mode

This mode is useful for:

  • Testing if a new model produces better tags than your current model
  • Evaluating whether to switch from one model to another
  • Quality assurance on existing AI tags

Note: This mode only compares bookmarks that already have AI-generated tags (tags with attachedBy: "ai"). Bookmarks without AI tags are automatically filtered out.

Usage Flow

  1. The tool fetches your latest link bookmarks from Karakeep

    • In model-vs-existing mode, only bookmarks with existing AI tags are included

  2. For each bookmark, it randomly assigns the options to "Model A" or "Model B" and runs tagging

  3. You'll see a side-by-side comparison (randomly shuffled each time):

    === Bookmark 1/10 ===
How to Build Better AI Systems
https://example.com/article
This article explores modern approaches to...

─────────────────────────────────────

Model A (blind):
  • ai
  • machine-learning
  • engineering

Model B (blind):
  • artificial-intelligence
  • ML
  • software-development

─────────────────────────────────────

Which tags do you prefer? [1=Model A, 2=Model B, s=skip, q=quit] >

  4. Choose your preference:

    • 1 - Vote for Model A
    • 2 - Vote for Model B
    • s or skip - Skip this comparison
    • q or quit - Exit early and show current results

  5. After completing all comparisons (or quitting early), results are displayed:

    ───────────────────────────────────────
=== FINAL RESULTS ===
───────────────────────────────────────
gpt-4o-mini: 6 votes
claude-3-5-sonnet: 3 votes
Skipped: 1
Errors: 0
───────────────────────────────────────
Total bookmarks tested: 10

🏆 WINNER: gpt-4o-mini
───────────────────────────────────────

  6. The actual model names are only shown in the final results - during voting you see only "Model A" and "Model B"

Bookmark Filtering

The tool currently tests only:

  • Link-type bookmarks (not text notes or assets)
  • Non-archived bookmarks
  • Latest N bookmarks (where N is COMPARE_LIMIT)
  • In model-vs-existing mode: Only bookmarks with existing AI tags (tags with attachedBy: "ai")

Architecture

This tool leverages Karakeep's shared infrastructure:

  • API Client: Uses @karakeep/sdk for type-safe API interactions with proper authentication
  • Inference: Reuses @karakeep/shared/inference for OpenAI client with structured output support
  • Prompts: Uses @karakeep/shared/prompts for consistent tagging prompt generation with token management
  • No code duplication - all core functionality is shared with the main Karakeep application

Error Handling

  • If a model fails to generate tags for a bookmark, an error is shown and comparison continues
  • Errors are counted separately in final results
  • Missing required environment variables will cause the tool to exit with a clear error message

Build

To build a standalone binary:

bash
pnpm build

The built binary will be in dist/index.js.

Notes

  • The tool is designed for manual, human-in-the-loop evaluation
  • No results are persisted - they're only displayed in console
  • Content is fetched with includeContent=true from Karakeep API
  • Uses Karakeep SDK (@karakeep/sdk) for type-safe API interactions
  • Inference runs sequentially to keep state management simple
  • Recommended to use pnpm run for the best experience (uses tsx for development)
  • Random shuffling: For each bookmark, models are randomly assigned to "Model A" or "Model B" to eliminate position bias. The actual model names are only revealed in the final results.