AgentPress Prompt Caching System

Overview

AgentPress implements mathematically optimized prompt caching for Anthropic Claude models to achieve 70-90% cost and latency savings in long conversations. The system uses dynamic token-based thresholds that adapt to conversation length, context window size, and message density.

How It Works

1. Dynamic Context Detection

• Auto-detects context window from model registry (200k-2M+ tokens)
• Supports all models: Claude 3.7 (200k), Claude Sonnet 4 (1M), Gemini 2.5 Pro (2M)
• Falls back to 200k default if model not found

2. Mathematical Threshold Calculation

Optimal Threshold = Base × Stage × Context × Density

Where:
• Base = 2.5% of context window
• Stage = Conversation length multiplier
• Context = Context window multiplier  
• Density = Token density multiplier

3. Conversation Stage Scaling

4. Context Window Scaling

Cache Threshold Examples

Real-World Thresholds by Model & Conversation Length

Cache Block Strategy

4-Block Distribution

1. Block 1: System prompt (cached if ≥1024 tokens)
2. Blocks 2-4: Conversation chunks (automatic management)

Cache Management

• Early blocks: Stable, reused longest
• Recent blocks: Dynamic, optimized for conversation flow

Token Counting

Uses LiteLLM's accurate tokenizers:

from litellm import token_counter
tokens = token_counter(model=model_name, text=content)

• Anthropic models: Uses Anthropic's actual tokenizer
• OpenAI models: Uses tiktoken
• Other models: Model-specific tokenizers
• Fallback: Word-based estimation (1.3x words)

Cost Benefits

Pricing Structure

• Cache Writes: 1.25x base cost for write operations
• Cache Hits: 0.1x base cost (90% savings)
• Break-even: 2-3 reuses for most chunks

Example Savings

• 200k context conversation: 70-85% cost reduction
• 1M context conversation: 80-90% cost reduction
• 500+ message threads: Up to 95% latency reduction

Implementation Flow

graph TD
    A[New Message] --> B{Anthropic Model?}
    B -->|No| C[Standard Processing]
    B -->|Yes| D[Get Context Window from Registry]
    D --> E[Calculate Optimal Threshold]
    E --> F[Count Existing Tokens]
    F --> G{Threshold Reached?}
    G -->|No| H[Add to Current Chunk]
    G -->|Yes| I[Create Cache Block]
    I --> J{Max Blocks Reached?}
    J -->|No| K[Continue Chunking]
    J -->|Yes| L[Add Remaining Uncached]
    H --> M[Send to LLM]
    K --> M
    L --> M
    C --> M

Key Features

✅ Prevents Cache Invalidation

• Fixed-size chunks never change once created
• New messages go into new chunks or remain uncached
• No more cache invalidation on every new message

✅ Scales Efficiently

• Handles 20-message conversations to 1000+ message threads
• Adapts chunk sizes to context window (200k-2M tokens)
• Preserves cache blocks for maximum reuse

✅ Cost Optimized

• Mathematical break-even analysis
• Early aggressive caching for quick wins
• Late conservative caching to preserve blocks

✅ Context Window Aware

• Prevents cache block preoccupation in large contexts
• Reserves 20% of context for new messages/outputs
• Handles oversized conversations gracefully

Usage

The caching system is automatically applied in ThreadManager.run_thread():

# Auto-detects context window and calculates optimal thresholds
prepared_messages = apply_anthropic_caching_strategy(
    system_prompt, 
    conversation_messages, 
    model_name  # e.g., "claude-sonnet-4"
)

Monitoring

Track cache performance via logs:

• 🔥 Block X: Cached chunk (Y tokens, Z messages)
• 🎯 Total cache blocks used: X/4
• 📊 Processing N messages (X tokens)
• 🧮 Calculated optimal cache threshold: X tokens

Result

70-90% cost and latency savings in long conversations while scaling efficiently across all context window sizes and conversation lengths.