posthog-self-driving-report.md
Generated 2026-07-14 for project CMEM (ID 463218)
PostHog Self-driving was configured for the claude-mem project. Error tracking (three trigger types), session replay, GitHub Issues, and support ticket signal sources were enabled; the scout troop was tuned to three scouts matching this project's primary surfaces (AI/LLM usage and MCP tool calls). Findings will start appearing in the Self-driving inbox within ~30 minutes.
Approved. Organization-level AI data processing consent was granted before this run.
Already connected — GitHub App integration (thedotmack, integration ID 175967) was present at setup time, connected on 2026-06-10. One repository is synced: thedotmack/claude-mem.
| Product | Status | Notes |
|---|---|---|
| Session Replay | Already active | Recordings flowing from cmem.ai; products-enable API not available in this deployment — server state confirmed by live recordings |
| Error Tracking | Already active | 1.1M+ events via posthog-node with enableExceptionAutocapture; products-enable API not available |
| Support (Conversations) | Follow-up required | Product could not be enabled via API (tool unavailable); see follow-ups for manual step |
This project uses posthog-node (backend), not posthog-js. No posthog.init override check was needed. Session replay for the web front-end (cmem.ai) appears to be instrumented from a separate repo or snippet.
| source_product | source_type | Action |
|---|---|---|
error_tracking | issue_created | Enabled (new, ID 019f6318-32fa-73a9-914f-55617f6800de) |
error_tracking | issue_reopened | Enabled (new, ID 019f6318-3775-7090-a2e5-d98f893734d0) |
error_tracking | issue_spiking | Enabled (new, ID 019f6318-39b6-7161-8bce-8e03a206d92b) |
session_replay | session_analysis_cluster | Enabled (new, ID 019f6318-3dd1-7dae-b4a0-36eba57cb746; default 10% sample rate) |
conversations | ticket | Enabled (new, ID 019f6318-4047-764c-94fc-ee62a511a9b2; dormant until a channel is connected) |
signals_scout | cross_source_issue | On by default — no config row needed; scout findings reach the inbox automatically |
llm_analytics | evaluation_report | Skipped — internal-only responder, not user-facing |
logs | — | Skipped — not a v1 responder |
| Tool | Status |
|---|---|
| GitHub Issues | Already connected — Github warehouse source (ID 019eb017-8f11-0000-8ce2-d7bed20cb960) was present; 1,766 issues synced as of 2026-07-14. Responder (github / issue, ID 019f6324-8a46-71ab-ad0b-8cea5b4fb795) enabled. |
| Linear | Not used — not selected |
| Zendesk | Not used — not selected |
| pganalyze | Not used — not selected |
Note: the GitHub warehouse source has a token error on the stargazers table (Access forbidden). The issues table (the one the responder reads) is syncing cleanly. See follow-ups.
3 enabled, 23 disabled.
| Scout | Reason |
|---|---|
signals-scout-general | Always on — sweeps cross-product correlations and surfaces no specialist covers |
signals-scout-ai-observability | This product wraps Claude/Anthropic APIs — LLM cost, latency, and error regressions are directly on-product |
signals-scout-mcp-tool-calls | This product IS an MCP server — $mcp_tool_call telemetry covers tool failure rates and confusing schemas |
| Scout | Reason |
|---|---|
signals-scout-error-tracking | Covered by the native error_tracking source (all three trigger types enabled above) |
signals-scout-session-replay | Covered by the native session_replay source enabled above |
signals-scout-product-analytics | No confirmed active funnels/retention insights — enable if you build out product analytics dashboards |
signals-scout-web-analytics | No confirmed UTM/referrer tracking in this repo — enable if you add web analytics to cmem.ai |
signals-scout-feature-flags | Feature flag usage not confirmed — enable if you adopt flags |
signals-scout-surveys | 0 surveys in use |
signals-scout-revenue-analytics | No payment SDK found |
signals-scout-logs | PostHog logs product not confirmed in use |
signals-scout-csp-violations | No CSP reporting configured |
signals-scout-experiments | No active A/B experiments |
signals-scout-customer-analytics | No group/accounts analytics confirmed |
signals-scout-data-pipelines | No CDP destinations or hog flows found |
signals-scout-replay-vision | Replay Vision scanners not configured |
signals-scout-apm | No OpenTelemetry/APM tracing found |
signals-scout-anomaly-detection | Covered adequately by general + specialists for this project size |
signals-scout-observability-gaps | Can enable later if event coverage gaps become a concern |
signals-scout-health-checks | Can enable if setup health becomes a concern |
signals-scout-inbox-validation | Not useful on a fresh setup — no shipped fixes to validate yet |
signals-scout-ingestion-warnings | Can enable if ingestion errors appear |
signals-scout-insight-alerts | No configured insight alerts found |
signals-scout-skills-store | Internal PostHog skill hygiene scout |
signals-scout-data-warehouse | GitHub source token issue on stargazers — see follow-ups |
signals-scout-web-vitals | No $web_vitals events confirmed from this repo |
Two surfaces were proposed and declined:
| Proposed scout | Surface | Why declined |
|---|---|---|
| Memory pipeline health | observer_turn_rollup → context_injected_rollup volume/ratio degradation | User dismissed the proposal |
| Install-to-first-memory funnel | install_completed → worker_started → first observation | User dismissed the proposal |
Gap analysis ruled out:
error_tracking native source (ChromaUnavailableError is the dominant issue)Noise escape hatch: to put any scout into dry-run (it runs and logs but emits nothing to the inbox), set emit: false on its config in PostHog > Self-driving settings.
products-enable API was not available in this PostHog deployment. Enable Session Replay, Error Tracking, and Conversations manually in Project settings > Products if not already on.conversations/ticket signal source is enabled and waiting. Connect an email, inbox, or Slack channel in PostHog so support tickets start reaching the inbox. (Integrations settings)stargazers table is failing with "Access forbidden / rate limits". Check the GitHub App token scope at Integrations settings — the issues table syncs cleanly so this doesn't block the Self-driving responder, but it may affect other warehouse queries.signals-scout-memory-pipeline — A custom scout watching observer_turn_rollup / context_injected_rollup volume and ratio degradation would catch silent pipeline failures before users notice worse memory quality. Create it via Self-driving scouts if you want this coverage.signals-scout-install-funnel — A custom scout watching install_completed → worker_started → first observation would catch broken first-run flows and platform-specific regressions. Create it if install conversion becomes a priority metric.The scout coordinator picks up fresh configs within ~30 minutes. Scouts run daily (1,440-minute interval) and emit findings as reports into the Self-driving inbox. Error tracking and session replay signals arrive sooner — they trigger on individual events as they land. Immediately-actionable reports can be turned into coding tasks directly from the inbox.