Troubleshooting Guide

This guide lists common issues you might encounter while installing, upgrading, or operating SourceMonitor, along with concrete steps to resolve them.

1. Mount Path Returns 404

Ensure the install generator added mount SourceMonitor::Engine, at: "/source_monitor" (or your custom path) inside the host config/routes.rb.
Restart the Rails server after modifying routes.
Confirm your host application routes are reloaded (run rbenv exec bin/rails routes | grep source_monitor).

2. Migrations or Solid Queue Tables Are Missing

Run rbenv exec bin/rails railties:install:migrations FROM=source_monitor followed by rbenv exec bin/rails db:migrate.
If you see duplicate migration timestamps, remove the older copy before rerunning the installer.
For Solid Queue tables, verify the host database contains the solid_queue_* tables shipped with the engine migration or run rbenv exec bin/rails solid_queue:install for a dedicated queue database.

3. Dashboard Metrics Show "Unavailable"

Solid Queue metrics require the solid_queue tables—see issue 2 above.
Ensure at least one Solid Queue worker is running; the dashboard reads visibility data via SourceMonitor::Jobs::Visibility.
When using mission control integration, keep config.mission_control_dashboard_path pointing at a valid route helper; otherwise the dashboard hides the link.

4. Recurring Jobs Not Running

Symptoms: Fetch scheduling, scrape scheduling, and cleanup jobs never fire. Sources never auto-fetch on their configured intervals.
Primary fix: Re-run the install generator, which automatically patches the dispatcher config:
bash
```
bin/rails generate source_monitor:install
```
Diagnostics: Run bin/source_monitor verify to check recurring task registration. The RecurringScheduleVerifier will report whether SourceMonitor recurring tasks are loaded into Solid Queue.
Manual check: Verify config/queue.yml includes recurring_schedule: config/recurring.yml under the dispatchers: section. Without this key, Solid Queue's dispatcher will not load the recurring schedule even though config/recurring.yml exists.

Manual fix (if generator cannot patch):

yaml

dispatchers:
  - polling_interval: 1
    batch_size: 500
    recurring_schedule: config/recurring.yml

5. Jobs Not Processing with bin/dev

Symptoms: bin/dev starts the web server but jobs never run. Running bin/rails solid_queue:start manually works fine.
Primary fix: Re-run the install generator, which automatically patches Procfile.dev:
bash
```
bin/rails generate source_monitor:install
```
Diagnostics: Run bin/source_monitor verify to check Solid Queue worker status. The SolidQueueVerifier will suggest Procfile.dev if no workers are detected.
Manual check: Verify Procfile.dev includes a jobs: line:
```
jobs: bundle exec rake solid_queue:start
```
Most Rails 8 apps use foreman or overmind via bin/dev. Without a jobs: entry, the process manager only starts the web server and asset watchers -- Solid Queue workers are not launched.

6. Realtime Updates Do Not Stream

Confirm Action Cable is mounted and ApplicationCable classes exist (see installation guide).
In production, verify WebSocket proxy settings allow the /cable endpoint.
When switching to Redis, add config.realtime.adapter = :redis and config.realtime.redis_url in the initializer, then restart web and worker processes.
For Solid Cable, check that the solid_cable_messages table exists and that no other process clears it unexpectedly.

7. Sources Show "Overdue" on Dashboard

Symptoms: Many sources show as overdue on the dashboard, especially after deploys or on sites with hundreds of sources.
Thundering herd: If many sources became due simultaneously (e.g., after a queue stall), they overwhelm the scheduler's per-run batch size (default 25). Run the stagger task to spread them out:
bash
```
bin/rails source_monitor:maintenance:stagger_fetch_times WINDOW_MINUTES=10
```
Stuck "fetching" sources: The stalled fetch reconciler automatically resets sources stuck in "fetching" status after config.fetching.stale_timeout_minutes (default 5 minutes). For manual recovery:
bash
```
bin/rails source_monitor:maintenance:recover_stalled_fetches
```
Batch size too small: If you have hundreds of sources, the default batch size of 25 may cause a backlog. Increase via config.fetching.scheduler_batch_size = 50 in your initializer.
Queue separation: Ensure your solid_queue.yml includes all three SourceMonitor queues (source_monitor_fetch, source_monitor_scrape, source_monitor_maintenance). Non-fetch jobs on the wrong queue can starve fetch processing.

8. Fetch Jobs Keep Failing

Review the most recent fetch log entry for the source; it stores the HTTP status, error class, and error message.
Increase config.http.timeout or config.http.retry_max if the feed is slow or prone to transient errors.
Supply custom headers or basic auth credentials via the source form when feeds require authentication.
Check for TLS issues on self-signed feeds; you may need to configure Faraday with custom SSL options.

9. Scraping Returns "Failed"

Confirm the source has scraping enabled and the configured adapter exists.
Override selectors in the source's scrape settings if the default Readability extraction misses key elements.
Inspect the scrape log to see the adapter status and content length. Logs store the HTTP status and any exception raised by the adapter.
Retry manually from the item detail page after fixing selectors.

10. Cleanup Rake Tasks Fail

Pass numeric values for FETCH_LOG_DAYS or SCRAPE_LOG_DAYS environment variables (e.g., FETCH_LOG_DAYS=30).
Ensure workers or the console environment have permission to soft delete (SOFT_DELETE=true) if you expect tombstones.
If job classes cannot load, verify SourceMonitor.configure ran before calling rake source_monitor:cleanup:*.

11. Test Suite Cannot Launch a Browser

System tests rely on Selenium + Chrome. Install Chrome/Chromium and set SELENIUM_CHROME_BINARY if the binary lives in a non-standard path.
You can run rbenv exec bin/test-coverage --verbose to inspect failures with additional logging.

12. Mission Control Jobs Link Returns 404

Mount MissionControl::Jobs::Engine in your host routes (for example, mount MissionControl::Jobs::Engine, at: "/mission_control").
Keep config.mission_control_enabled = true and config.mission_control_dashboard_path pointing at that mounted route helper. Call SourceMonitor.mission_control_dashboard_path in the Rails console to confirm it resolves.
When hosting Mission Control in a separate app, provide a full URL instead of a route helper and ensure CORS/WebSocket settings allow the dashboard iframe.

13. Tailwind Build Fails or Admin UI Loads Without Styles

Running test/dummy/bin/dev before configuring the bundling pipeline will serve the admin UI without Tailwind styles or Stimulus behaviours. This happens because the engine no longer ships precompiled assets; see .ai/engine-asset-configuration.md:11-44 for the required npm setup.
Fix by running npm install followed by npm run build inside the engine root so that app/assets/builds/source_monitor/application.css and application.js exist. The Rake task app:source_monitor:assets:build wraps the same scripts for CI usage.
When the UI is still unstyled, confirm the dummy app can read the namespaced asset directories noted in .ai/engine-asset-configuration.md:32-44 and restart bin/dev so the CSS/JS watchers reconnect.

14. Feed Returns Cloudflare Challenge Page

Symptoms: Fetch logs show HTML containing "Checking your browser" or "Just a moment..." instead of feed XML. The source may display a "Blocked" badge.
How it works: SourceMonitor automatically detects Cloudflare challenge responses and attempts cookie replay with UA rotation on subsequent fetches. No configuration is needed -- this is enabled by default.
If fetches remain blocked: Some sites employ aggressive bot mitigation that cannot be bypassed with cookie replay alone. In these cases:
- Try setting a custom user_agent on the source's custom headers to match a specific browser.
- Consider whether the site offers an alternative feed URL that is not behind Cloudflare.
- Check the fetch log's error_category field -- a value of blocked confirms Cloudflare detection triggered.

Still Stuck?

Collect the following and open an issue or start a discussion:

Engine version (SourceMonitor::VERSION)
Host Rails version and Ruby version
Relevant configuration snippet from config/initializers/source_monitor.rb
Recent fetch/scrape log entries or stack traces

We track known issues and roadmap items in .ai/tasks.md; check the next slice to see if your problem is already scheduled.