.ai/project_overview.md
SourceMonitor Engine is a production-ready Rails 8 mountable engine for aggregating, monitoring, and managing RSS/Atom/JSON feeds at scale. Built with Rails 8 defaults and minimal dependencies, it provides a complete solution for feed ingestion, content scraping, and real-time monitoring.
Add and Configure Multiple Sources
Source Health Monitoring
Source Status Control
Intelligent HTTP Fetching
Scheduled Fetching
Item Creation & Deduplication
Standard Feed Fields
Extended Metadata
Feed-Level Metadata
Full Article Extraction
Scraper Adapters
Scraping Control
Dashboard
Source Management Views
Item Browser
Log Viewers
Solid Queue Integration
Job Types
Job Features
Flexible Scheduler Architecture
Scheduler.runScheduling Strategies
Source Scheduling
Turbo Streams
Progressive Enhancement
Interactive UI Components
Retention Policies
Cleanup Automation
Data Integrity
ActiveSupport Notifications
source_monitor.fetch.start - Before fetch beginssource_monitor.fetch.finish - After fetch completessource_monitor.scrape.start - Before scrapingsource_monitor.scrape.finish - After scrapingHealth Monitoring
/health endpoint with system statusLogging
Metrics & Analytics
Smart Retry Logic
Self-Healing Features
Error Tracking
Alerting System
Configuration DSL
SourceMonitor.configure do |config|
config.fetch_timeout = 30
config.scrape_timeout = 60
config.user_agent = "MyApp Bot"
config.default_fetch_interval = 6
config.retention_days = 30
end
Event Callbacks
after_item_created - Process new itemsafter_item_scraped - Handle scraped contentafter_fetch_completed - React to fetch eventsModel Extensions
Custom Fields
Database Optimization
Caching Strategy
Concurrent Processing
Scalability Features
Input Validation
SSRF Protection
Authentication & Authorization
Security Headers
Generator Tasks
Configuration Options
Example Applications
Complete feed source configuration and state tracking.
Key Fields:
Aggregated content with multiple storage layers.
Key Fields:
Complete audit trail of fetch operations.
Key Fields:
Audit trail of scraping operations.
Key Fields:
Implement the Scrapers::Base interface to add new extraction methods:
Hook into the feed lifecycle:
Extend item processing:
Modify the admin interface:
source_monitor/application); host apps can override controller registrations with importmap.rbstimulus-use transitions when available and automatically fall back to class toggling when the module is not pinnedbin/rails app:source_monitor:assets:build and app:source_monitor:assets:verify keep Tailwind builds current; verification runs before rails testrbenv exec bundle exec rails testrbenv exec bundle exec rubocoprbenv exec bundle exec rake app:source_monitor:assets:verifyrbenv exec bundle exec gem build source_monitor.gemspecCHANGELOG.md with release notes and tag commit (git tag vX.Y.Z)rbenv exec gem push pkg/source_monitor-X.Y.Z.gem)Content Aggregation
Monitoring & Tracking
Content Processing
Internal Tools
gem 'source_monitor'rails g source_monitor:installrails db:migratemount SourceMonitor::Engine => "/source_monitor"http://localhost:3000/source_monitorThe engine is production-ready from day one with sensible defaults and can be customized extensively for specific needs.