O3 Web Crawler

A Python tool that uses OpenAI's o3 model and Firecrawl to intelligently crawl websites based on specific objectives.

Features

• Maps website URLs to identify the most relevant pages for your objective
• Uses OpenAI's o3 model to analyze and rank pages by relevance
• Extracts specific information from web pages based on your objective
• Provides detailed, color-coded terminal output to track progress

Prerequisites

• Python 3.6+
• Firecrawl API key
• OpenAI API key

Installation

1. Clone this repository
2. Install dependencies:
   Copy

   pip install -r requirements.txt
   

3. Create a .env file based on .env.example with your API keys

Usage

Run the script:

python o3-web-crawler.py

You will be prompted to:

1. Enter a website URL to crawl
2. Specify your objective (what information you want to extract)

The script will:

• Analyze your objective to determine optimal search parameters
• Map the website to find relevant pages
• Rank pages by relevance to your objective
• Scrape and analyze top pages to extract the requested information
• Display results in JSON format

Example

Enter the website to crawl: https://example.com
Enter your objective: Find the company's contact information and headquarters location

The script will intelligently crawl the website and extract the requested information.

License

MIT