websurfer/README.md

# WebSurfer: Parallel Playwright Crawler

A highly-concurrent, fully automated website crawler built with Python and [Playwright](https://playwright.dev/python/). 

This crawler rapidly crawls an entire domain using multiple visible or headless browser tabs. It uses a Breadth-First Search (BFS) approach powered by an `asyncio.Queue` to spread the crawling workload across a customizable pool of worker tabs.

## Key Features

- **Multi-Tab Concurrency**: Spawns multiple asynchronous browser tabs (default: 5) to dramatically speed up crawl times.
- **Automatic Cookie Consent Acceptance**: Automatically detects and accepts cookie consent banners / CMPs (Complianz, Borlabs Cookie, Cookiebot, OneTrust, Usercentrics, Klaro, CookieYes, Real Cookie Banner, CCM19, and generic accept patterns) across pages and iframes so analytics, marketing tags, and tracking scripts fire properly.
- **Intelligent Scope**: 
  - Recursively crawls all internal links belonging to the starting domain.
  - Visits external links (depth 1) to verify they are alive, but intentionally avoids following their sub-links to prevent crawling the entire internet.
- **Crawler Trap Protection**: Automatically strips query parameters (e.g. `?page=2`) and URL fragments (e.g. `#section`) from internal links to prevent infinite loops from endless pagination or dynamically generated URLs. External links retain their original query parameters to prevent breaking things like YouTube videos.
- **Configurable Iteration Scope & Visit Limits**:
  - **All Websites (Default)**: Crawls all reachable internal and external level-1 links until the queue is fully exhausted.
  - **Limit Visits Mode**: Caps each iteration to a configurable number of website visits before starting a new iteration. Ideal for multi-iteration testing across fresh Tor circuits and rotating User-Agents on large websites without having to crawl every single page.
- **Randomized Crawl Queue**: Uniformly picks pages at random from the queue of discovered URLs instead of strict FIFO breadth-first order. Ensures broad, diverse coverage across various sections and depths of the website on every iteration.
- **Error & Broken Link Tracking**: Tracks any pages that result in HTTP errors (404, 500, etc.) or connection timeouts. At the end of the crawl, it outputs a clean list of broken URLs alongside the exact source page where the broken link was discovered.
- **Tor Proxy, Flexible Circuit Rotation & European Exit Nodes**: Optionally routes all crawler traffic securely through a headless Tor proxy (SOCKS5 on port `9050`).
  - **Circuit Rotation Modes**: Rotate per crawl iteration or dynamically after every *N* crawled pages.
  - **European Exit Node Selector**: Select 0 to *N* European countries (Germany, Netherlands, France, Switzerland, Sweden, UK, etc.) to strictly/preferentially route traffic through specific European exit relays. If no country is selected, global Tor exit relays are used worldwide.
- **Live Tor Network & IP Geolocation Inspector**: Shows active Tor exit IP, country, city, ISP/organization, exit node filter, and circuit rotation count in real-time.

## Web Dashboard & Frontend

WebSurfer includes a modern, real-time Web Dashboard built with FastAPI and WebSockets.

### Features in the Web Dashboard:
- **Interactive Configuration**: Configure start URL, concurrency, iterations, Tor routing, headless mode, cookie handling, and timeouts directly in the browser.
- **`.env` Synchronization**: One-click saving to `.env` or loading stored parameters.
- **Live Metrics & All-Iterations Cumulative Statistics**: Real-time KPI tiles for Visited Pages, Internal vs. External links, Cookies Accepted, Queue Backlog, and Timer, paired with a dedicated all-iterations summary bar and subtitle badges tracking total websites visited and total errors encountered across all iterations.
- **Dedicated Error & Broken Link Section**: Structured, searchable, and exportable (CSV / JSON) error registry displaying the failed URL, error reason / HTTP status, and referring source page.
- **Live Streaming Terminal**: Real-time color-coded crawl logs with auto-scrolling, search query filtering, and level filtering.

### Running the Web Dashboard:
```bash
python web.py
```
*(or `python main.py --web`)*

Open your browser at [http://127.0.0.1:8000](http://127.0.0.1:8000).

## Project Structure

The project is organized into clean, single-responsibility modules:

```
websurfer/
├── crawler/
│   ├── __init__.py      # Public exports (Crawler, CrawlerConfig, TorManager, etc.)
│   ├── config.py        # Configuration dataclass & interactive/env/dict loaders
│   ├── tor.py           # Tor daemon lifecycle & circuit rotation (Stem/SOCKS5)
│   ├── cookies.py       # CMP selectors, text regexes & auto-consent execution
│   ├── url_utils.py     # URL normalization, domain matching & static file filters
│   └── engine.py        # Playwright browser lifecycle, worker queue, events & crawl loop
├── web/
│   ├── __init__.py      # Web package entry point & server runner
│   ├── app.py           # FastAPI REST API & WebSocket endpoints
│   ├── manager.py       # Async crawl session manager & event broadcaster
│   └── static/
│       ├── index.html   # Modern responsive dashboard UI
│       ├── style.css    # Dark glassmorphism styling
│       └── app.js       # Real-time WebSocket controller & data renderer
├── main.py              # Primary entry point (supports CLI and --web)
├── web.py               # Standalone Web Dashboard launcher
├── browser_crawler.py   # Backwards-compatible launcher
├── .env.example         # Example configuration settings
└── requirements.txt     # Project dependencies
```

## Prerequisites

You need Python 3.8+ installed. If you intend to use the Tor proxy feature, ensure the `tor` command-line package is installed on your system:
- **macOS**: `brew install tor`
- **Linux (Ubuntu/Debian)**: `sudo apt install tor`
- **Windows**: Install Tor Expert Bundle from [torproject.org](https://www.torproject.org/) and add it to your PATH.
*(No Tor Browser GUI needed!)*

## Installation

1. Navigate to the project directory:
   ```bash
   cd websurfer
   ```
2. Create and activate an OS-specific virtual environment:
   - **macOS / Linux**:
     ```bash
     python3 -m venv venv
     source venv/bin/activate
     ```
   - **Windows (PowerShell)**:
     ```powershell
     python -m venv .venv
     .\.venv\Scripts\Activate.ps1
     ```
   - **Windows (Command Prompt)**:
     ```cmd
     python -m venv .venv
     .\.venv\Scripts\activate.bat
     ```
3. Install the dependencies:
   ```bash
   pip install -r requirements.txt
   ```
4. Install the required Playwright Chromium binaries:
   ```bash
   playwright install chromium
   ```

## Usage

### 1. Web Dashboard (Recommended)
```bash
python web.py
```
Or:
```bash
python main.py --web
```

### 2. Running with Docker & Tor (Containerized)
WebSurfer can be run in a Docker container with Playwright headless Chromium and an integrated background Tor proxy:

```bash
# Build and start container (Web Dashboard on port 8080)
docker compose up --build

# Run in background
docker compose up -d

# View logs
docker compose logs -f
```

Open your browser at [http://127.0.0.1:8080](http://127.0.0.1:8080) (or customize the port with `WEB_PORT=8000` in `.env`).

To run a headless crawl via CLI inside the container:
```bash
docker compose run --rm websurfer python main.py
```

### 3. Running via CLI
Run the main entry point in interactive/CLI mode:
```bash
python main.py
```
*(or `python browser_crawler.py`)*

The script will interactively prompt you (or read from `.env`):
1. **Starting URL**: The base website you want to crawl (e.g., `https://example.com`).
2. **Tor Proxy**: Whether you want to route traffic through Tor (y/n).
3. **Concurrent Tabs**: The number of browser tabs you want running simultaneously (defaults to 5).
4. **Iterations**: Number of times to run the crawl.
5. **Headless Mode**: Run browser with or without UI.
6. **Accept Cookies**: Automatically accept cookies on pages (defaults to yes).

### 4. Programmatic Usage

You can also use WebSurfer in your own Python scripts:

```python
import asyncio
from crawler import Crawler, CrawlerConfig

async def run_my_crawl():
    config = CrawlerConfig(
        start_url="https://example.com",
        num_tabs=4,
        headless=True,
        accept_cookies=True,
    )
    crawler = Crawler(config)
    await crawler.run()

if __name__ == "__main__":
    asyncio.run(run_my_crawl())
```