WebSurfer: Parallel Playwright Crawler

A highly-concurrent, fully automated website crawler built with Python and Playwright.

This crawler rapidly crawls an entire domain using multiple visible or headless browser tabs. It uses a Breadth-First Search (BFS) approach powered by an asyncio.Queue to spread the crawling workload across a customizable pool of worker tabs.

Key Features

• Multi-Tab Concurrency: Spawns multiple asynchronous browser tabs (default: 5) to dramatically speed up crawl times.
• Automatic Cookie Consent Acceptance: Automatically detects and accepts cookie consent banners / CMPs (Complianz, Borlabs Cookie, Cookiebot, OneTrust, Usercentrics, Klaro, CookieYes, Real Cookie Banner, CCM19, and generic accept patterns) across pages and iframes so analytics, marketing tags, and tracking scripts fire properly.
• Intelligent Scope:
  • Recursively crawls all internal links belonging to the starting domain.
  • Visits external links (depth 1) to verify they are alive, but intentionally avoids following their sub-links to prevent crawling the entire internet.
• Crawler Trap Protection: Automatically strips query parameters (e.g. ?page=2) and URL fragments (e.g. #section) from internal links to prevent infinite loops from endless pagination or dynamically generated URLs. External links retain their original query parameters to prevent breaking things like YouTube videos.
• Configurable Iteration Scope & Visit Limits:
  • All Websites (Default): Crawls all reachable internal and external level-1 links until the queue is fully exhausted.
  • Limit Visits Mode: Caps each iteration to a configurable number of website visits before starting a new iteration. Ideal for multi-iteration testing across fresh Tor circuits and rotating User-Agents on large websites without having to crawl every single page.
• Randomized Crawl Queue: Uniformly picks pages at random from the queue of discovered URLs instead of strict FIFO breadth-first order. Ensures broad, diverse coverage across various sections and depths of the website on every iteration.
• Error & Broken Link Tracking: Tracks any pages that result in HTTP errors (404, 500, etc.) or connection timeouts. At the end of the crawl, it outputs a clean list of broken URLs alongside the exact source page where the broken link was discovered.
• Tor Proxy, Flexible Circuit Rotation & European Exit Nodes: Optionally routes all crawler traffic securely through a headless Tor proxy (SOCKS5 on port 9050).
  • Circuit Rotation Modes: Rotate per crawl iteration or dynamically after every N crawled pages.
  • European Exit Node Selector: Select 0 to N European countries (Germany, Netherlands, France, Switzerland, Sweden, UK, etc.) to strictly/preferentially route traffic through specific European exit relays. If no country is selected, global Tor exit relays are used worldwide.
• Live Tor Network & IP Geolocation Inspector: Shows active Tor exit IP, country, city, ISP/organization, exit node filter, and circuit rotation count in real-time.

Web Dashboard & Frontend

WebSurfer includes a modern, real-time Web Dashboard built with FastAPI and WebSockets.

Features in the Web Dashboard:

• Interactive Configuration: Configure start URL, concurrency, iterations, Tor routing, headless mode, cookie handling, and timeouts directly in the browser.
• .env Synchronization: One-click saving to .env or loading stored parameters.
• Live Metrics & All-Iterations Cumulative Statistics: Real-time KPI tiles for Visited Pages, Internal vs. External links, Cookies Accepted, Queue Backlog, and Timer, paired with a dedicated all-iterations summary bar and subtitle badges tracking total websites visited and total errors encountered across all iterations.
• Dedicated Error & Broken Link Section: Structured, searchable, and exportable (CSV / JSON) error registry displaying the failed URL, error reason / HTTP status, and referring source page.
• Live Streaming Terminal: Real-time color-coded crawl logs with auto-scrolling, search query filtering, and level filtering.

Running the Web Dashboard:

python web.py

(or python main.py --web)

Open your browser at http://127.0.0.1:8000.

Project Structure

The project is organized into clean, single-responsibility modules:

websurfer/
├── crawler/
│   ├── __init__.py      # Public exports (Crawler, CrawlerConfig, TorManager, etc.)
│   ├── config.py        # Configuration dataclass & interactive/env/dict loaders
│   ├── tor.py           # Tor daemon lifecycle & circuit rotation (Stem/SOCKS5)
│   ├── cookies.py       # CMP selectors, text regexes & auto-consent execution
│   ├── url_utils.py     # URL normalization, domain matching & static file filters
│   └── engine.py        # Playwright browser lifecycle, worker queue, events & crawl loop
├── web/
│   ├── __init__.py      # Web package entry point & server runner
│   ├── app.py           # FastAPI REST API & WebSocket endpoints
│   ├── manager.py       # Async crawl session manager & event broadcaster
│   └── static/
│       ├── index.html   # Modern responsive dashboard UI
│       ├── style.css    # Dark glassmorphism styling
│       └── app.js       # Real-time WebSocket controller & data renderer
├── main.py              # Primary entry point (supports CLI and --web)
├── web.py               # Standalone Web Dashboard launcher
├── browser_crawler.py   # Backwards-compatible launcher
├── .env.example         # Example configuration settings
└── requirements.txt     # Project dependencies

Prerequisites

You need Python 3.8+ installed. If you intend to use the Tor proxy feature, ensure the tor command-line package is installed on your system:

• macOS: brew install tor
• Linux (Ubuntu/Debian): sudo apt install tor
• Windows: Install Tor Expert Bundle from torproject.org and add it to your PATH. (No Tor Browser GUI needed!)

Installation

1. Navigate to the project directory:

   cd websurfer
   

2. Create and activate an OS-specific virtual environment:
   • macOS / Linux:

     python3 -m venv venv
     source venv/bin/activate
     

   • Windows (PowerShell):

     python -m venv .venv
     .\.venv\Scripts\Activate.ps1
     

   • Windows (Command Prompt):

     python -m venv .venv
     .\.venv\Scripts\activate.bat
     

3. Install the dependencies:

   pip install -r requirements.txt
   

4. Install the required Playwright Chromium binaries:

   playwright install chromium
   

Usage

1. Web Dashboard (Recommended)

python web.py

Or:

python main.py --web

2. Running with Docker & Tor (Containerized)

WebSurfer can be run in a Docker container with Playwright headless Chromium and an integrated background Tor proxy:

# Build and start container (Web Dashboard on port 8080)
docker compose up --build

# Run in background
docker compose up -d

# View logs
docker compose logs -f

Open your browser at http://127.0.0.1:8080 (or customize the port with WEB_PORT=8000 in .env).

To run a headless crawl via CLI inside the container:

docker compose run --rm websurfer python main.py

3. Running via CLI

Run the main entry point in interactive/CLI mode:

python main.py

(or python browser_crawler.py)

The script will interactively prompt you (or read from .env):

1. Starting URL: The base website you want to crawl (e.g., https://example.com).
2. Tor Proxy: Whether you want to route traffic through Tor (y/n).
3. Concurrent Tabs: The number of browser tabs you want running simultaneously (defaults to 5).
4. Iterations: Number of times to run the crawl.
5. Headless Mode: Run browser with or without UI.
6. Accept Cookies: Automatically accept cookies on pages (defaults to yes).

4. Programmatic Usage

You can also use WebSurfer in your own Python scripts:

import asyncio
from crawler import Crawler, CrawlerConfig

async def run_my_crawl():
    config = CrawlerConfig(
        start_url="https://example.com",
        num_tabs=4,
        headless=True,
        accept_cookies=True,
    )
    crawler = Crawler(config)
    await crawler.run()

if __name__ == "__main__":
    asyncio.run(run_my_crawl())