OEM Vehicle Stock Crawler

A production-ready, modular platform for crawling car manufacturer (OEM) inventory, tracking vehicle stock, specifications, optional equipment, and price changes over time.

📁 Project Architecture

oem_crawler/
├── Dockerfile                   # Lean production container using uv
├── docker-compose.yml           # PostgreSQL 16 + crawler container orchestrator
├── .env.example                 # Environment configuration template
├── pyproject.toml               # Dependencies and CLI entrypoints (`oem-crawler`, `oem-inspect`)
├── schema.sql                   # PostgreSQL database DDL
│
├── data/                        # Development SQLite databases (gitignored)
│   └── crawler_stock.db         # Default local SQLite database file
│
├── src/oem_crawler/
│   ├── config.py                # Environment configuration (pydantic-settings)
│   ├── cli.py                   # Main CLI application (`oem-crawler`)
│   │
│   ├── tools/                   # Developer & stock inspection tools
│   │   ├── README.md            # Tools documentation and sqlite3 recipes
│   │   └── view_stock.py        # Terminal inventory inspector (`oem-inspect`)
│   │
│   ├── db/                      # Database Layer
│   │   ├── models.py            # SQLAlchemy 2.0 ORM mappings
│   │   ├── schemas.py           # Pydantic v2 validation models & SHA-256 hashing
│   │   ├── session.py           # Connection and session factory
│   │   └── pipeline.py          # Ingestion pipeline, change-detection & delisting
│   │
│   └── crawlers/                # Modular OEM Crawlers
│       ├── base.py              # BaseCrawler interface & CrawlerRegistry
│       ├── audi/                # Audi Implementation
│       │   ├── crawler.py       # Stock Car Service (SCS) REST client
│       │   └── parser.py        # PR option codes, EV specs & GPS coordinates
│       ├── bmw/                 # BMW Implementation
│       │   ├── crawler.py       # REST API client & pagination
│       │   └── parser.py        # OEM response normalization
│       ├── mercedes/            # Mercedes-Benz Implementation
│       │   ├── crawler.py       # OneSearch GraphQL + PLS + VPC client
│       │   └── parser.py        # OneWeb normalization logic
│       │   └── query.py         # Tested GraphQL query definitions
│       └── porsche/             # Porsche Implementation
│           ├── crawler.py       # Finder Next.js RSC client & pagination
│           └── parser.py        # Option C, equipment & metadata normalization
│
└── tests/                       # Automated Test Suite
    ├── test_schema.py           # Schema & hashing unit tests
    ├── test_audi_parser.py      # Audi parser & registry unit tests
    ├── test_bmw_parser.py       # BMW parser & registry unit tests
    ├── test_mercedes_parser.py  # Mercedes parser unit tests
    ├── test_porsche_parser.py   # Porsche parser unit tests
    └── test_pipeline.py         # Ingestion pipeline unit tests

🔌 Adding a New Car Manufacturer Crawler

Adding a new OEM crawler (e.g. Mercedes, Audi, Tesla, Porsche) takes 2 simple steps:

Create a new folder under src/oem_crawler/crawlers/<brand>/ (e.g. mercedes/):
Implement your crawler inheriting from BaseCrawler and register it:

from oem_crawler.crawlers.base import BaseCrawler, CrawlerRegistry
from oem_crawler.db.schemas import VehicleIngestSchema

@CrawlerRegistry.register("mercedes")
class MercedesStockCrawler(BaseCrawler):
    @property
    def brand_name(self) -> str:
        return "Mercedes-Benz"

    def crawl(self, max_vehicles=None, page_size=20, search_filter=None):
        # 1. Fetch from OEM API / site
        # 2. Parse into VehicleIngestSchema
        # 3. Yield parsed vehicle
        ...

The CLI (oem-crawler --brand mercedes) and pipeline will automatically discover and support it.

🐳 Running with Docker

1. Start PostgreSQL Database & Run a Crawl

docker compose up --build

2. Run Ad-hoc Crawl inside Docker

# Fetch 100 vehicles for BMW
docker compose run --rm crawler --brand bmw --limit 100

3. Query PostgreSQL Database inside Docker

You can query the database directly in your terminal using docker compose exec:

# View latest vehicles in stock
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT brand, series, model_name, price_gross, exterior_color, location_city FROM vehicles LIMIT 10;"

# View stored factory equipment & options for a vehicle
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT option_code, name, category, is_standard, price_gross FROM vehicle_options LIMIT 15;"

# View crawl run history and telemetry
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT id, brand, status, items_discovered, items_inserted, duration_ms, started_at FROM crawl_runs;"

# View price and status history snapshots
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT vehicle_id, price_gross, monthly_rate, status, recorded_at FROM vehicle_price_history LIMIT 10;"

4. Connect with GUI Database Clients (DBeaver, TablePlus, pgAdmin)

Host: localhost
Port: 5432
Database: oem_stock_db
User: crawler_user
Password: crawler_password

💻 Local Development (without Docker)

1. Run Unit Tests

uv run pytest

2. Run Local Crawl & Persistent Ingestion

Note

: The default local database is automatically stored in data/crawler_stock.db (gitignored). You do not need to specify --db-url unless targeting a custom location or PostgreSQL.

# Crawl New BMW Vehicles (Neuwagensuche)
uv run oem-crawler --brand bmw --stock-type new --limit 50

# Crawl New Mercedes-Benz Vehicles (Neuwagensuche via OneSearch GraphQL + PLS/VPC)
uv run oem-crawler --brand mercedes --stock-type new --limit 50

# Crawl Used Mercedes-Benz Vehicles (Gebrauchtwagen via gebrauchtwagen.mercedes-benz.de)
uv run oem-crawler --brand mercedes --stock-type used --limit 50

# Crawl Both New & Used Vehicles
uv run oem-crawler --brand mercedes --stock-type all --limit 100

# Crawl Used BMW Vehicles (Gebrauchtwagen with mileage & first registration)
uv run oem-crawler --brand bmw --stock-type used --limit 50

# Crawl New Porsche Vehicles (Finder Neuwagen via Next.js RSC)
uv run oem-crawler --brand porsche --stock-type new --limit 50

# Crawl Used / Approved Porsche Vehicles (Finder Gebrauchtwagen via Next.js RSC)
uv run oem-crawler --brand porsche --stock-type used --limit 50

# Crawl Both New & Used Porsche Vehicles
uv run oem-crawler --brand porsche --stock-type all --limit 100

# Or export Porsche stock directly to local JSON
uv run oem-crawler --brand porsche --stock-type new --limit 20 --output-json porsche_new_stock.json

# Crawl New Audi Vehicles (Neuwagenbörse via Stock Car Service scs.audi.de)
uv run oem-crawler --brand audi --stock-type new --limit 50

# Crawl Used Audi Vehicles (Gebrauchtwagenbörse with mileage, dealer GPS & PR options)
uv run oem-crawler --brand audi --stock-type used --limit 50

# Crawl Both New & Used Audi Vehicles
uv run oem-crawler --brand audi --stock-type all --limit 100

# --- Full Scans & Multi-Brand Ingestion ---

# 🚀 Full Scan across ALL Brands (Audi, BMW, Mercedes-Benz, Porsche)
uv run oem-crawler --all-brands --full-scan --delay 0.3

# Full Scan across all brands for New Cars only
uv run oem-crawler --all-brands --stock-type new --full-scan --delay 0.3

# Scan a custom selection of brands in one command
uv run oem-crawler --brand bmw,audi,porsche --full-scan --delay 0.3

3. Inspect Stored Inventory in Terminal (`oem-inspect`)

The project includes an inspection and auditing utility located in src/oem_crawler/tools/:

# View inventory summary and brand breakdown table
uv run oem-inspect --summary

# View first 10 vehicles across all brands
uv run oem-inspect --limit 10

# Filter by brand
uv run oem-inspect --brand audi --limit 5
uv run oem-inspect --brand porsche --limit 10

# Or run via Python module directly
uv run python -m oem_crawler.tools.view_stock --summary

For more options and direct SQLite query recipes, see the Tools Documentation.

8.5 KiB Raw Permalink Blame History