# OEM Vehicle Stock Crawler

A production-ready, modular platform for crawling car manufacturer (OEM) inventory, tracking vehicle stock, specifications, optional equipment, and price changes over time.

---

## 📁 Project Architecture

```
oem_crawler/
├── Dockerfile                   # Lean production container using uv
├── docker-compose.yml           # PostgreSQL 16 + crawler container orchestrator
├── .env.example                 # Environment configuration template
├── pyproject.toml               # Dependencies and CLI entrypoints (`oem-crawler`, `oem-inspect`)
├── schema.sql                   # PostgreSQL database DDL
│
├── data/                        # Development SQLite databases (gitignored)
│   └── crawler_stock.db         # Default local SQLite database file
│
├── src/oem_crawler/
│   ├── config.py                # Environment configuration (pydantic-settings)
│   ├── cli.py                   # Main CLI application (`oem-crawler`)
│   │
│   ├── tools/                   # Developer & stock inspection tools
│   │   ├── README.md            # Tools documentation and sqlite3 recipes
│   │   └── view_stock.py        # Terminal inventory inspector (`oem-inspect`)
│   │
│   ├── db/                      # Database Layer
│   │   ├── models.py            # SQLAlchemy 2.0 ORM mappings
│   │   ├── schemas.py           # Pydantic v2 validation models & SHA-256 hashing
│   │   ├── session.py           # Connection and session factory
│   │   └── pipeline.py          # Ingestion pipeline, change-detection & delisting
│   │
│   └── crawlers/                # Modular OEM Crawlers
│       ├── base.py              # BaseCrawler interface & CrawlerRegistry
│       ├── audi/                # Audi Implementation
│       │   ├── crawler.py       # Stock Car Service (SCS) REST client
│       │   └── parser.py        # PR option codes, EV specs & GPS coordinates
│       ├── bmw/                 # BMW Implementation
│       │   ├── crawler.py       # REST API client & pagination
│       │   └── parser.py        # OEM response normalization
│       ├── mercedes/            # Mercedes-Benz Implementation
│       │   ├── crawler.py       # OneSearch GraphQL + PLS + VPC client
│       │   └── parser.py        # OneWeb normalization logic
│       │   └── query.py         # Tested GraphQL query definitions
│       └── porsche/             # Porsche Implementation
│           ├── crawler.py       # Finder Next.js RSC client & pagination
│           └── parser.py        # Option C, equipment & metadata normalization
│
└── tests/                       # Automated Test Suite
    ├── test_schema.py           # Schema & hashing unit tests
    ├── test_audi_parser.py      # Audi parser & registry unit tests
    ├── test_bmw_parser.py       # BMW parser & registry unit tests
    ├── test_mercedes_parser.py  # Mercedes parser unit tests
    ├── test_porsche_parser.py   # Porsche parser unit tests
    └── test_pipeline.py         # Ingestion pipeline unit tests
```

---

## 🔌 Adding a New Car Manufacturer Crawler

Adding a new OEM crawler (e.g. Mercedes, Audi, Tesla, Porsche) takes 2 simple steps:

1. Create a new folder under `src/oem_crawler/crawlers/<brand>/` (e.g. `mercedes/`):
2. Implement your crawler inheriting from `BaseCrawler` and register it:

```python
from oem_crawler.crawlers.base import BaseCrawler, CrawlerRegistry
from oem_crawler.db.schemas import VehicleIngestSchema

@CrawlerRegistry.register("mercedes")
class MercedesStockCrawler(BaseCrawler):
    @property
    def brand_name(self) -> str:
        return "Mercedes-Benz"

    def crawl(self, max_vehicles=None, page_size=20, search_filter=None):
        # 1. Fetch from OEM API / site
        # 2. Parse into VehicleIngestSchema
        # 3. Yield parsed vehicle
        ...
```

The CLI (`oem-crawler --brand mercedes`) and pipeline will automatically discover and support it.

---

## 🐳 Running with Docker

### 1. Start PostgreSQL Database & Run a Crawl
```bash
docker compose up --build
```

### 2. Run Ad-hoc Crawl inside Docker
```bash
# Fetch 100 vehicles for BMW
docker compose run --rm crawler --brand bmw --limit 100
```

### 3. Query PostgreSQL Database inside Docker

You can query the database directly in your terminal using `docker compose exec`:

```bash
# View latest vehicles in stock
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT brand, series, model_name, price_gross, exterior_color, location_city FROM vehicles LIMIT 10;"

# View stored factory equipment & options for a vehicle
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT option_code, name, category, is_standard, price_gross FROM vehicle_options LIMIT 15;"

# View crawl run history and telemetry
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT id, brand, status, items_discovered, items_inserted, duration_ms, started_at FROM crawl_runs;"

# View price and status history snapshots
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
  "SELECT vehicle_id, price_gross, monthly_rate, status, recorded_at FROM vehicle_price_history LIMIT 10;"
```

### 4. Connect with GUI Database Clients (DBeaver, TablePlus, pgAdmin)
- **Host**: `localhost`
- **Port**: `5432`
- **Database**: `oem_stock_db`
- **User**: `crawler_user`
- **Password**: `crawler_password`

---

## 💻 Local Development (without Docker)

### 1. Run Unit Tests
```bash
uv run pytest
```

### 2. Run Local Crawl & Persistent Ingestion

> **Note**: The default local database is automatically stored in `data/crawler_stock.db` (gitignored). You do not need to specify `--db-url` unless targeting a custom location or PostgreSQL.

```bash
# Crawl New BMW Vehicles (Neuwagensuche)
uv run oem-crawler --brand bmw --stock-type new --limit 50

# Crawl New Mercedes-Benz Vehicles (Neuwagensuche via OneSearch GraphQL + PLS/VPC)
uv run oem-crawler --brand mercedes --stock-type new --limit 50

# Crawl Used Mercedes-Benz Vehicles (Gebrauchtwagen via gebrauchtwagen.mercedes-benz.de)
uv run oem-crawler --brand mercedes --stock-type used --limit 50

# Crawl Both New & Used Vehicles
uv run oem-crawler --brand mercedes --stock-type all --limit 100

# Crawl Used BMW Vehicles (Gebrauchtwagen with mileage & first registration)
uv run oem-crawler --brand bmw --stock-type used --limit 50

# Crawl New Porsche Vehicles (Finder Neuwagen via Next.js RSC)
uv run oem-crawler --brand porsche --stock-type new --limit 50

# Crawl Used / Approved Porsche Vehicles (Finder Gebrauchtwagen via Next.js RSC)
uv run oem-crawler --brand porsche --stock-type used --limit 50

# Crawl Both New & Used Porsche Vehicles
uv run oem-crawler --brand porsche --stock-type all --limit 100

# Or export Porsche stock directly to local JSON
uv run oem-crawler --brand porsche --stock-type new --limit 20 --output-json porsche_new_stock.json

# Crawl New Audi Vehicles (Neuwagenbörse via Stock Car Service scs.audi.de)
uv run oem-crawler --brand audi --stock-type new --limit 50

# Crawl Used Audi Vehicles (Gebrauchtwagenbörse with mileage, dealer GPS & PR options)
uv run oem-crawler --brand audi --stock-type used --limit 50

# Crawl Both New & Used Audi Vehicles
uv run oem-crawler --brand audi --stock-type all --limit 100

# --- Full Scans & Multi-Brand Ingestion ---

# 🚀 Full Scan across ALL Brands (Audi, BMW, Mercedes-Benz, Porsche)
uv run oem-crawler --all-brands --full-scan --delay 0.3

# Full Scan across all brands for New Cars only
uv run oem-crawler --all-brands --stock-type new --full-scan --delay 0.3

# Scan a custom selection of brands in one command
uv run oem-crawler --brand bmw,audi,porsche --full-scan --delay 0.3
```

### 3. Inspect Stored Inventory in Terminal (`oem-inspect`)

The project includes an inspection and auditing utility located in `src/oem_crawler/tools/`:

```bash
# View inventory summary and brand breakdown table
uv run oem-inspect --summary

# View first 10 vehicles across all brands
uv run oem-inspect --limit 10

# Filter by brand
uv run oem-inspect --brand audi --limit 5
uv run oem-inspect --brand porsche --limit 10

# Or run via Python module directly
uv run python -m oem_crawler.tools.view_stock --summary
```

For more options and direct SQLite query recipes, see the [Tools Documentation](src/oem_crawler/tools/README.md).