213 lines
8.5 KiB
Markdown
213 lines
8.5 KiB
Markdown
# OEM Vehicle Stock Crawler
|
|
|
|
A production-ready, modular platform for crawling car manufacturer (OEM) inventory, tracking vehicle stock, specifications, optional equipment, and price changes over time.
|
|
|
|
---
|
|
|
|
## 📁 Project Architecture
|
|
|
|
```
|
|
oem_crawler/
|
|
├── Dockerfile # Lean production container using uv
|
|
├── docker-compose.yml # PostgreSQL 16 + crawler container orchestrator
|
|
├── .env.example # Environment configuration template
|
|
├── pyproject.toml # Dependencies and CLI entrypoints (`oem-crawler`, `oem-inspect`)
|
|
├── schema.sql # PostgreSQL database DDL
|
|
│
|
|
├── data/ # Development SQLite databases (gitignored)
|
|
│ └── crawler_stock.db # Default local SQLite database file
|
|
│
|
|
├── src/oem_crawler/
|
|
│ ├── config.py # Environment configuration (pydantic-settings)
|
|
│ ├── cli.py # Main CLI application (`oem-crawler`)
|
|
│ │
|
|
│ ├── tools/ # Developer & stock inspection tools
|
|
│ │ ├── README.md # Tools documentation and sqlite3 recipes
|
|
│ │ └── view_stock.py # Terminal inventory inspector (`oem-inspect`)
|
|
│ │
|
|
│ ├── db/ # Database Layer
|
|
│ │ ├── models.py # SQLAlchemy 2.0 ORM mappings
|
|
│ │ ├── schemas.py # Pydantic v2 validation models & SHA-256 hashing
|
|
│ │ ├── session.py # Connection and session factory
|
|
│ │ └── pipeline.py # Ingestion pipeline, change-detection & delisting
|
|
│ │
|
|
│ └── crawlers/ # Modular OEM Crawlers
|
|
│ ├── base.py # BaseCrawler interface & CrawlerRegistry
|
|
│ ├── audi/ # Audi Implementation
|
|
│ │ ├── crawler.py # Stock Car Service (SCS) REST client
|
|
│ │ └── parser.py # PR option codes, EV specs & GPS coordinates
|
|
│ ├── bmw/ # BMW Implementation
|
|
│ │ ├── crawler.py # REST API client & pagination
|
|
│ │ └── parser.py # OEM response normalization
|
|
│ ├── mercedes/ # Mercedes-Benz Implementation
|
|
│ │ ├── crawler.py # OneSearch GraphQL + PLS + VPC client
|
|
│ │ └── parser.py # OneWeb normalization logic
|
|
│ │ └── query.py # Tested GraphQL query definitions
|
|
│ └── porsche/ # Porsche Implementation
|
|
│ ├── crawler.py # Finder Next.js RSC client & pagination
|
|
│ └── parser.py # Option C, equipment & metadata normalization
|
|
│
|
|
└── tests/ # Automated Test Suite
|
|
├── test_schema.py # Schema & hashing unit tests
|
|
├── test_audi_parser.py # Audi parser & registry unit tests
|
|
├── test_bmw_parser.py # BMW parser & registry unit tests
|
|
├── test_mercedes_parser.py # Mercedes parser unit tests
|
|
├── test_porsche_parser.py # Porsche parser unit tests
|
|
└── test_pipeline.py # Ingestion pipeline unit tests
|
|
```
|
|
|
|
---
|
|
|
|
## 🔌 Adding a New Car Manufacturer Crawler
|
|
|
|
Adding a new OEM crawler (e.g. Mercedes, Audi, Tesla, Porsche) takes 2 simple steps:
|
|
|
|
1. Create a new folder under `src/oem_crawler/crawlers/<brand>/` (e.g. `mercedes/`):
|
|
2. Implement your crawler inheriting from `BaseCrawler` and register it:
|
|
|
|
```python
|
|
from oem_crawler.crawlers.base import BaseCrawler, CrawlerRegistry
|
|
from oem_crawler.db.schemas import VehicleIngestSchema
|
|
|
|
@CrawlerRegistry.register("mercedes")
|
|
class MercedesStockCrawler(BaseCrawler):
|
|
@property
|
|
def brand_name(self) -> str:
|
|
return "Mercedes-Benz"
|
|
|
|
def crawl(self, max_vehicles=None, page_size=20, search_filter=None):
|
|
# 1. Fetch from OEM API / site
|
|
# 2. Parse into VehicleIngestSchema
|
|
# 3. Yield parsed vehicle
|
|
...
|
|
```
|
|
|
|
The CLI (`oem-crawler --brand mercedes`) and pipeline will automatically discover and support it.
|
|
|
|
---
|
|
|
|
## 🐳 Running with Docker
|
|
|
|
### 1. Start PostgreSQL Database & Run a Crawl
|
|
```bash
|
|
docker compose up --build
|
|
```
|
|
|
|
### 2. Run Ad-hoc Crawl inside Docker
|
|
```bash
|
|
# Fetch 100 vehicles for BMW
|
|
docker compose run --rm crawler --brand bmw --limit 100
|
|
```
|
|
|
|
### 3. Query PostgreSQL Database inside Docker
|
|
|
|
You can query the database directly in your terminal using `docker compose exec`:
|
|
|
|
```bash
|
|
# View latest vehicles in stock
|
|
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
|
|
"SELECT brand, series, model_name, price_gross, exterior_color, location_city FROM vehicles LIMIT 10;"
|
|
|
|
# View stored factory equipment & options for a vehicle
|
|
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
|
|
"SELECT option_code, name, category, is_standard, price_gross FROM vehicle_options LIMIT 15;"
|
|
|
|
# View crawl run history and telemetry
|
|
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
|
|
"SELECT id, brand, status, items_discovered, items_inserted, duration_ms, started_at FROM crawl_runs;"
|
|
|
|
# View price and status history snapshots
|
|
docker compose exec postgres psql -U crawler_user -d oem_stock_db -c \
|
|
"SELECT vehicle_id, price_gross, monthly_rate, status, recorded_at FROM vehicle_price_history LIMIT 10;"
|
|
```
|
|
|
|
### 4. Connect with GUI Database Clients (DBeaver, TablePlus, pgAdmin)
|
|
- **Host**: `localhost`
|
|
- **Port**: `5432`
|
|
- **Database**: `oem_stock_db`
|
|
- **User**: `crawler_user`
|
|
- **Password**: `crawler_password`
|
|
|
|
---
|
|
|
|
## 💻 Local Development (without Docker)
|
|
|
|
### 1. Run Unit Tests
|
|
```bash
|
|
uv run pytest
|
|
```
|
|
|
|
### 2. Run Local Crawl & Persistent Ingestion
|
|
|
|
> **Note**: The default local database is automatically stored in `data/crawler_stock.db` (gitignored). You do not need to specify `--db-url` unless targeting a custom location or PostgreSQL.
|
|
|
|
```bash
|
|
# Crawl New BMW Vehicles (Neuwagensuche)
|
|
uv run oem-crawler --brand bmw --stock-type new --limit 50
|
|
|
|
# Crawl New Mercedes-Benz Vehicles (Neuwagensuche via OneSearch GraphQL + PLS/VPC)
|
|
uv run oem-crawler --brand mercedes --stock-type new --limit 50
|
|
|
|
# Crawl Used Mercedes-Benz Vehicles (Gebrauchtwagen via gebrauchtwagen.mercedes-benz.de)
|
|
uv run oem-crawler --brand mercedes --stock-type used --limit 50
|
|
|
|
# Crawl Both New & Used Vehicles
|
|
uv run oem-crawler --brand mercedes --stock-type all --limit 100
|
|
|
|
# Crawl Used BMW Vehicles (Gebrauchtwagen with mileage & first registration)
|
|
uv run oem-crawler --brand bmw --stock-type used --limit 50
|
|
|
|
# Crawl New Porsche Vehicles (Finder Neuwagen via Next.js RSC)
|
|
uv run oem-crawler --brand porsche --stock-type new --limit 50
|
|
|
|
# Crawl Used / Approved Porsche Vehicles (Finder Gebrauchtwagen via Next.js RSC)
|
|
uv run oem-crawler --brand porsche --stock-type used --limit 50
|
|
|
|
# Crawl Both New & Used Porsche Vehicles
|
|
uv run oem-crawler --brand porsche --stock-type all --limit 100
|
|
|
|
# Or export Porsche stock directly to local JSON
|
|
uv run oem-crawler --brand porsche --stock-type new --limit 20 --output-json porsche_new_stock.json
|
|
|
|
# Crawl New Audi Vehicles (Neuwagenbörse via Stock Car Service scs.audi.de)
|
|
uv run oem-crawler --brand audi --stock-type new --limit 50
|
|
|
|
# Crawl Used Audi Vehicles (Gebrauchtwagenbörse with mileage, dealer GPS & PR options)
|
|
uv run oem-crawler --brand audi --stock-type used --limit 50
|
|
|
|
# Crawl Both New & Used Audi Vehicles
|
|
uv run oem-crawler --brand audi --stock-type all --limit 100
|
|
|
|
# --- Full Scans & Multi-Brand Ingestion ---
|
|
|
|
# 🚀 Full Scan across ALL Brands (Audi, BMW, Mercedes-Benz, Porsche)
|
|
uv run oem-crawler --all-brands --full-scan --delay 0.3
|
|
|
|
# Full Scan across all brands for New Cars only
|
|
uv run oem-crawler --all-brands --stock-type new --full-scan --delay 0.3
|
|
|
|
# Scan a custom selection of brands in one command
|
|
uv run oem-crawler --brand bmw,audi,porsche --full-scan --delay 0.3
|
|
```
|
|
|
|
### 3. Inspect Stored Inventory in Terminal (`oem-inspect`)
|
|
|
|
The project includes an inspection and auditing utility located in `src/oem_crawler/tools/`:
|
|
|
|
```bash
|
|
# View inventory summary and brand breakdown table
|
|
uv run oem-inspect --summary
|
|
|
|
# View first 10 vehicles across all brands
|
|
uv run oem-inspect --limit 10
|
|
|
|
# Filter by brand
|
|
uv run oem-inspect --brand audi --limit 5
|
|
uv run oem-inspect --brand porsche --limit 10
|
|
|
|
# Or run via Python module directly
|
|
uv run python -m oem_crawler.tools.view_stock --summary
|
|
```
|
|
|
|
For more options and direct SQLite query recipes, see the [Tools Documentation](src/oem_crawler/tools/README.md).
|
|
|