Architecture
30 Government Feeds ──► Raspberry Pi (Adaptive ETL Engine)
GeoJSON │
ArcGIS ├── Normalise to GeoJSON schema
RSS / XML ├── Geocode missing coordinates
CAP-AU ├── Classify event types
Custom JSON ├── Filter noise + duplicates
│
▼
Supabase (PostgreSQL + PostGIS)
│
▼
Vercel (Next.js API + Website)
│
▼
Your ApplicationA dedicated Raspberry Pi runs the Adaptive ETL (Extract, Transform, Load) Engine 24/7. It polls all 30 feeds on adaptive schedules (every 30 seconds for fast feeds, every 5 minutes for slower ones). Each feed has a YAML configuration that defines the source URL, parser, field mappings, and event type classifications.
Incoming data is normalised into a consistent GeoJSON schema, geocoded if coordinates are missing, deduplicated, and upserted into a PostgreSQL database with PostGIS spatial indexes. The API serves this data via Vercel.
When a feed changes its format or goes down, the engine adapts automatically. Circuit breakers prevent cascading failures. Quality checks flag anomalies. The whole pipeline is self-healing.