QB.
GitHubLinkedInContact
All projects

Case 02 / Public data with provenance

AirData.kz

An open air-quality platform that turns fragmented government, community-sensor, and aggregator measurements into traceable public data.

Live platform Open data
Role
Founder; data architecture and product
Status
Live + open dataLive platform; open data updated daily
Period
2019 - present
Key decision
Store raw source responses unchanged before any harmonization, accepting more storage and another processing layer in exchange for traceability.
Evidence
73 cities and towns, 334+ stations, and 10M+ collected measurements. Public-site snapshot: September 8, 2026.
AirData.kz homepage with the Open Air Quality Data for Kazakhstan headline and public impact panel.
The public interface sits downstream of the same quality-controlled data released for reuse.
01 / Problem

Four networks. No common definition of a reading.

Air-quality measurements across Kazakhstan arrive through networks with different station types, units, schemas, histories, and failure modes.

A public chart is easy to make. A public number that can be traced to its source, interpreted in the right unit, and excluded when quality checks fail requires a data product around the chart.

02 / Architecture

Keep the raw reading. Publish the checked one.

Source-specific collectors write untouched responses to Layer 0. The pipeline then converts units, resolves stations, unpivots schemas, and attaches validity and quality flags in a narrow measurement layer. SQL and Soda gates block failed data before Layer 2 exports reach the repository and website.

System map / AirData
AirData three-layer publication systemFour source networks write untouched responses to a raw layer. A harmonization layer converts units, resolves stations, and attaches validity and quality-control flags while preserving lineage. SQL and Soda checks block failed exports before quality-controlled CSV, Parquet, GeoJSON, and website data are published.convert, resolve, flaglineageKGMTAirGradientOpenAQWAQILayer 0Raw responsessource-specific tables, stored untouchedLayer 1Clean and harmonized measurementscommon units, station registry, validity and QC flagsPublication gatesSQL checks + Soda CoreLayer 2Public dataCSV, Parquet, GeoJSON, web
  1. 01Four source networksKGMT, AirGradient, OpenAQ, WAQI
  2. 02Layer 0: rawUntouched source responses
  3. 03Layer 1: cleanUnits, stations, QC flags
  4. 04Quality gatesSQL checks + Soda
  5. 05Layer 2: publicCSV, Parquet, GeoJSON, web
Every public value can be traced through the harmonized row to the source response that arrived.
03 / Decisions

The trade-offs.

01

Keep raw data immutable

Instead of
Normalize incoming values as they arrive and retain only the clean table.
Choice
Store each source response untouched and link every harmonized measurement back to its raw row, value, and unit.
Why it holds
A conversion or quality rule can be audited and rerun without losing what the provider originally sent.
Cost
Storage grows faster, and every public value passes through a second processing layer.
02

Tolerate source failure, not quality failure

Instead of
Fail the whole run when any upstream API is unavailable, or publish despite failed checks.
Choice
Mark source ingestion failures as degraded so other networks can continue, while treating transformation, QA, and export gates as fatal.
Why it holds
One unavailable network should not erase valid readings from the others, but bad data should not become public.
Cost
Freshness can differ by source, so run status and step-level evidence have to remain visible to operators.
03

Harmonize units without erasing the original

Instead of
Expose each provider in its native schema and make every downstream user reconcile them.
Choice
Publish common units and a narrow measurement schema while preserving raw value, raw unit, and conversion notes.
Why it holds
Researchers and journalists get comparable fields without losing the evidence needed to question a conversion.
Cost
Parameter registries, conversion rules, and station matching become maintained data contracts.
04 / Build & evidence

Daily open data, with quality gates.

Built

  • Collectors for four active networks with retries, deduplication, per-source validation, and degraded-run handling.
  • A PostgreSQL and TimescaleDB model with source tables, a unified station registry, a measurement hypertable, and row-level QC flags.
  • A registry-driven Python pipeline with step-level run history, explicit failure policy, and independent SQL and Soda quality gates.
  • Daily public CSV, Parquet, and GeoJSON artifacts plus a multilingual Astro website.

Evaluation

  • Quality is checked at insertion, transformation, statistical cleaning, and pre-export gates.
  • Published hourly files include only measurements that passed the documented checks; raw values and conversion notes stay alongside harmonized values where applicable.

Supported result

  • The open-data repository gives researchers, journalists, educators, and civic technologists documented files without accounts or a paywall.
  • On September 8, 2026, the live platform reported 73 cities and towns, 334+ monitoring stations, a 20-minute collection interval, and more than 10 million measurements.
  • Every public file remains free to access without an account or paywall.
05 / Limits

What this does not solve.

Current limitations

  • Reference stations, low-cost sensors, and aggregators are not equivalent measurement networks; harmonization does not remove those differences.
  • Coverage and history vary by city, pollutant, and source.
  • The current pipeline writes exports directly; its architecture notes identify staged, atomic publication with a manifest as a next reliability step.

Next improvements

  • Publish a manifest with upstream freshness, partition checksums, and row-count invariants for every release.
  • Stage and validate exports before one atomic switch makes them public.
  • Generalize city pipelines so the same registry and contracts cover Almaty, Astana, and Karaganda.