Data
Ingest anything. Destroy nothing. Explain everything.
The ingestion layer does not care whether information arrives from PostgreSQL, Oracle, SAP, Excel, Kafka, an API, a PDF, a GIS file, object storage or a sensor. Everything becomes a Source with an owner, a sensitivity and a refresh policy.
01
The Source object
A source is registered, not just connected. Ownership and sensitivity are recorded before the first byte is read.
SOURCE | +-- source_id +-- organization +-- connector +-- credentials_reference +-- schema +-- sensitivity +-- refresh_policy +-- owner +-- provenance
02
Raw is immutable
Curation is a layer on top of the original, never a replacement for it.
Source | Connector | Raw dataset <- never destroyed | Version | Transformation | Curated dataset
Because the raw layer survives, a disputed figure can be recomputed from the original record years later, including with a corrected transformation.
03
Lineage
Every displayed number can be walked back to the record and moment it came from.
Dashboard number | Metric | Model | Curated dataset | Transformation #<id> | Raw dataset version #<n> | Source system | Ingestion timestamp
04
Entity resolution
Government and company data is messy. The same organization appears under a trade name in one register, a legal name in another and a domain in a third.
ENTITY ORG-<id> Canonical name AURANT KhanFly LLC Aliases KhanFly KhanFly LLC AURANT KhanFly Linked records Source A -> record <id> Source B -> record <id> Source C -> record <id> MATCH PROBABILITY 0.97 domain match +0.40 registration match +0.35 phone match +0.15 name similarity +0.07
Uncertainty is retained
The system never silently declares two records the same. It states a probability and the signals behind it, and a human can dispute either.
Merges are reversible
A canonical entity is a view over linked records, not a destructive merge. Unlinking restores the original records untouched.
05
Data quality gates
schema
Shape
Types, required fields and unit checks run at ingestion; failures quarantine the version rather than the whole source.
freshness
Freshness
Each source declares a refresh policy; a stale source is visibly stale everywhere it is used.
coverage
Coverage
Missing regions, periods or categories are reported as gaps rather than silently treated as zero.