Due to backend changes on some of our data sources, we have completed a full re-haul of our backend data ingestion platform to provide more reliable data. Covering against data risks such as:
- Adding more integrity checks in areas where data is exposed to multiple processes between its source and destination
- Conducting quality checks after data loads
- Data downtime prevention measures
We also identified more work to do in areas such as data observability (freshness, volume anomaly detection) and data lineage tracking.