Data Quality at Scale: Building a Robust Framework for Complex data Models

As data models grow in complexity and number, ensuring data quality becomes exponentially challenging. At Sopht, we measure the carbon footprint of IT systems through a complex data architecture spanning hundreds of tables with varied data sources and calculation methods.
This talk demonstrates our comprehensive testing framework that maintains data integrity at scale. We'll cover technical and functional unit testing, complexity management, production-like staging environments, enhanced error visibility through custom monitoring, and our latest approach to detecting data discrepancies through automated snapshots.
Join us to learn practical strategies for implementing dev-inspired testing practices that can transform chaotic data environments into reliable, maintainable systems