From Data Warehouses to AI: How Enterprise Data Quality Has Changed Over the Last 20 Years

An interview with Marcin Chudeusz, co-founder and CEO of digna

Two decades ago, enterprise data quality looked very different.

Organizations were building centralized data warehouses, business intelligence projects revolved around structured reporting, and most data quality initiatives relied on thousands of manually created validation rules. The objective was simple: ensure the data entering reports was accurate enough for decision-making.

Today, the landscape has changed dramatically.

Cloud platforms, real-time analytics, artificial intelligence, and machine learning have transformed both the volume of data organizations generate and the expectations placed upon it. At the same time, data quality has evolved from a back-office technical concern into a strategic business capability.

Having spent much of his career designing enterprise data warehouses before founding digna in 2020, Marcin Chudeusz has witnessed that transformation firsthand.

We spoke with him about how enterprise data quality has changed over the last twenty years, why traditional approaches eventually reached their limits, and what the next generation of data reliability looks like.

Q: Take us back 20 years. What did data quality look like in the classic data warehouse era?

It was almost entirely manual and rule-based, and it was slow.

You had a data warehouse loaded by overnight batch jobs - ETL running while everyone slept - and data quality meant a person sitting down with the business to define checks in advance.

You'd write rules: this column can't be null, this value must fall in this range, this total must reconcile with that total.

Then you'd maintain those rules by hand, forever. The whole model assumed the world was stable and predictable.

You defined "correct" once, and you policed it.

Q: When did you realize that approach would no longer scale?

The fundamental flaw was that you had to anticipate every failure upfront, and you can't. Real data breaks in ways nobody predicted.

So teams ended up with thousands of rules, huge maintenance burdens, and they still missed the problems that mattered, because the failures that hurt you most are the ones you didn't think to write a rule for.

And because everything was batch, you often found out about a data problem a day late, after bad data had already flowed into reports and decisions.

I spent years watching good teams pour effort into rule libraries that were always one surprise behind reality.

Q: How did the move to the cloud and big data change things?

It changed the scale and the speed of the problem faster than it changed the solutions.

Suddenly data volumes exploded, sources multiplied, and everything got faster, streaming, cloud warehouses, data lakes holding enormous amounts of semi-structured data.

But for a long time, the approach to quality didn't keep up. People were trying to apply the old rule-based, define-everything-upfront mindset to data that was now too big, too fast, and too varied for anyone to manually define rules for.

That gap is really what created the modern data quality crisis. The data got 21st-century infrastructure while quality was still being managed with early-2000s methods.

The mismatch between how data moved and how we checked it kept widening.

Q: Is that the gap that gave rise to "data observability" as a concept?

Yes, exactly.

Data observability emerged because monitoring outputs wasn't enough anymore. The industry borrowed the idea from software engineering; you don't just check the final number, you continuously monitor the health of the system producing it.

Freshness, volume, schema changes, distribution shifts. The insight was that you need to watch the data pipeline the way an SRE watches a production system, in real time, so you catch anomalies as they happen rather than discovering them downstream.

That was a genuine philosophical shift: from periodic inspection to continuous monitoring. It's the thinking behind our own anomaly detection approach, where the system learns what normal looks like and flags deviations automatically instead of waiting for a rule to fire.

Q: And now AI is both the tool and the challenge. How has that changed the picture?

In two ways, and they pull in opposite directions.

First, AI became part of the solution: instead of writing thousands of rules by hand, you can have models learn what normal data behaviour looks like and flag deviations automatically. That's a huge leap, because it finally addresses the problem I couldn't solve as a consultant: catching the failures you didn't anticipate.

Second, and this is the newer pressure, AI became a demanding consumer of data.

When enterprises feed data into machine learning models, automated decisioning, and now large language models, the quality of that data matters more than ever, because the AI acts on it at machine speed with no human sanity-checking each number.

Bad data used to produce a wrong report. Now it can produce a wrong decision, automatically, thousands of times before anyone notices.

Q: So the stakes went up just as the methods matured?

Precisely. And regulation followed.

With frameworks like the EU AI Act, data quality in automated systems is becoming a governance and compliance obligation, not just an engineering nicety.

So the modern enterprise needs both things at once: AI-driven, automated detection for the unknown failures, and the classic explicit rules for the specific checks compliance demands.

That combination is what a mature data quality platform has to offer today.

Q: After 20 years, what's actually changed - and what hasn't?

What's changed is the method. We've gone from manual rules and overnight batches to AI that learns your data and monitors it continuously and in real time.

What hasn't changed is the core truth: data quality is hard because you can't predict every way data will fail, and trust has to be earned continuously, not declared once. Every era rediscovers that.

The tools got dramatically better, but the humility the problem demands is exactly the same as it was when I started.

That's why we built digna to run inside the customer's own environment and keep learning from their data as it changes, because the moment you assume the problem is solved, the data changes and proves you wrong. You can read more about that approach at digna.ai.

Q: One area you've spoken about recently is business observability. Why is that becoming important?

Traditional observability focuses on technical behavior.

Business observability focuses on operational behavior.

Organizations increasingly want answers to questions like:

• Why are transaction volumes changing?
• Why is customer activity behaving differently?
• Why has product demand shifted?

These aren't purely technical questions. They're business questions.

Modern platforms need to connect technical monitoring with business insights.

That's where I believe the industry is heading.

Q: Looking ahead, what do you think enterprise data quality will look like five years from now?

I think we'll continue moving toward unified platforms.

Organizations don't want separate systems for validation, monitoring, observability, analytics, and governance. They want trusted data supported by intelligent automation.

We'll also see much greater emphasis on adaptive systems.

Instead of relying solely on manually defined rules, platforms will continuously learn from changing data while still supporting deterministic validation where regulations require it.

Ultimately, success won't come from collecting more data. It will come from helping organizations understand and trust the data they already have.

About digna

Founded in Austria in 2020, digna is a European data quality and observability platform that helps organizations improve data trust through AI-powered anomaly detection, rule-based validation, data timeliness monitoring, schema tracking, and business analytics.

Its modular architecture enables organizations to combine traditional data quality controls with modern observability and analytics, supporting cloud, hybrid, and on-premises deployments across industries including finance, healthcare, telecommunications, and government.

Learn more about how digna Data Analytics extends observability with business insights and time-series analysis.