Data quality before AI: what the standards actually define

Short answer

How should you check data quality before an AI project?

In my view, start by naming the handful of tables and documents the AI system will draw on and assess those first. Quality means fit for the job in hand. It does not mean perfect. Six dimensions cover most checks. They are completeness, uniqueness, timeliness, validity, accuracy and consistency. No standard sets your pass mark. You decide what is good enough for each use. Then correct the data and improve the process that produced the fault.

§ 1 Why the pipeline comes first

One established way to make an AI system answer from your own material is retrieval. The 2020 paper that introduced retrieval-augmented generation describes a model that combines what it learned in training with a second, external memory. That memory is an index of documents that it searches when it answers.[1] The practical consequence follows directly. If the index holds a duplicate, an out-of-date price list or a mislabelled table, the system can retrieve it and build an answer on it.

Research on AI practitioners points in the same direction. A 2021 study by Google researchers interviewed 53 practitioners working on high-stakes AI in India, East and West Africa and the United States. It found that 92% had experienced what the authors call data cascades. Those are compounding problems that begin with a data issue and surface later, downstream.[2] The study covered fields such as health and conservation. It did not cover marketing. So treat it as evidence of how data problems behave. It is not a statistic about your own sector.

§ 2 What "quality" means

The UK Government Data Quality Framework was published in December 2020. It defines quality as fitness for purpose. Data is good enough when it meets the needs of the people using it for a particular task.[3] A customer list that is fine for counting sales by region may be unfit for personalised email.

To make fitness measurable, practitioners break it into dimensions. A DAMA UK working group published a white paper in October 2013. It defines a data quality dimension as a feature of data that can be measured or assessed against defined standards. The paper explains why the group wrote it. Even professionals did not agree on what the key dimensions were.[4]

§ 3 The six core dimensions

The DAMA UK paper settles on six dimensions.[4] The UK government's guidance uses the same six.[5]

Definitions paraphrased from the DAMA UK paper.[4] The illustrations and the final column are my own and are not taken from either source.

Two points from the sources are easy to miss. Valid is not the same as accurate. A date can be perfectly formatted and still wrong.[5] The DAMA UK paper notes that accuracy cannot be judged from the data alone. It has to be checked against the real thing or against an authoritative reference set.[4]

§ 4 What the standards actually define

Three international standards come up most often. Their full texts are sold by standards bodies and I have not read them in full. What follows is drawn from official catalogue abstracts. For ISO/IEC 25012 it also draws on the summary published by the ISO 25000 portal.

ISO/IEC 25012:2008

This standard defines a general data quality model for data held in a structured format within a computer system. It sorts quality into fifteen characteristics seen from two points of view. Inherent means properties of the data itself. System dependent means properties that rely on the technology holding it.[6] The ISO 25000 portal groups them as shown below.[7]

This adds something to the six dimensions. It asks whether data can be traced, understood, recovered and kept confidential. In my view those questions matter even more once an AI system is reading the data on someone's behalf.

ISO 8000

ISO 8000 is a series of several documents. Part 1, published in 2022, is the overview. Its scope covers the principles of information and data quality, the path to data quality and the structure of the other parts.[8]

ISO/IEC 5259

The newest of the three is aimed at AI. ISO/IEC 5259-1 was published in July 2024. It opens a series on data quality for analytics and machine learning. It supplies the shared concepts, terms and examples for the other parts.[9]

None of these sources gives you a pass mark. The DAMA UK paper is explicit on this. Each organisation selects the dimensions that matter. It sets its own thresholds according to its context and level of risk.[4]

§ 5 Where defects get in

The government framework describes a data lifecycle of six stages. They are plan, collect, prepare and store, use, share, then archive or destroy. One of its five principles is to assess quality throughout that lifecycle instead of only at the end.[3] The diagram applies the idea to a typical marketing data flow.

My reading of the cascade research is that the earlier a defect is caught, the less there is to unpick. A wrong value found at collection is one wrong value. The same value found after it has shaped a forecast or an AI-generated answer is an investigation.

§ 6 A practical order of work

The DAMA UK paper outlines an assessment routine that works as a pre-flight check for any analytics or AI project.[4] In summary:

1. Pick the data items that matter most to the decision or system.
2. Choose the dimensions that apply and decide how much weight each carries.
3. Define, for each one, what good and bad look like.
4. Measure the data against those definitions.
5. Review the results and decide whether the quality is acceptable.
6. Correct the data and improve the process that produced the fault.
7. Repeat on a schedule and watch the trend.

In my opinion the step most worth insisting on before an AI project is the first. Name the handful of tables and documents the system will actually draw on and assess those, instead of trying to certify everything at once.

Storage choices can help. The table formats described in Open table formats explained keep snapshots of earlier table versions (sourced in that entry). That supports the traceability that ISO/IEC 25012 lists. The checks themselves need not involve heavy infrastructure. See When data is not big.