A practical framework for preparing enterprise data for reliable AI-powered answers.
A dataset is AI-ready when an AI system can answer a defined set of business questions from it correctly, consistently, and with evidence that its answers can be trusted.
What looks like a model failure is usually a context failure. An ambiguous metric, an undocumented join or an incomplete reporting period produces a plausible answer from a perfectly executable query. The model reasoned correctly over information that was incomplete.
This article sets out twelve features of an AI-ready data product, the failure each prevents, and how to prove the resulting system is fit for its intended use. It covers structured data; documents need further features, summarised near the end.
Readiness is relative to the questions a system must answer, the people asking them, and the cost of a wrong answer. A marketing dataset can be ready for exploratory segmentation and unready for a board-approved revenue figure. A clinical dataset can support exploration long before it can support a regulated decision.
Before preparing any dataset, define three things:
These requirements decide which controls are mandatory and how readiness will be tested. They also separate three levels of readiness:
| Level | The question it answers |
|---|---|
| Data readiness | Is the data trustworthy, interpretable, current and documented for this use? |
| Access and execution readiness | Can the system find and query approved data correctly, under the user's permissions? |
| Application readiness | Does the complete system answer representative questions accurately, consistently and safely? |
The twelve features below address the first level directly and supply the metadata and controls the other two depend on.
A typical data-answering system works in eight steps, and each step can use only the information and controls it is given.
Two principles follow. The system cannot resolve definitions or limitations that were never written down, so tribal knowledge stays invisible to it. And errors compound: a wrong dataset or join at step 2 or 4 becomes a confident, well-formatted wrong answer at step 8.
Structure tells the system what a row represents, how records relate and which operations are valid. Errors here produce plausible numbers that are wrong.
What it is. One sentence defining what a row represents: one patient, one patient per visit, one adverse event, one order line. State whether the record is an entity, an event, a snapshot or a relationship.
How the AI uses it. Grain fixes the meaning of every count. A question about patients over a table with several events per patient needs a distinct patient count.
Without it. The system counts rows. A patient with three headaches is counted three times, and the number of patients is inflated with nothing in the output to show it.
What it is. Primary keys, foreign keys, composite keys, and the approved relationships between datasets with their expected cardinality.
How the AI uses it. The system selects valid joins and knows whether a join will preserve, multiply or drop records. Column names are an unreliable guide: two tables can both hold a SUBJECT_ID with different meanings, and the true relationship may need two columns.
Without it. A one-to-many join treated as one-to-one duplicates rows and multiplies every sum. A join on a partial key links unrelated records.
In practice. Validate the documented cardinality with tests, and state explicitly where a join changes the grain of the result.
What it is. True data types, one representation per column, documented null meanings, and explicit constraints on required values.
How the AI uses it. Generated queries depend on the difference between dates, numbers, strings, categories and missing values.
Without it. Dates stored as text sort alphabetically. Numbers stored as text compare as strings, so "100" sorts below "9". A null meaning "not applicable" is averaged like one meaning "unknown". The query runs without error, which is what makes the mistake hard to see.
In practice. Test type conformance, uniqueness and required values wherever they can be enforced. Documentation states the rule; tests prove it holds.
A query can be structurally perfect and still answer the wrong question. Semantic features establish what the data means in business terms.
What it is. Plain-language descriptions of each table and important column: what it represents, how it is derived and which questions it supports, including the synonyms people use when asking.
How the AI uses it. Descriptions decide whether a dataset is found at step 2 and which of several similar fields is chosen at step 4.
Without it. A column called STATUS could describe enrolment, data quality or processing. Faced with REVENUE, NET_REVENUE and REV_ADJ, the system picks the one whose name looks right.
In practice. Hold descriptions in the governed catalogue the AI reads, so they cannot drift from a separate wiki.
What it is. The allowed values of every coded field and their meanings, referencing the standard or internal code list they come from, such as MedDRA or CDISC controlled terminology.
How the AI uses it. Filters need the stored value. "Serious" may be held as Y, YES, 1 or a code.
Without it. A filter on seriousness = 'Serious' against a field storing Y returns nothing, and the system reports that no serious events occurred. A formatting mismatch becomes a false clinical statement.
Hierarchies add a second trap. In MedDRA a preferred term can sit under several system organ classes, so summing across classes double-counts unless the aggregation rule is stated. Document valid values, their versions and any hierarchy rules together.
What it is. The unit of every measure, plus currency, scale and conversion rules, held in machine-readable form.
How the AI uses it. Units decide whether values can be compared, averaged or added.
Without it. Lab results in mg/dL from some sites and mmol/L from others are averaged together. Revenue stored in thousands is reported as units. A total across currencies means nothing.
In practice. A label is insufficient where a column mixes units. Normalise the values, or flag the column so the system refuses to aggregate it.
What it is. The approved filters, populations and calculations behind business measures: what counts as an active customer, which accounts are test accounts, which subjects form an analysis population, how a conversion rate is computed.
How the AI uses it. The system applies one consistent rule for which records count and how a metric is calculated.
Without it. Two users ask the same question and get different numbers, because the system invented a slightly different filter each time. In a trial, counting all randomised subjects instead of the protocol-defined analysis population changes every reported rate.
A dataset can be well structured and well defined and still mislead, because it is stale, incomplete or read against the wrong point in time.
What it is. Machine-readable records of the last successful refresh, the expected schedule, the data cut-off, completeness, and whether the dataset is live, locked or historical. Every date field also needs a stated meaning: event date, recorded date or processing date.
How the AI uses it. The system picks the right date field, knows which periods are covered, and discloses the temporal scope of its answer.
Without it. Asked about "this month", the system queries data last refreshed six weeks ago and reports a decline that is only missing data. Filtering on processing date instead of event date moves records between quarters.
In practice. Capture expected latency, late-arriving records and the rule that declares a period complete. Where the data does not yet cover the requested period, the system should say so or decline.
What it is. A documented approach to change over time: whether old values are overwritten or kept with effective dates, which version of each coding dictionary applies, and when published figures were restated.
How the AI uses it. Historical questions require reconstructing what was true then, using effective dates and versioned reference data.
Without it. Last year's sales by region use this year's customer-region assignments, because history was overwritten. Adverse events coded under an older MedDRA version are grouped under a newer classification.
Historical questions also come in two kinds: what we now believe happened, and what we knew or reported at the time. These need different views of the data, and the metadata must say which one a table provides.
Every real dataset has defects that experienced analysts route around from memory. Readiness means making those workarounds discoverable.
What it is. A maintained register attached to the dataset, in three categories:
1900-01-01 birth date.How the AI uses it. The system applies exclusions, recognises suspicious results, qualifies its answers and declines questions beyond the data.
Without it. An average age includes 1900 birth dates. A clinical count includes test patients. A dataset that cannot separate new from returning customers still produces a confident answer to "how many new customers did we acquire?"
In practice. Link each issue to a machine-readable quality rule or test where possible; free-text warnings are the fallback.
Governance decides which data the system may use, whether its answers can be trusted, and who fixes them when they cannot.
What it is. Table- and column-level classifications for personal data, regulated data such as GxP records, commercially restricted data, and sensitive clinical information such as blinded treatment assignments. The classifications drive enforceable access policies.
How the AI uses it. The system discovers and queries only data within the user's authorised scope. The data layer enforces permissions; the application applies disclosure rules to the response.
Without it. An AI assistant becomes the easiest route around existing restrictions. A user without rights to patient identifiers asks a question whose answer reveals them. In a blinded study, a harmless-sounding comparison reveals treatment allocation.
A model that knows a restriction exists may still breach it, so access is enforced at the data or execution layer: row- and column-level controls, limits on combining fields, aggregation thresholds, and audit of every query and answer. Test that prohibited access is actually blocked, including inference from results that are individually permitted.
What it is. Three related things:
How the AI uses it. Lineage identifies the authoritative source. The boundary limits discovery to governed data. Ownership gives every defect a route to a fix.
Without it. Catalogues hold raw extracts, analyst sandboxes, deprecated tables and curated versions of the same information. The system picks whichever matches the question's wording best, often a table nobody maintains. When the answer is wrong, nobody owns the fix, and every user meets the same error.
In practice. Name the authoritative data products, document their lineage, assign accountable owners, and state when each source should be used where several legitimately exist.
This is an example I came across in one of my advisory roles, with a company that runs decentralised clinical trials and remote patient monitoring.
The question:
"How many patients experienced a serious adverse event in Q2?"
The query runs successfully either way, and the answer looks reasonable either way. Only the context decides whether it is right.
| Decision point | Without readiness | With readiness | Feature |
|---|---|---|---|
| What to count | Event rows | Distinct patients | 1 Grain |
| "Serious" | Filter on the word 'Serious', which matches nothing | Approved definition and stored code | 5 Valid values, 7 Conventions |
| Which date | Processing date | Event onset date | 8 Freshness and as-of |
| Is Q2 complete? | Assumed | Checked against the cut-off; disclosed if open | 8 Freshness and as-of |
| Who may see it | Whatever the query returns | Results within the user's permissions; blinded data protected | 11 Sensitivity |
| Is it right? | Trusted because it ran | Reconciled against an independent calculation | Evaluation |
| What is reported | A number | The number, its definition, the cut-off and limitations | 12 Lineage |
The example shows why the features work as a set. Documentation is necessary, and the generation, execution, validation and access controls must also work for the answer to be correct.
The question, from a commercial team at a consumer data provider:
"How many unique consumers in Ohio can we reach by email?"
Identity data adds a complication the clinical example lacks: the entity being counted is itself inferred. A "person" is the output of a matching process that links emails, devices, postal addresses and households, with varying confidence.
| Decision point | Without readiness | With readiness | Feature |
|---|---|---|---|
| What to count | Email addresses | Distinct persons, with households stated separately | 1 Grain |
| How tables connect | Person, email and device tables joined freely; many-to-many links multiply rows | Documented cardinality; joins that preserve person grain | 2 Join paths |
| Which links count | Every link, deterministic or probabilistic | Links above the approved confidence threshold | 7 Conventions |
| "Valid" email | Any email on file | Approved deliverability status codes | 5 Valid values |
| Still reachable? | Emails last seen years ago included | Recency window applied; identifiers decay as people change addresses and devices | 8 Freshness |
| "In Ohio" | Any address ever recorded in Ohio | Current residential address, by the documented address-history rule | 9 History |
| Who may be counted | Everyone in the graph | Opt-outs and suppressions enforced at the data layer; sensitive segments restricted | 11 Sensitivity |
| Which sources | All suppliers, double-counting overlaps and ignoring licence terms | Sources licensed for this use, deduplicated by lineage | 12 Lineage and boundary |
The unready answer is higher on almost every row, which makes it attractive and wrong. In a commercial setting, that inflation goes into a client proposal.
One feature carries more weight here than anywhere else. In many identity graphs, person identifiers can change when the graph is rebuilt, so the same consumer may carry different IDs in different months. Unless the graph version is recorded (feature 9), a trend in "unique consumers" may only reflect a rebuild of the graph.
The twelve features describe what the system needs. Evaluation shows whether the complete system uses it correctly, and it is the only basis for calling a dataset ready.
Collect real questions in the vocabulary of the people who will ask them, including important questions the data cannot answer. This focuses documentation on business value and exposes gaps before deployment.
For each question, establish the correct answer through an independently reviewed calculation, an authoritative report or another trusted reference. Engineer-written SQL is a starting point; it becomes ground truth once a data steward or domain expert confirms it uses the right definition.
Pin every expected answer to a frozen data snapshot, and record its definition, population, period and assumptions. Without the snapshot, a changed result cannot be attributed to the system or to the data.
Size the set by the ways the system could fail. Fifteen to twenty-five questions is a sensible pilot; complexity, variety and the cost of error determine how far it grows. Cover:
A query that executes has passed only the first of five tests:
| Dimension | What it tests |
|---|---|
| Execution | Does the query run safely and successfully? |
| Answer | Does the result match the verified answer within the agreed tolerance? |
| Semantics | Were the right metric, population, filters, joins and period used? |
| Safety and provenance | Were permissions respected and evidence provided? |
| Refusal | Did the system decline when the data could not support an answer? |
Set thresholds in advance, in proportion to the consequences of error. An exploratory assistant and a system supporting clinical or financial decisions warrant different bars.
Model output varies between runs. Repeat representative questions to expose unstable plans and interpretations, and report pass rates and failure types.
Score the system with and without the new metadata or controls, holding everything else constant. The difference shows what the work achieved and which improvements matter most. Interpret it with care: the result depends on the model, prompt, retrieval approach and execution environment.
Re-run the suite after any material change to data, metadata, definitions, models, prompts or query logic, so a fix for one question cannot silently break another. Version the question set and record each configuration.
Latency, cost per correct answer, retry rate and how often users correct the system decide whether a correct solution is also a viable one. These measure the application, and they belong in the readiness decision.
Evaluation pays off when each failure points to a specific improvement, and regression testing confirms the improvement worked.
| Observed failure | Where to look first | Typical improvement |
|---|---|---|
| Counts inflated by duplicates | Grain, join cardinality | Declare grain; validate join paths |
| Right query shape, wrong measure | Business definitions | Rewrite descriptions; call governed metrics |
| Zero results for a known population | Coded values, filters | Document stored values; test filter logic |
| Different answers to the same question | Metric conventions, query generation | Govern definitions; test repeatability |
| Apparent decline in recent periods | Freshness, completeness | Publish cut-offs; add completeness checks |
| Unexplained jump between months | Versioning, restatements | Record graph or dictionary versions; document restatements |
| Confident answer to an unsupported question | Known limitations, refusal behaviour | Register limitations; test unanswerable questions |
| Restricted information in a response | Access enforcement, inference controls | Correct policies; test prohibited queries |
| Answer from a deprecated table | Lineage, product boundary | Name authoritative sources; restrict discovery |
A symptom can have several causes acting together, so confirm the cause before applying the fix.
Protocols, policies, contracts and reports reach the model through retrieval, and they fail in their own ways. The same principle applies: make explicit what an experienced reader would know. The additions that matter most:
Use this to establish whether each feature is documented, implemented and supported by evidence for the intended use. Any item marked not applicable needs a stated reason, especially where security, regulation or consequential decisions are involved.
Much of the metadata already exists, scattered across the enterprise. Clinical submissions carry Define-XML. Catalogues hold schemas, definitions and owners. Transformation pipelines encode derivation logic, and query logs show how analysts really join and filter the data.
Harvest these first. Generate draft descriptions and relationships, then ask owners to review drafts instead of writing from blank pages. Grain, metric definitions, valid values and freshness touch almost every query, so they come first; lineage, historical reconstruction and advanced controls follow as the use case demands.
A practical rollout runs in five moves:
The result is a feedback loop between business demand, metadata, technical controls and measured performance.
Technology only matters in context, and for an AI system the context is everything the data says about itself. A model can only be as right as what it is told.
Start with the questions the business needs answered. Write down what the data behind them means and where it falls short, enforce the rules that matter, and test the answers against results you have verified independently.
Then call a dataset ready for a defined use, on the strength of the evidence.