The conversation around AI reliability tends to focus on model capability. Larger models, better fine-tuning, improved alignment. These matter, but they address only part of the problem. An LLM that receives stale data will produce stale answers. An LLM that receives contradictory context will produce confused answers. An LLM that receives a truncated document because the context window was mismanaged will produce incomplete answers. The model can be perfect and the output still wrong.

Reliable AI system output depends on what goes in. Data quality, context management, and pipeline determinism are the foundations that formal methods can address directly. These are not model problems. They are systems engineering problems, familiar from decades of work on distributed systems and formal specifications. This article examines why model capability is necessary but insufficient, how data quality can be treated as a formal property, what context management requires in practice, why pipeline determinism matters for reproducibility, and how testing and specification techniques apply to the data layer.

Why model capability is not enough

A language model is a function from input to output. The quality of the output is bounded by two factors: the capability of the model and the quality of the input. The industry has invested enormously in the first factor. The second factor receives less attention, but it is often the binding constraint.

Consider a retrieval-augmented generation (RAG) system. The model's task is to answer questions using retrieved documents. If the retrieval step returns irrelevant documents, the model hallucinates or gives wrong answers. If the retrieval step returns the right documents but they are outdated, the answers are outdated. If the retrieved documents are truncated because of context window limits, the answers are incomplete. In each case, the model is performing correctly given its input. The failure is upstream.

This is not a new observation. The principle that output quality is bounded by input quality has been understood in data engineering for decades. What is new is the scale at which AI systems consume data and the complexity of the pipelines that prepare it. A modern AI application might pull from vector databases, knowledge graphs, real-time event streams, cached API responses, and user conversation history. Each source has its own freshness, consistency, and reliability characteristics. The composition of these sources into a single context window is a systems problem that benefits from the same rigour applied to any distributed data pipeline.

Data quality as a formal property

Data quality is often discussed informally: the data should be "good" or "clean" or "reliable." Formal methods encourage precision. Data quality can be decomposed into specific, checkable properties:

  • CompletenessAll expected data is present. No required fields are null or missing. Coverage metrics indicate the fraction of the expected domain that is represented.
  • ConsistencyData does not contradict itself. Values that represent the same entity agree across sources. Temporal consistency means that data reflects a coherent point in time, not a mix of states from different moments.
  • TimelinessData reflects the current state of the world within acceptable bounds. Staleness is bounded and measurable. This is particularly important for AI systems that answer questions about current facts.
  • AccuracyData values correspond to reality. Measurements are within known error bounds. Labels are correct. This is the hardest property to verify automatically, but it can be approximated through cross-validation and statistical checks.
  • ConformanceData conforms to its schema. Types are correct, ranges are respected, referential integrity is maintained. This is the most straightforward property to check and the most commonly neglected.

Each of these properties can be expressed as a formal specification. Completeness is an invariant: for every record of type T, fields F1 through Fn are non-null. Consistency is a constraint across records: for every pair of records referring to entity E, the value of field F agrees. Timeliness is a temporal property: for every record, the time since last update is less than threshold D.

Once expressed formally, these properties can be checked mechanically. This is not novel technology. Schema validation, constraint checking, and data profiling tools have existed for years. What formal methods add is the discipline of writing these properties explicitly, the ability to compose them, and the connection to a broader verification framework that includes the rest of the system.

Context management for LLMs

Context management is the process of selecting, ordering, and formatting the information that goes into an LLM's context window. It is a surprisingly complex optimisation problem with correctness implications.

The context window is finite. Current models accept tens of thousands to hundreds of thousands of tokens, but the information available to a system often exceeds this by orders of magnitude. Context management must select the most relevant subset and arrange it for maximum utility.

Several concerns arise:

  • RelevanceThe selected content must be relevant to the query. Retrieval quality depends on embedding models, indexing strategies, and query reformulation. Irrelevant context wastes tokens and can mislead the model.
  • OrderingPosition in the context window affects attention. Information at the beginning and end of the context tends to receive more attention than information in the middle. Ordering is not just cosmetic; it affects output quality.
  • DeduplicationMultiple retrieval sources may return overlapping content. Including the same information multiple times wastes context budget and can distort the model's weighting of that information.
  • ConsistencyThe assembled context must be internally consistent. If two retrieved documents disagree on a fact, the context should either resolve the conflict or flag it. Presenting contradictory information without resolution invites hallucination.
  • Truncation safetyWhen content must be truncated to fit the window, the truncation must not break semantic units. Cutting a document mid-sentence or mid-argument creates misleading fragments.

These are specifiable properties. Relevance scoring can be validated against labelled datasets. Ordering strategies can be evaluated systematically. Deduplication can be verified by checking for semantic overlap. Consistency can be checked by cross-referencing key claims. Truncation can be constrained to respect document structure. The discipline of writing specifications for these properties transforms context management from an ad hoc process into an engineered subsystem.

Pipeline determinism and reproducibility

When an AI system produces an incorrect answer, the first question is: why? Answering this question requires reproducing the conditions that led to the output. This means the pipeline must be deterministic, or at least reproducible, at every stage.

Pipeline determinism means that given the same inputs, the pipeline produces the same outputs. This sounds straightforward, but many common practices violate it. Embedding models may be updated without versioning. Vector database indices may be rebuilt with different parameters. Data sources may return different results at different times. Even the order of results from a database query may vary between executions.

Reproducibility requires versioning at every layer:

  • Data versioningSnapshots of the data used at each pipeline stage. This includes raw data, intermediate representations, and final context assemblies.
  • Model versioningPinned versions of embedding models, reranking models, and the LLM itself. A change in any model component can change the output.
  • Configuration versioningThe parameters of every pipeline stage: retrieval thresholds, chunk sizes, context window budgets, prompt templates. These are as much part of the system's behaviour as the code.
  • Execution loggingA record of the actual inputs and outputs at each stage for each request. This enables post-hoc diagnosis when outputs are incorrect.

From a formal methods perspective, pipeline determinism is a functional specification: pipeline(inputs, config, models) = output. Violations of determinism indicate hidden state or uncontrolled dependencies. Identifying and eliminating these is standard systems engineering practice, applied to the specific context of AI pipelines.

Testing and monitoring data quality

Testing data quality requires different techniques than testing code. Code tests exercise specific execution paths. Data tests check statistical properties of datasets and structural properties of individual records.

Schema validation is the first layer. Every record entering the pipeline should conform to its expected schema. Types, ranges, required fields, and referential constraints can be checked at ingestion time. This catches obvious errors, corrupted exports, changed API formats, missing fields, before they propagate downstream.

Statistical tests form the second layer. Data distributions should be stable over time. A sudden shift in the distribution of a feature, known as data drift, may indicate a data quality problem or a genuine change in the underlying phenomenon. Either way, the AI system's behaviour may change, and the shift should be detected and investigated.

Embedding quality monitoring is specific to AI pipelines. If the embedding model changes or the input distribution shifts, retrieval quality may degrade. Monitoring retrieval relevance scores over time, and alerting on significant drops, provides early warning of context quality problems.

These monitoring requirements fit naturally into observability pipelines. Data quality metrics, schema validation results, drift detection alerts, and retrieval relevance scores are telemetry signals that belong alongside latency, error rates, and throughput in the system's monitoring dashboard.

Specifications for data pipelines

Putting it together, a formally specified data pipeline for an AI system includes:

  • Input specificationsSchemas and quality constraints for each data source. These are the preconditions that the pipeline assumes.
  • Transform specificationsFor each processing stage, a description of what it does to the data: what properties it preserves, what properties it establishes, and what invariants it maintains.
  • Output specificationsThe properties that the assembled context must satisfy before it is passed to the model. Relevance bounds, consistency requirements, completeness criteria, and size constraints.
  • Monitoring specificationsThe metrics to track, the thresholds for alerts, and the expected ranges for statistical properties. These are the runtime checks that complement the static specifications.

This is not a heavyweight process. Many of these specifications are simple assertions or schema definitions. The value comes from writing them explicitly, checking them automatically, and treating violations as incidents rather than warnings. Formal verification tools are increasingly accessible for this kind of lightweight specification, particularly when AI assists with specification generation.

The argument is not that every data pipeline needs a complete formal proof of correctness. The argument is that the properties that make AI output reliable, data completeness, consistency, timeliness, and context relevance, are specifiable, checkable, and monitorable. Treating them with the same discipline applied to protocol correctness or system invariants is a practical investment. The model is only as reliable as the data it receives.

Related notes

Back to topic • Back to Notes index