The term "AI factory" has gained traction across enterprise technology teams, and like most popular terms, it risks meaning everything and nothing. At its core, the idea is straightforward: a repeatable, governed pipeline that moves data through feature engineering, model training, validation, deployment, and monitoring, producing reliable predictions at the other end. This is a distributed systems problem. The data flows through heterogeneous stages, each with different latency characteristics, failure modes, and ownership boundaries. The concerns are familiar from other areas of distributed systems design: observability across boundaries, failure handling, coordination costs, and the tension between flexibility and control.

This article looks at what an AI factory actually requires at the architecture level, how governance gates enforce quality between stages, why data lineage and model versioning are structural rather than cosmetic concerns, what failure modes are specific to ML pipelines, and what observability looks like when the "service" is a training job that runs for hours. It builds on earlier discussions of hybrid pipeline placement and agentic system complexity, focusing here on the structural patterns that make these pipelines reliable enough for production use.

What an AI factory is (and is not)

An AI factory is a system for producing and maintaining models. Not a one-off training script, not a notebook that a data scientist runs manually, not a proof of concept that works on a laptop. It is the infrastructure that turns experimentation into repeatable production capability. The analogy to manufacturing is deliberate: raw materials (data) enter, pass through a series of processing stages with quality checks, and finished goods (deployed models serving predictions) emerge.

What it is not: a single product, a vendor platform, or a silver bullet. Many organisations adopt an ML platform and assume the factory problem is solved. The platform provides tooling, but the factory is the end-to-end process, including the organisational agreements about who owns each stage, what "good enough" means at each gate, and how failures are handled. You can have excellent tooling and a dysfunctional factory if the process around it is unclear.

The factory framing is useful because it forces attention on the whole pipeline rather than individual stages. Optimising training speed is pointless if the bottleneck is a manual approval step that takes two weeks. Improving model accuracy is wasted if the deployment process is fragile and rollbacks are untested. Systems thinking, the kind explored across these notes, applies directly.

Pipeline stages and handoffs

A typical AI factory pipeline includes the following stages, though the exact decomposition varies by organisation.

  • Data ingestionCollecting raw data from source systems. This involves connectors, scheduling, schema validation, and handling late-arriving or out-of-order records. Data quality problems here propagate through every downstream stage.
  • Feature engineeringTransforming raw data into features suitable for model training. This may include aggregation, normalisation, encoding categorical variables, and joining data from multiple sources. Feature stores provide a shared repository so that the same feature definitions are used in training and serving.
  • Model trainingRunning training jobs against prepared feature sets. This stage is compute-intensive and often runs on specialised hardware (GPUs, TPUs). Training configurations, hyperparameters, and random seeds should be captured for reproducibility.
  • Validation and evaluationAssessing model quality against held-out test data, fairness criteria, and business metrics. This is where governance gates are most critical. A model that passes statistical tests but violates regulatory constraints should not proceed.
  • DeploymentPackaging the validated model and deploying it to a serving environment. This may be a real-time inference endpoint, a batch scoring job, or an embedded model in an edge device. Canary deployments and shadow scoring reduce the risk of deploying a degraded model.
  • MonitoringTracking model performance in production against the metrics established during validation. Detecting data drift, prediction drift, and performance degradation. Triggering retraining when metrics fall below thresholds.

The handoffs between stages are where most problems occur. Each stage typically has different tooling, different teams, and different cadences. Data engineers manage ingestion. ML engineers handle training. Platform teams manage deployment infrastructure. The interfaces between these teams, what artefacts are passed, what metadata accompanies them, what contracts govern acceptable inputs, need to be explicit. Specs and contracts are as relevant here as they are in traditional software systems.

Governance gates

A governance gate is a checkpoint between pipeline stages that verifies whether an artefact meets defined criteria before it proceeds. Gates serve multiple purposes: quality assurance, regulatory compliance, auditability, and risk management.

Between data ingestion and feature engineering, a gate might verify that incoming data meets schema expectations, that no personally identifiable information has leaked into columns that should be anonymised, and that data volumes are within expected ranges. Anomalous drops in data volume can indicate upstream source failures that would silently degrade downstream models.

Between training and validation, a gate checks that the training run completed successfully, that metrics exceed minimum thresholds, and that the model's behaviour on specific test cases (particularly edge cases and fairness-sensitive subgroups) is acceptable. Automated gates handle the statistical checks. Human review gates, slower but sometimes necessary, handle the cases where judgement is required: does this model's behaviour align with business intent?

Between validation and deployment, a gate confirms that the model artefact is properly versioned, that its lineage (which data, which features, which training configuration) is recorded, and that deployment targets are healthy. In regulated industries, this gate may also require sign-off from a compliance officer or a risk committee.

The important property of gates is that they are enforced, not advisory. A pipeline that allows bypassing gates "just this once" will have them bypassed routinely. This is the same principle that applies to invariants in formal methods: constraints that are not enforced are constraints that will be violated.

Data lineage and model versioning

In traditional software, you can inspect a deployed binary and trace it back to a specific commit, build configuration, and set of dependencies. ML models require the same traceability, but the dependency graph is wider. A deployed model depends not just on code but on a specific dataset, a specific feature engineering pipeline, specific hyperparameters, and often a specific random seed.

Data lineage records which data was used at each stage and how it was transformed. When a model produces unexpected predictions in production, lineage allows you to trace back to the training data and identify whether the issue is a data quality problem, a feature engineering bug, or a genuine shift in the underlying distribution. Without lineage, debugging model behaviour becomes guesswork.

Model versioning goes beyond storing model files with version numbers. Effective versioning captures the full context: the code version of the training pipeline, the dataset version, the feature store snapshot, the hyperparameters, the validation metrics, and the deployment configuration. This allows you to reproduce any historical model, compare versions meaningfully, and roll back with confidence.

The tooling for this has matured significantly. Model registries, experiment trackers, and feature stores provide the mechanical infrastructure. The harder part is organisational discipline: ensuring that every training run is logged, every dataset is versioned, and every deployment is linked to its source artefacts. Teams that skip this step during early experimentation often regret it when they need to audit a production model's provenance six months later.

Failure modes and rollback

ML pipelines have failure modes that differ from traditional service architectures, in addition to sharing the usual ones.

Data drift occurs when the statistical properties of incoming data shift relative to the training data. A model trained on pre-pandemic purchasing patterns will perform poorly on pandemic-era data. Drift can be gradual or sudden, and detecting it requires monitoring input feature distributions, not just output prediction metrics. By the time prediction accuracy degrades measurably, the drift may have been present for weeks.

Training divergence happens when a training run produces a model that is statistically valid on test data but operationally problematic. This can occur due to overfitting, label noise, or subtle bugs in feature engineering that only manifest under certain data distributions. Validation gates catch many of these cases, but not all.

Deployment failures in ML systems include the usual infrastructure problems (container crashes, resource exhaustion, network issues) plus ML-specific ones: model serialisation incompatibilities, feature serving latency that exceeds inference time budgets, and version mismatches between the feature pipeline used in training and the one used in serving. This last category, training-serving skew, is one of the most common and insidious sources of degraded model performance.

Rollback must be a first-class capability. When a newly deployed model underperforms, the system needs to revert to the previous version quickly and reliably. This means the previous model version must remain available, the serving infrastructure must support rapid switching, and the monitoring system must detect the need for rollback before the impact becomes widespread. The resilience patterns that apply to service deployments, canary releases, circuit breakers, blast radius containment, apply equally to model deployments.

Observability requirements

Observability for an AI factory pipeline extends beyond the standard metrics, logs, and traces model for request-serving systems. The pipeline includes long-running batch jobs, asynchronous handoffs, and artefacts that change infrequently but with high impact.

  • Pipeline-level tracingA trace that follows a dataset from ingestion through feature engineering, training, validation, and deployment. This is analogous to distributed tracing across microservices, but the "request" is a training run that may span hours or days.
  • Data quality metricsMonitoring input data for schema violations, null rates, distribution shifts, and volume anomalies. These are leading indicators. Catching a data quality problem at ingestion is far cheaper than discovering it through degraded predictions in production.
  • Model performance metricsTracking prediction accuracy, latency, and throughput in production. Segmented by relevant dimensions (customer segment, geography, time of day) to detect localised degradation that aggregate metrics would mask.
  • Drift detectionStatistical tests comparing current input distributions and prediction distributions against baselines. Alerts when drift exceeds thresholds, triggering investigation or automatic retraining.
  • Governance audit logsRecording every gate decision, every human approval, every override. These logs serve both operational purposes (understanding why a particular model was deployed) and compliance purposes (demonstrating to regulators that the process was followed).

The challenge with pipeline observability is that the feedback loops are long. In a request-serving system, a bug in a new deployment manifests within minutes. In an ML pipeline, a subtle data quality issue introduced during ingestion might not surface as degraded predictions for days or weeks. This makes proactive monitoring of intermediate stages, not just end-to-end outcomes, essential.

The architecture of an AI factory is fundamentally a distributed systems problem with domain-specific constraints. The stages are heterogeneous. The failure modes combine infrastructure failures with statistical ones. The governance requirements add gates that do not exist in traditional pipelines. And the observability needs span timescales from milliseconds (inference latency) to weeks (drift detection). Teams that approach this with the same rigour they apply to production service architecture, explicit contracts, tested failure handling, comprehensive observability, and enforced governance, are the ones that move from experimental AI to reliable production capability.

Related notes

Back to topic • Back to Notes index