Why Enterprises Are Still Running Batch When Real-Time Is Possible
The median enterprise AI program runs on batch pipelines that are six to twenty-four hours stale at the point of inference. This is not because real-time data pipelines are technically infeasible: they have been feasible for years and the tooling has matured significantly. It is because most organizations built their first AI programs on the data infrastructure they already had, which was designed for analytics reporting rather than real-time inference, and never made the infrastructure investment to update it.
The business cost of this lag varies dramatically by use case. A fraud detection model running on 24-hour-old transaction data is missing the most valuable signal: recent transaction velocity and session behavior in the minutes before the fraudulent transaction. A demand forecasting model refreshed daily is responding to yesterday's demand signals rather than the signals that will determine tomorrow's inventory decisions. A customer churn model predicting on last week's engagement data misses the real-time engagement decline that is the strongest leading indicator of churn intent.
The first question is not "how do we build real-time pipelines?" It is "which use cases genuinely require real-time data, and which batch pipelines are performing adequately?" Not every AI use case benefits from real-time data. Investing in streaming infrastructure for use cases where hourly or daily batch is sufficient is an expensive mistake in the opposite direction.
Use Case Latency Requirements: The Decision Framework
The appropriate data pipeline architecture is determined by the business requirement for feature freshness, not by architectural preference. Different use cases have different tolerance for feature staleness, and the infrastructure investment required increases significantly as latency requirements tighten. Most organizations benefit from running multiple pipeline tiers simultaneously rather than standardizing on a single latency tier.
| Use Case Category | Required Latency | Pipeline Type | Example Use Cases |
|---|---|---|---|
| Real-time transaction decisions | Under 100ms | Streaming + Online Feature Store | Fraud detection, real-time credit decisioning, dynamic pricing at transaction time |
| Session-aware personalization | Under 1 second | Micro-batch or Streaming | Real-time product recommendation, content ranking, search personalization |
| Operational monitoring and alerting | Under 5 minutes | Near-real-time streaming | Equipment anomaly detection, supply chain disruption signals, customer service routing |
| Hourly operational decisions | Under 1 hour | Micro-batch (15-60 min) | Demand signal updates, inventory optimization, predictive maintenance scheduling |
| Daily strategic decisions | 24 hours | Daily batch | Credit risk scoring, churn propensity (weekly cadence), next-best-action (daily) |
| Weekly analytics and planning | 7 days | Weekly batch | Market mix modeling, strategic demand forecasting, long-horizon capacity planning |
The Four-Layer Real-Time AI Pipeline Architecture
A production-grade real-time data pipeline for AI has four architectural layers. Organizations that skip layers create pipelines that work in development but fail under production load, or that lack the observability needed to diagnose failures when they occur.
Ingestion
Event Streaming and Data Ingestion
The ingestion layer captures events from source systems (transactional databases, application event streams, IoT sensors, clickstream events) and delivers them to the processing layer with guaranteed delivery semantics. Apache Kafka is the standard for high-throughput enterprise event streaming. Amazon Kinesis and Google Pub/Sub are the cloud-managed alternatives with lower operational overhead.
Processing
Stream Processing and Feature Computation
The processing layer applies transformation logic to compute feature values from raw events. Apache Flink provides the strongest exactly-once processing semantics and stateful stream processing for complex aggregations. Spark Structured Streaming offers a unified batch/streaming API that reduces the codebase complexity of maintaining separate training and serving pipelines. Both require significant operational expertise.
Storage
Online Feature Store and Serving Cache
Computed feature values are written to a low-latency online store that the model inference service reads at prediction time. Redis is the standard choice for sub-millisecond read latency. DynamoDB provides better horizontal scaling for very high read volumes. Apache Cassandra is used when wide-column storage and geographic distribution are required. Latency SLAs for the serving layer should be defined before technology selection.
Observability
Pipeline Monitoring and SLA Management
Real-time pipelines fail in ways that batch pipelines do not: consumer lag accumulates, partition rebalancing causes latency spikes, and exactly-once processing semantics can break silently. Observability requires monitoring of consumer lag, end-to-end latency distributions, processing throughput, error rates, and feature freshness in the online store. Pipeline health monitoring is not optional at production scale.
Four Streaming Architecture Patterns
The appropriate streaming architecture depends on feature complexity, latency requirements, and team capability. Organizations that jump to the most sophisticated pattern before they have mastered simpler ones typically produce brittle infrastructure. Start with the simplest pattern that meets your latency requirements.
The Five Failure Modes of Real-Time AI Pipelines
Real-time AI pipelines fail in ways that are distinct from batch pipelines and that are harder to detect before they cause production incidents. Understanding these failure modes before building prevents the most expensive surprises.
Consumer lag accumulation
Stream processing consumers fall behind the event production rate, causing the online feature store to serve increasingly stale values without any visible error. The fraud model receives technically valid features that are 45 minutes old during a traffic spike. Standard monitoring shows green. The fraud model's effective latency has silently degraded to batch-equivalent. Consumer lag monitoring with hard SLA alerts is the only defense.
Late-arriving event handling failures
Events arrive at the stream processor after the window they should have been assigned to has already closed. Without explicit watermark configuration, late events are either silently dropped or assigned to the wrong window, corrupting rolling aggregation features. For transaction velocity features in fraud detection, a 5-second late-arriving event that is incorrectly dropped reduces apparent transaction velocity and increases fraud risk.
Online store cold-start after failures
When a streaming pipeline restarts after a failure, the online store may contain stale or partial feature values until the pipeline has caught up. Models that query the online store during the catch-up period receive degraded feature values without any indication that recovery is in progress. Cold-start protocols with fallback to batch-computed defaults prevent silent degradation during recovery windows.
Transformation inconsistency between training and serving
The same feature transformation implemented in different frameworks for training (PySpark) and serving (Flink or Kafka Streams) produces different results due to differences in NULL handling, floating-point precision, or time zone treatment. The model trains on feature A and serves on feature A-prime. The difference is subtle enough to pass feature validation but large enough to degrade production performance in specific edge cases. This is the most common cause of unexplained production degradation in streaming AI programs.
State store sizing failures under load
Stateful stream processors maintain state (windowed aggregations, session state, entity profiles) in an embedded key-value store. Under unexpected traffic spikes or cardinality growth, state stores fill and processors begin failing or evicting state. State store capacity planning based on average rather than peak cardinality is the most common sizing mistake. State store overflow causes exactly-once semantics to degrade to at-most-once, which corrupts aggregation features silently.
Technology Selection: Kafka, Flink, and the Managed Alternatives
The technology selection for streaming pipelines has three decision points: the event bus, the stream processor, and the online serving store. Each decision has significant operational implications that compound over time.
Event bus selection is effectively Kafka versus managed alternatives. Apache Kafka on Kubernetes or via Confluent Platform provides the highest throughput and lowest latency but requires significant operational expertise. Amazon Kinesis, Google Pub/Sub, and Azure Event Hubs are fully managed alternatives with lower operational burden but reduced customization and higher per-event costs at scale. For organizations with strong Kafka expertise, self-managed Kafka on cloud-managed Kubernetes (EKS, GKE) is typically the most cost-effective option at production scale. For organizations without streaming infrastructure expertise, starting with a managed service and migrating to self-managed Kafka when scale justifies it is the more pragmatic path.
Stream processor selection depends primarily on feature complexity and team capability. Apache Flink provides the strongest exactly-once guarantees, the richest stateful processing API, and the best performance for complex windowed aggregations. It also requires the most operational expertise. Spark Structured Streaming is the better choice for organizations with existing Spark infrastructure and teams, offering a unified batch/streaming API that reduces code duplication between training and serving feature computation. Kafka Streams is appropriate for simpler transformation logic that can run as part of the Kafka consumer without a separate processing cluster.
Online store selection is primarily a latency and scale decision. Redis provides the lowest latency (sub-millisecond read at p99) and the richest data structure support but requires careful memory planning and is expensive at large feature set sizes. DynamoDB provides better scaling headroom for very high read volumes at the cost of slightly higher latency (single-digit milliseconds). Cassandra is the choice for geographic distribution requirements or very large feature set sizes where Redis cost becomes prohibitive.
