Choose the serving mode from the factory decision deadline, not from how modern the model looks. Use batch inference when people act once per shift or day, micro-batch when a short review queue is acceptable, and online inference only when an event must change an operational decision within seconds or milliseconds.
This choice sounds technical, but it is primarily an operating-design question: when is the data complete, who acts on the output, what happens if the model is unavailable, and which service measure matters?
Batch and online describe serving, not training
Inference is the stage where a trained model produces an output for new data. The 2026 open textbook Introduction to Machine Learning Systems, Volume I, Model Serving explains that serving systems trade latency, throughput and resource use. It also notes that batching can improve throughput while adding waiting time.
In an online service, each request reaches a deployed endpoint and receives a response. In a batch job, a known input set is read from storage, processed, and written back for later use. A third pattern—micro-batch—runs the same logic every few minutes. It is often enough for supervisor review without accepting the operational burden of an always-on endpoint.
Start with the factory decision clock
Batch: choose it when the decision uses a complete time window and the action happens later. Examples include tomorrow’s maintenance worklist, a weekly quality review, or the next production plan. The useful measures are job completion time, records processed, data freshness and rerun success.
Micro-batch: choose it when decisions need fresh data every few minutes but do not require a response to every event. Examples include refreshing an anomaly queue or reprioritizing material moves. Measure end-to-end freshness, backlog age and replay reliability.
Online: choose it when each arriving event can change an immediate action and a defined fallback exists. Examples include routing a part to human review or adjusting an advisory setpoint. Measure p95 or p99 response latency, availability, timeouts and fallback use—not just average model execution time.
A worked maintenance example
Consider a fictional plant with 10,000 motors. The planning team closes the data window at 22:00 and needs a ranked maintenance list by 06:00. A nightly batch is the natural default: it can score all assets, preserve the input snapshot and rerun failed partitions before the morning meeting.
Suppose supervisors also want new anomalies every five minutes. That requirement supports a micro-batch queue, not necessarily one endpoint per sensor event. Only if a specific event must trigger an immediate advisory would online inference be justified. Even then, a safety PLC or interlock remains deterministic; a model timeout must never remove the safe state.
The 22:00, 06:00, five-minute and event-time values are synthetic teaching assumptions, not a performance benchmark or a recommendation for a particular plant.
What changes in the architecture
A batch design needs an input snapshot, job identifier, output table, completion check and rerun path. A micro-batch design adds watermarks, backlog monitoring and duplicate-safe replay. An online design adds endpoint availability, request IDs, timeouts, capacity limits and a tested fallback.
Dynamic batching can combine online requests to raise throughput, but it creates a queueing trade-off: waiting for a larger batch may reduce per-item cost while increasing latency. That is why the deadline must be written before tuning the service.
Do not optimize the wrong clock
A 50-millisecond model response does not create value if the supervisor reviews the queue the next morning. Conversely, a nightly job is too slow if the result must divert a part before the next station. Measure the full path from data availability to a verified action. That principle also applies to AI scheduling agents writing back to MES, the production data thread in an AI-server microfactory, and any attempt to measure whole-workflow AI productivity.
Limitations
Cloud documentation explains the mechanics of batch jobs and online endpoints; it does not establish a universal price, latency or reliability result. Network design, data volume, model size, plant controls and safety obligations can change the answer. Validate the chosen deadline and fallback with the process owner, IT/OT team and safety authority.
Three takeaways
- Choose the slowest serving mode that still meets the real decision deadline.
- Treat micro-batch as a deliberate operating pattern, not a failed version of real time.
- Define the fallback and the end-to-end measure before deploying an online endpoint.
Sources
- Vijay Janapa Reddi, Introduction to Machine Learning Systems, MIT Press, Volume I v0.7.2, 2026; chapter: Model Serving.
- Google Cloud, Get online inferences, accessed October 16, 2026.
- Google Cloud, Get batch inferences, accessed October 16, 2026.
- Vijay Janapa Reddi et al., MLPerf Inference Benchmark, submitted 2019, revised 2020.
Next action
For one AI workflow, write a single row with five fields: decision deadline, when the data is ready, action owner, failure fallback and primary operating measure. Then choose the slowest serving architecture that satisfies that row.
Leave a Reply