Model serving patterns

The big idea

A trained model is a file. Serving is everything needed to turn that file into answers at the moment the product needs them. The first question is not "which framework?" but "when is the answer needed?". If it can be computed overnight, compute it overnight and look it up. If it depends on what the user is doing right now, compute it on request, within a few tens of milliseconds. If it must react to events as they happen, attach it to the event stream.

Think of a bakery again. Bread for the morning is baked overnight in big batches (batch serving). A custom cake is made to order while the customer waits (online serving). And the oven alarm that fires the moment the temperature spikes reacts to events, not orders (streaming). Same kitchen, three rhythms, and a sensible bakery does not bake every loaf to order.

How it works

Step 1: Three serving patterns

Batch serving precomputes predictions on a schedule: tomorrow's demand per store, a weekly churn score per customer, "recommended for you" lists refreshed nightly. It runs on cheap batch compute, is easy to debug (the output is a table you can inspect) and puts no model on the request path. Its weakness is staleness: a user who signed up an hour ago has no row, and a score from last night ignores what happened this morning.

Online serving puts the model behind an API. The caller sends a request over REST or gRPCTwo common ways to call a service over the network: REST sends JSON over HTTP; gRPC sends compact binary messages over HTTP/2 using a schema, which is usually faster and is typed. with the request's context (user, query, location), the service fetches stored features from the online store, runs the model and returns a score. Use it when the input only exists at request time: a search query, the items in a basket, a ride request.

Streaming serving attaches the model to an event stream ( Stream processing ). Each payment, click or sensor reading is scored as it flows past, and the result becomes another event: an alert, a block, an updated feature. It suits monitoring and fraud signals, where someone must react within seconds but nobody is waiting on an HTTP response.

Many systems mix them. A recommender might precompute candidate lists in batch, re-rank them online with the user's live context, and update features from a stream.

Step 2: The latency budget

An online model never owns the whole response time. If a product page must render in 200 ms, the ranking call might get 50 ms, and those 50 ms are split again. As an illustration: about 5 ms of network, 10 ms to fetch features, 25 ms of model inference and 10 ms of margin for slow requests. That budget is set at a high percentile such as p99, not the average, because a page that calls several services waits for the slowest one ( Latency vs throughput, and why percentiles matter shows how fan-out multiplies the tail).

The budget drives design choices: a smaller or distilled model (compression is covered later in this module, Model compression ), fewer features with expensive lookups, precomputing parts of the work in batch, and caching.

Step 3: Model servers and dynamic batching

You can wrap a model in a small web app, but dedicated model serversSoftware that loads trained models and exposes them as network endpoints, handling versions, batching, hardware and metrics, such as TensorFlow Serving or NVIDIA Triton. handle the repeated work: loading model versions, exposing endpoints, batching and metrics. TensorFlow Serving offers gRPC and REST APIs, and its REST predict URL can name a version, as in /v1/models/eta/versions/2:predict; leave the version out and it serves the latest. NVIDIA Triton serves models from several frameworks.

Hardware such as a GPU is much more efficient on a batch than on one example, because most of the cost of a call is fixed overhead. Dynamic batchingA model-server feature that holds incoming requests for a short time and runs them together as one batch, trading a little latency for much higher throughput. exploits this: the server waits a few milliseconds to collect requests that arrive close together and runs them as one batch. Triton's documentation puts the trade-off plainly: batching typically increases throughput, and a larger batch size or a nonzero queue delay trades increased latency for increased throughput.

Step 4: Caching predictions

If the same input arrives often, store the answer. A prediction cache keyed by the input (the item, the user, the normalized query) skips the model entirely on a hit. Popularity is usually skewed, so a small cache covers a large share of traffic. Two rules: give entries a time to live that matches how fast the inputs change, and include the model version in the key, so a new model does not serve its predecessor's cached answers.

Step 5: Rolling out a new model safely

Offline metrics do not prove that a model works in production: the feature pipeline may differ ( Feature stores and training-serving skew ), latency may blow the budget, or the model may misbehave on traffic the test set never covered. Two standard steps reduce the risk.

• A shadow deploymentRunning a new model on a copy of live traffic without using its answers, to compare its predictions, latency and errors with the current model at no risk to users. sends a copy of live requests to the new model and throws its answers away after logging them. You learn its latency under real load, how often it disagrees with the champion and whether it crashes on real inputs, with zero user impact.
• A canary deploymentReleasing a new version to a small share of real traffic first, comparing its metrics with the current version, and widening the share only if it holds up. gives the new model to a small share of users, watches errors, latency and business metrics, and widens the share in steps or rolls back. It is the same technique used for any service release ( Deployment strategies ); for models, follow it with a proper A/B test ( Offline vs online evaluation and A/B tests ) when the business impact matters.

A model registry with versions makes both cheap: the router points at "v7" and rollback is pointing back at "v6".

Going deeper

Code: batching and caching, measured

import numpy as np
rng = np.random.default_rng(6)
COST = lambda b: 4.0 + 0.25 * b            # ms to score a batch of b on one GPU: fixed overhead + per item

def serve(rate_per_s, max_batch, max_wait_ms, n=20000):
    """One model server, requests arriving at random. Returns p50 and p99 latency in ms."""
    arrive = np.cumsum(rng.exponential(1000 / rate_per_s, n))
    done, i, free_at = np.empty(n), 0, 0.0
    while i < n:
        start = max(free_at, arrive[i])                      # the oldest waiting request
        start = max(start, min(arrive[i] + max_wait_ms,      # wait up to max_wait_ms for the batch to fill
                               arrive[min(i + max_batch - 1, n - 1)]))
        j = min(i + max_batch, np.searchsorted(arrive, start, side="right"))
        free_at = start + COST(j - i); done[i:j] = free_at; i = j
    lat = done - arrive
    return np.percentile(lat, 50), np.percentile(lat, 99)

print("requests/s  batching            p50 ms   p99 ms")
for rate in [100, 200, 800, 2000]:
    for name, mb, wait in [("one at a time", 1, 0), ("dynamic, 2 ms wait", 64, 2)]:
        p50, p99 = serve(rate, mb, wait)
        print(f"{rate:10d}  {name:18s} " + (f"{p50:7.1f}  {p99:7.1f}" if p99 < 1000 else "overloaded: the queue grows without limit"))

# Prediction cache: 1M requests over 100,000 items with Zipf-like popularity; cache the top items.
items = 100_000
pop = 1 / np.arange(1, items + 1) ** 1.0; pop /= pop.sum()
req = rng.choice(items, 1_000_000, p=pop)
for size in [100, 1000, 10000]:
    print(f"cache of the {size:6d} most popular items: hit rate {np.isin(req, np.arange(size)).mean():.1%}")

requests/s  batching            p50 ms   p99 ms
       100  one at a time          4.2     15.4
       100  dynamic, 2 ms wait     6.2      8.6
       200  one at a time         11.2     49.7
       200  dynamic, 2 ms wait     6.3      9.0
       800  one at a time      overloaded: the queue grows without limit
       800  dynamic, 2 ms wait     7.7     11.2
      2000  one at a time      overloaded: the queue grows without limit
      2000  dynamic, 2 ms wait    12.3     19.4
cache of the    100 most popular items: hit rate 42.9%
cache of the   1000 most popular items: hit rate 61.9%
cache of the  10000 most popular items: hit rate 81.0%

The model costs 4 ms of fixed overhead plus 0.25 ms per example in a batch, typical of a GPU, where the fixed part dominates. Served one request at a time, each call takes 4.25 ms, so one server can handle at most 1000 / 4.25, about 235 requests per second.

• At 100 requests per second, one-at-a-time is fastest at the median (4.2 ms against 6.2 ms with batching, which spends up to 2 ms waiting for company), but batching already has the better p99: 8.6 ms against 15.4 ms.
• At 200 per second the single-request server is close to its limit, and queueing pushes its p99 to 49.7 ms. Batching stays at 9.0 ms.
• At 800 and 2,000 per second the single-request server is overloaded and its queue grows without limit. Dynamic batching serves 2,000 per second at a p99 of 19.4 ms, because bigger batches spread the fixed overhead.

The cache experiment sends a million requests over 100,000 items with Zipf-like popularity (the k-th most popular item is requested in proportion to 1/k). Caching just the 100 most popular items answers 42.9 percent of requests without touching the model; 1,000 items answer 61.9 percent, and 10,000 answer 81.0 percent.

Trade-offs

Common mistakes

• Building an online service for predictions that could be computed nightly and looked up.
• Setting the latency budget on the average instead of p99, then missing it on every busy page.
• Caching predictions without the model version in the key, so a rollout appears to change nothing.
• Shipping a new model straight to 100 percent of traffic without shadow or canary steps.
• Treating a canary as an A/B test: a few minutes of error rates catch breakage, not a small drop in conversion.

In the wild

Check yourself

Recap