Latency & Cost Control for LLM Apps

What it is

Every LLM call costs money per token and takes hundreds of milliseconds to seconds. Latency and cost control is the set of decisions that keep both inside a budget: which model, how many tokens in and out, what to cache, what to stream, when to give up and what to do when the provider is slow or down.

The two are linked. Fewer tokens in the prompt means less cost and faster first token. A smaller model is cheaper and faster. Streaming does not reduce cost but makes the same latency feel much shorter. Caching removes both for repeated questions.

On this site, the choices were: a small model, five short chunks instead of the whole knowledge base, a bounded history, a fixed reply for the contact intent, lead extraction only after a cheap regex pre-filter and only in a background task, and a keyword fallback when OpenAI fails. The voice agent adds streaming at every stage and a hard session timer.

Why it's needed

Users leave slow products and businesses cancel expensive ones. A chat feature that costs more per conversation than the revenue it protects will be switched off. A voice agent that pauses three seconds before every reply will never be used twice.

Cost is also the question every client asks second, right after 'does it work'. Being able to say 'about this much per conversation, here is how I measured it and here is the lever if it needs to be lower' is what separates an engineer from a demo builder.

How it works

• Measure first: log tokens in, tokens out, latency to first token and total latency for every call, tagged by feature. You cannot optimise what you have not measured.
• Pick the smallest model that passes your evaluation set. Try the cheap model first and upgrade only where the eval shows it fails.
• Budget the prompt: cap retrieved context, trim history, keep the system prompt lean. Input tokens are usually the biggest cost in RAG.
• Cap the output with max_tokens and ask for short replies in the prompt. Output tokens cost more and take longer.
• Stream responses to the UI so the user reads while the model writes. For voice, stream text to speech sentence by sentence.
• Cache: identical questions can return a stored answer; provider-side prompt caching cuts the cost of a long, stable system prompt. Set timeouts and a fallback path so a slow provider degrades the experience instead of breaking it.

Code example

import time, hashlib, functools
from openai import OpenAI

client = OpenAI(timeout=15.0, max_retries=1)      # never hang a worker

@functools.lru_cache(maxsize=512)
def _cached(key: str, prompt: str) -> str:
    r = client.chat.completions.create(
        model="gpt-4o-mini", max_tokens=300,       # cap the expensive side
        messages=[{"role": "user", "content": prompt}])
    return r.choices[0].message.content

def answer(question: str, context: str, fallback) -> dict:
    prompt = f"Context:\n{context}\n\nAnswer briefly: {question}"
    key = hashlib.sha256(prompt.encode()).hexdigest()
    t0 = time.perf_counter()
    try:
        text, source = _cached(key, prompt), "llm"
    except Exception as e:                        # timeout, outage, quota
        text, source = fallback(question), f"fallback:{type(e).__name__}"
    ms = round((time.perf_counter() - t0) * 1000)
    print({"source": source, "latency_ms": ms, "prompt_chars": len(prompt)})
    return {"text": text, "source": source, "latency_ms": ms}

# Streaming for the UI: yield tokens as they arrive
def stream(prompt: str):
    with client.chat.completions.stream(model="gpt-4o-mini",
            messages=[{"role": "user", "content": prompt}]) as s:
        for event in s:
            if event.type == "content.delta":
                yield event.delta

Where I used this

Use cases

• Chat products with a per-conversation cost target
• Voice agents where every stage has a latency budget
• High-volume classification and extraction pipelines
• Public demos that must survive traffic spikes without a surprise bill
• Any feature that needs a graceful answer when the provider is down

Common mistakes

• Defaulting to the biggest model. Most product tasks pass on a small model; the eval set, not instinct, should decide.
• Stuffing the whole document into the prompt 'to be safe'. It multiplies cost and makes answers worse.
• No timeout, so one slow upstream call blocks a worker and the queue behind it.
• Measuring average latency only. Users feel the p95; report percentiles and time to first token.
• Caching without a key that includes the context. A cached answer from an old knowledge base is a stale answer.

Interview questions