Conversation Memory (Multi-Turn Chat)

What it is

LLM APIs are stateless. Every call starts from nothing. The model only 'remembers' the previous turns because your code sends them again in the messages list. Conversation memory is the set of techniques for deciding what to resend, how much, and in what form.

The simplest version is to replay the full history: system prompt, then every user and assistant message so far, then the new question. That works until the history gets long, expensive or full of noise.

In a RAG chatbot there is a second memory problem. The retrieval step sees only the latest question. 'And before that?' embeds to nothing useful, so the follow-up has to be rewritten into a standalone question before it is used for retrieval.

Why it's needed

Without memory, every follow-up fails. 'Where did he work in 2023?' followed by 'what did he build there?' needs the model to know what 'there' means. Users expect a chat to behave like a conversation, and a chatbot that forgets the previous line feels broken.

Memory also has a cost side. Each replayed token is billed on every call, and context windows are finite. A ten-minute conversation with full history can cost more per message at the end than at the start, so memory management is both a quality and a budget decision.

How it works

• Keep the history on the client or in a session store and send it with each request as a list of role/content messages.
• Replay the system prompt first, then the history in order, then the new user message. Order matters.
• Window the history: keep only the last N turns, typically 6 to 10, so cost and context stay bounded.
• Summarise older turns: when the window overflows, ask the model to compress the dropped turns into a short summary that stays in the system prompt.
• For RAG, rewrite follow-up questions into standalone ones ('what did he build there?' becomes 'what did Abdur build at Aircod Technologies?') before embedding them for retrieval.
• Store long-term facts about the user separately (name, preferences) and inject them, rather than relying on them surviving in the history.

Code example

from openai import OpenAI

client = OpenAI()
SYSTEM = "You are Abdur's portfolio assistant. Answer from the context."
MAX_TURNS = 8  # user+assistant pairs to keep verbatim

def standalone_question(history: list[dict], question: str) -> str:
    """Rewrite a follow-up into a self-contained query for retrieval."""
    if not history:
        return question
    convo = "\n".join(f"{m['role']}: {m['content']}" for m in history[-4:])
    r = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content":
                   f"Conversation:\n{convo}\n\nRewrite this follow-up as a "
                   f"standalone question. Reply with the question only.\n{question}"}],
    )
    return r.choices[0].message.content.strip()

def chat(history: list[dict], question: str, retrieve) -> str:
    query = standalone_question(history, question)
    context = "\n\n".join(retrieve(query))          # RAG step uses the rewrite
    messages = [{"role": "system", "content": f"{SYSTEM}\n\nContext:\n{context}"}]
    messages += history[-2 * MAX_TURNS:]           # sliding window
    messages.append({"role": "user", "content": question})
    r = client.chat.completions.create(model="gpt-4o-mini", messages=messages)
    return r.choices[0].message.content

Where I used this

Use cases

• Any chat interface with follow-up questions
• Support bots that need to remember the ticket details across turns
• Voice agents, where every turn is short and context carries the meaning
• Agent loops where tool results from earlier steps inform later ones

Common mistakes

• Retrieving with the raw follow-up text. 'What about Docker?' embeds to a vague vector and retrieval misses.
• Unbounded history: the conversation works for five turns and then starts timing out or hitting the context limit.
• Putting the retrieved context into the history. Old context accumulates across turns and the prompt fills with stale chunks. Inject fresh context per turn.
• Trusting the client-supplied history blindly. Cap its length and size on the server; a malicious client can send a megabyte.
• Summarising too aggressively and losing the one detail the user refers back to.

Interview questions