System Prompt Design for Products

What it is

The system prompt is the standing instruction set your application sends with every request. It defines who the model is, what it knows, what format it must reply in, and what it must never do. In a product it is code, not prose: it is versioned, tested and reviewed like any other code path, because a one-word change can alter every response.

A good production prompt has a predictable structure. Role and scope first. Then the injected context, such as retrieved chunks. Then the output contract, ideally a JSON schema. Then behavioural rules, from most to least important. Then edge-case handling and style rules. The order matters because models weight the start and the end of the prompt most.

The prompt behind this site's chat follows that structure. It names the assistant, injects the retrieved knowledge-base excerpts, demands JSON with an intent field and an answer, then lists rules for each intent, including a fixed sentence for contact requests, a ban on invented employers or dates, and a style rule about punctuation.

Why it's needed

The prompt is the cheapest lever in the whole system. Changing a model costs money and migration; changing chunking means re-indexing; changing a rule in the prompt takes thirty seconds and affects every answer immediately. That power cuts both ways, which is why prompts need the same discipline as code.

It is also where product behaviour lives. The decision that a 'contact' question renders a card instead of a paragraph, that answers stay under three sentences unless the question is technical, that the bot redirects off-topic questions politely: all of that is prompt design, and it is what the user actually experiences.

How it works

• Role and scope: one or two sentences stating who the model is and what it is for. 'You are Abdur's portfolio assistant. Answer questions about his career from the excerpts below.'
• Context slot: a clearly delimited block for retrieved text, with a label so the model knows it is reference material, not instructions.
• Output contract: the exact format, as a schema. If the frontend reads a field, define it here and enforce it with structured output.
• Rules in priority order, phrased as positive instructions where possible. 'Keep replies under three sentences' beats 'do not be verbose'.
• Edge cases with explicit handling: what to say when the answer is not in the context, how to treat off-topic questions, what to do with contact requests.
• Keep it under control: every rule should exist because a test case failed without it. Delete rules that no longer earn their place.

Code example

SYSTEM_PROMPT = """You are Abdur Rehman Afzal's AI Portfolio Assistant. Answer
questions about Abdur using only the Knowledge Base excerpts below, and
decide which UI component the frontend should render.

Knowledge Base (most relevant excerpts for this question):
======================
{knowledge_base}
======================

Respond with valid JSON matching exactly:
{{"intent": "me" | "projects" | "resume" | "skills" | "contact" | "general",
  "ai_text": "your conversational reply"}}

Rules:
- "me" for introductions. The frontend shows a profile card, so do not
  repeat his bio; write one friendly line that invites a question.
- "projects" only when the user asks to see or list projects. Follow-up
  questions about one project are "general".
- "contact" when the user wants to reach him. Set ai_text to EXACTLY:
  "You can reach Abdur through the contact info above. What's on your mind?"
- Keep ai_text under 3 sentences unless the question is technical; then
  use 1-2 short paragraphs with bullets where helpful.
- Never invent employers, dates, links or certifications not in the
  excerpts. If unsure, say so and point to a real contact channel.
- Never use em dashes. Use commas, colons or periods.
"""

def build_messages(chunks: list[str], history: list[dict], question: str):
    system = SYSTEM_PROMPT.format(knowledge_base="\n\n".join(chunks))
    return ([{"role": "system", "content": system}]
            + history[-16:]
            + [{"role": "user", "content": question}])

Where I used this

Use cases

• Chat assistants whose replies drive UI components
• Support bots with strict scope and escalation rules
• Extraction and classification tasks with fixed output schemas
• Voice agents where reply length and tone must be tightly controlled
• Any LLM feature that must behave the same way for every user

Common mistakes

• Rules written as a wish list with no tests behind them. If you cannot show the case a rule fixes, you cannot tell when it stops working.
• Burying the output format in the middle. Put the contract near the top and enforce it with JSON mode or a schema, not with hope.
• Negative-only instructions. Models follow 'do X' more reliably than 'never do Y'; where you need a ban, pair it with what to do instead.
• Letting retrieved context and instructions blur together. Delimit the context block clearly or a document can hijack the prompt.
• Editing the prompt in production without a changelog. Version it, diff it and run the evaluation set before deploying.

Interview questions