The prompt as an input-output contract
A prompt is the most concrete artifact in a generative AI application: it is the only input the model sees at inference time, and it must encode both the task and the expected shape of the response. When the downstream pipeline needs a structured payload (JSON for a tool call, a fixed enum for a classification label, a Markdown table for a UI widget), the prompt design decides whether the model emits a parseable string on the first try or whether the chain has to retry, repair, and re-validate. Treating the prompt as a contract changes the design questions: instead of asking "how do I phrase this nicely", you ask "what is the input schema, what is the output schema, what fields are mandatory, what defaults apply on missing fields, and how does the contract fail closed when the input is adversarial".
The contract has three layers. The outer layer is the role layout: a system message that describes the agent's identity and global constraints, a user message that carries the request, and optional assistant turns that demonstrate the desired behavior. The middle layer is the task statement: a single declarative sentence describing what to produce, including any constraints on length, language, tone, or factuality. The inner layer is the output schema: an explicit description of the response format, the field names, the allowed values, and the failure mode when no answer applies. Most production failures live in the inner layer because designers write a clear task statement but stop short of pinning the output format.
💡 Exam Trap: Questions will present a prompt that says "respond in JSON" and ask why parsing fails. The answer is almost always that the prompt did not specify the exact schema, did not forbid prose around the JSON, or did not show an example of the empty-result case. Pinning the format means listing keys, types, and a worked example.
Structured output techniques on Foundation Model APIs
Databricks Foundation Model APIs and Mosaic AI Model Serving expose three mechanisms for getting structured output from a chat-completions endpoint. The first is response format constraints: an OpenAI-compatible response_format={"type": "json_object"} parameter that forces the model to emit a single JSON object. The second is tool calling: the chat schema accepts a tools array, and when the model selects a tool, the API returns a tool_calls payload with a strict JSON arguments field whose schema you supplied. The third is a guided prompt with few-shot examples: the prompt itself shows the structure, the model is told to mirror it, and a downstream parser validates. The three techniques are not exclusive; production prompts often combine all three.
Tool calling is the strongest contract because the schema is validated against a JSON Schema you provided, and Foundation Model APIs reject malformed tool calls before they reach your code. Response format is weaker because the model is only obligated to return valid JSON, not to match your fields. Few-shot prompting is the weakest because the model can drift from the shown structure on edge inputs, but it is the only option when the model does not support the other two mechanisms or when the schema is genuinely dynamic.
from databricks.sdk import WorkspaceClient
from databricks.sdk.service.serving import ChatMessage, ChatMessageRole
w = WorkspaceClient()
system_prompt = """You are a claims triage assistant for Northwind Insurance.
For every claim, return a single JSON object with these fields:
- severity: one of "low", "medium", "high"
- category: one of "auto", "home", "health", "other"
- rationale: a single sentence, at most 25 words
- escalate: boolean, true only when severity == "high"
If the input is not a claim, return {"severity":"low","category":"other","rationale":"not a claim","escalate":false}.
Return only the JSON object. Do not include backticks, prose, or markdown.
"""
user_prompt = "My car hit a deer on Route 9, the airbag deployed, I am at the hospital."
response = w.serving_endpoints.query(
name="databricks-meta-llama-3-3-70b-instruct",
messages=[
ChatMessage(role=ChatMessageRole.SYSTEM, content=system_prompt),
ChatMessage(role=ChatMessageRole.USER, content=user_prompt),
],
temperature=0.0,
max_tokens=200,
)
print(response.choices[0].message.content)
The prompt above pins five things: the field names, the field domains, a length cap on free text, a guard for off-task inputs, and a no-prose constraint. Temperature is set to zero because the response shape is deterministic; for free text generation, this would change.
Format anchors: schema, examples, and guardrails
A schema is the explicit list of fields, types, and allowed values. Examples are demonstrations of valid outputs for representative inputs. Guardrails are the negative space: instructions that say what the model must not do. A well-formed format-eliciting prompt contains all three. The schema anchors the parser, the examples anchor the model's expectation, and the guardrails close the easy escape routes (prose around the JSON, code fences, apology preambles, hedging suffixes).
📊 Limit: Tool-call argument schemas on Foundation Model APIs follow standard JSON Schema. Nested depth, array sizes, and total schema size count against the model's input token budget, so a 30 KB schema can crowd out the user's question on a small-context model.
The example block is the single highest-yield investment in a format-eliciting prompt. One worked example that exactly matches your schema (including a representative non-answer case) reduces format-violation rates more than any other change. Two examples are better; three start to plateau and may bias the model toward the example content. Always include an "empty result" example so the model knows what to emit when the request is unanswerable, otherwise it will invent.
Examples:
Input: "I tripped on a sidewalk crack and bruised my knee."
Output: {"severity":"low","category":"other","rationale":"minor injury, no property damage","escalate":false}
Input: "Garage fire spread to the kitchen, total loss, family safe."
Output: {"severity":"high","category":"home","rationale":"total loss, escalate to senior adjuster","escalate":true}
Input: "What is the weather in Boston today?"
Output: {"severity":"low","category":"other","rationale":"not a claim","escalate":false}
Common format failures and how to defeat them
Three failure patterns recur in production. The first is the "polite preamble": the model writes "Sure, here is the JSON:" before the object. Defeat it by saying "Return only the JSON object. No prose, no greetings, no closing remarks." The second is the "markdown fence wrap": the model wraps the JSON in triple backticks. Defeat it by saying "Do not include backticks or code fences." The third is the "field invention": the model adds fields you did not ask for, or renames yours. Defeat it by saying "Use exactly these field names. Do not add or rename fields." When the model still misbehaves, switch from response-format prompting to tool calling, which validates the schema at the API boundary.
⚠️ Anti-Pattern: Writing "respond in JSON format" with no schema and no example, then relying on a downstream
json.loadsto fail loudly. Every retry is a paid token and a latency budget hit. The schema and example belong in the prompt the first time, not in a repair loop.
🎯 Scenario: A retail company asks for a prompt that classifies customer reviews into "praise", "complaint", or "question" and extracts the product name. The team writes "Classify the review and pull the product name". Parsing fails on 18% of inputs because the model sometimes returns prose, sometimes JSON, sometimes both. The fix is a system prompt with an explicit JSON schema (
{"label": ..., "product": ...}), three labeled examples (one per class), an empty-result example for off-topic text, andresponse_format={"type": "json_object"}. Failure rate drops below 1% without changing the model.
Temperature, decoding, and determinism for formatted output
When the output is structured, temperature should be zero or near zero. Higher temperature samples lower-probability tokens, which is exactly what you do not want when the next character must be { or "severity". Top-p and top-k follow the same logic: tight sampling for structured output, broader sampling for creative free text. Repetition penalties can interact badly with structured output because they push the model away from repeated quote marks and braces, so leave them at default for JSON tasks. Max tokens should be set tight enough that runaway generation cannot push your parser into an out-of-memory error, but loose enough that a valid response fits.
Decision Anchor
Choose tool calling when the output is a function invocation, the schema is fixed, and downstream code calls a specific function with the arguments. Choose response_format={"type": "json_object"} plus schema-in-prompt when the output is a single data record and you need a strict JSON guarantee. Choose few-shot prompting with a free-text parser when the schema is dynamic, multi-document, or the model lacks structured-output support. When the same prompt is going to run at high volume, prefer tool calling because the API-side validation removes a class of repair retries.
💡 Exam Trap: A common distractor pairs "use temperature=0.7" with structured output. Higher temperature is the wrong answer for JSON, enum classification, or any other task with a single correct shape. Temperature controls randomness over the next-token distribution, and structured tasks have low-entropy next-token distributions by design.
Prompt versioning and the design-evaluation loop
A prompt is a piece of source code. It has versions, owners, regression tests, and a deployment pipeline. The Databricks platform integrates prompt registration with MLflow's Prompt Registry, where each prompt has a name, a numbered version, and a Unity Catalog scope. Treat the first format-eliciting prompt as version 1, run it against a held-out set of inputs, score the parse rate and content correctness, and only promote when both metrics meet target. The same MLflow trace that captures the LLM call captures the prompt version, so when production output drifts you can correlate it to a prompt change without guesswork. This loop is the bridge from prompt design (this chapter) to evaluation and monitoring (Chapter 6 of the syllabus).