AI engineering

Why "Prompt Engineering" Isn't Engineering — Yet

The word "engineering" implies measurement, repeatability, and the ability to hand a design off to someone else. Most "prompt engineering" today involves none of those. It's craft — sometimes very good craft — but it isn't engineering yet. Here's what the upgrade looks like.

Search for "prompt engineering" and you get millions of results teaching prompt tricks: chain-of-thought, few-shot, role assignment, "let's think step by step". Useful. But those are techniques, not a discipline.

Engineering has a specific shape. It's the practice of designing systems that can be measured, versioned, reviewed, evolved and handed over. By that definition, what most teams do with prompts isn't engineering — it's still artisanal work that lives in a Slack thread or an unversioned Google Doc.

Here's what turning it into actual engineering looks like.

1. Prompts are code

Every substantive prompt in your product should live in your codebase, under version control, with the same review process as any other production asset. Not in a database. Not in a config panel someone can hot-edit. Not in a comment. In code.

The rule we apply: any prompt change should be traceable to a pull request, reviewed by another engineer, and tested before it hits users. If you can't tell which prompt version was live yesterday afternoon, you don't have engineering. You have hope.

2. Prompts have evals, or they don't ship

Every prompt that reaches production should have an evaluation set attached to it — a curated collection of inputs and expected properties of the output. Not a huge one; 30–100 examples per prompt is usually enough to detect regressions.

The eval runs automatically before merge. If quality drops, the PR fails. Prompts that change without an eval are the leading source of AI-feature regressions we see in the wild.

3. Prompts have versions, and observability tells you which is live

Every LLM call in production logs which prompt version it used. Two implications:

• When something goes wrong in production, you can point to exactly the prompt that produced the bad output.
• You can A/B test prompt changes — half the traffic on v3.2, half on v3.3, compare quality metrics before rolling forward.

Neither is possible if your prompt is a hard-coded string that a well-meaning engineer tweaked at 4pm on Friday.

4. Prompts are structured, not blobs

Real prompt engineering rarely means one big string. It means composing prompts from parts:

• A static role/system preamble (stable, prompt-cacheable).
• A slow-changing context layer (per tenant, per user).
• A fast-changing content layer (per request).
• Instructions and output-format directives.
• Optional examples (few-shot).

Each part has its own owner, its own tests, and its own change velocity. You can update few-shot examples without touching the system role. You can add a new tenant without editing the base prompt. This is where prompt design starts to feel like software architecture rather than string manipulation.

5. Prompts are decoupled from models

Every prompt in production should carry metadata about which model it was designed for, and there should be at least one alternative model wired up as a fallback. Two reasons:

• Model providers change pricing, capacity and safety filters at short notice. Being locked to one is now a business risk.
• The right model for a task changes over time. New model releases can turn today's expensive prompt into tomorrow's cheap one.

Practically: your prompt registry stores per-model variants, and a switching layer in front of the LLM call selects at runtime.

6. Prompt quality is a metric, not a vibe

You'd never ship a search feature and let engineers ask "does it feel like the ranking is good?" You'd measure click-through, dwell time, satisfaction. Same principle for prompts:

• Automated quality metrics on every request (or a sampled subset).
• User-facing feedback (thumbs, correctness rating, edit-vs-accept).
• Regular human evaluation on a fresh sample.

The dashboard tells you what "the AI is a bit rubbish this week" means in numbers.

The organisational upgrade

None of this is a solo effort. It requires:

• Engineering: prompt registry, versioning, observability.
• Data: eval sets, quality metrics, dashboards.
• Product: what "good" actually means for each feature.
• Ops: on-call runbook for AI incidents (unexpected refusals, hallucination spikes, provider outages).

Teams that make this leap stop having "the AI is a bit rubbish this month" conversations. They start having "quality on feature X dropped 3% on Tuesday when we changed the system prompt; here's the fix" conversations. That's the difference between craft and engineering.