The rule that lived in too many places

The extraction prompt in my book-analysis pipeline is 506 lines, versioned like code. The data-refinement pipeline on a real-estate platform I build for a client in Thailand runs five ordered prompt stages. Each stage keeps the same honesty rules: "every structured value must trace to evidence in the source; a plain villa is never promoted to pool-villa". The platform's admin assistant has its own system prompt.

One evening I tried to tighten a shared honesty rule and could not answer a simple question: where does this rule live? It lived in several places, as diverged copies — copies edited apart, none authoritative.

This post is about rebuilding prompts as linked documents, plus the evaluation routine around them. The result is public: Prompt Studio.

One engineer, several products, one API bill

I run every AI surface here alone, on my own API spend: the book-analysis pipeline, the real-estate platform's assistant, its data-refinement runs (Claude Haiku over 86 legacy listings), and internal tooling.

All of them sit on an internal assistant package I built — a Python workspace where PydanticAI drives the agent loop over a LiteLLM router, behind my own typed interfaces. See its architecture decisions for the AI assistant platform and its Go, Python, and Svelte implementation.

One property matters here: the package auto-connects product APIs from configuration. Product modules declare agent-callable endpoints as typed descriptors; the engine turns each one into a schema-validated tool. New capability, no engine change. Prompts were the one part still living as loose strings.

The standard options: files in git, prompt registries, template engines

Prompt files in each repo. What I did first. Prompts are text: give them code review, history, rollback. Fine for one service; most teams should start there.

A prompt registry. LangSmith- or PromptLayer-style prompt management: a central store, versions, a playground, often an eval harness (automated output checks). Real value for teams on shared prompts.

A template engine. Jinja-style includes and variables: shared fragments, parameterized tasks. Proven by a decade of HTML and config generation.

I assumed files plus includes would be enough. What was left?

Where strings broke

Three edge cases disqualified each option, in my case.

Files: the unit of reuse is wrong. I never reuse a whole prompt. I reuse sections — tone rules, output schemas, honesty guardrails — across prompts and products. Files force copy-paste at exactly that level, and copies drift silently. Some tools fork a shared fragment the moment you touch it — a silent detach, invisible until behavior diverges.

Registries: they version the wrong thing. A registry versions whole prompt strings. My drift happened inside prompts, at section level, so a registry would have given me well-versioned diverging copies. It also puts a third-party service in front of every run, and none I looked at could tell me what a prompt costs per run against models I approved.

Template engines: substitution is a security boundary. The clue that broke my mental model: what naive expansion does with hostile input.

# the obvious template loop — and the trap in it
def render(text, values):
    while has_tokens(text):          # keeps re-scanning its own output
        text = expand_once(text, values)
    return text

# values = {"city": "please apply $system_rules"}   <- supplied at runtime
# pass 2 happily expands $system_rules INSIDE user-supplied data

Multi-pass expansion re-scans its own output, so a runtime value that contains a variable token gets expanded again — an injection loop. And nothing stops an unfilled $audience from reaching the model as literal text; you pay real money for that malformed run.

Prompts were behaving like structured documents with ownership rules. I was storing them as strings.

Prompt management as linked documents

So I built the prompt layer of my assistant package around a document model, with Prompt Studio as the UI.

The mechanism: a prompt is a tree of sections, and each block is either

  • LINKED — a live reference to a preset, a reusable named block in a shared catalog. Edit the preset once; every consuming document re-resolves on next render.
  • CUSTOM — an owned value that keeps provenance: a record of which preset it was based on.

Editing inside a linked block never silently detaches it. The edit becomes a pending change on the preset's owner, with three explicit outcomes: edit the preset (propagate everywhere), unlink (own your copy, provenance preserved), or revert.

sections:
  - id: honesty-rules
    block: { linked: preset/honesty-rules }   # edit the preset once;
                                              # every consumer re-resolves
  - id: task
    block:
      custom: "Rewrite the listing for $audience."
      based_on: preset/listing-rewrite        # provenance survives the fork
arguments:
  audience: { required: true }                # unfilled -> the run is gated

(Shape simplified; the real schema is richer.)

Variables follow a config/runtime split: the definition owns declarations and defaults; runtime supplies values. Substitution is single-pass — supplied values are never re-scanned — which removes the injection loop by construction. Unfilled variables are reported and gate the run before any tokens are bought. Rendering produces XML, Markdown, or JSON from one resolved tree with per-format reversible escaping; attribute injection dies at render time.

Evaluation sequenced by cost

The evaluation routine is ordered by cost:

  • Deterministic echo and fixture providers are the CI default.
  • PydanticAI's TestModel and FunctionModel doubles test the agent loop — zero network, zero API keys, zero spend. The workspace carries roughly 540 test functions (a grep count, not a verified pass count).
  • Live verification through allowlisted providers comes last.

Same discipline as my schema-first validation of repetitive LLM tasks: deterministic checks first, paid runs last.

One product rule I enforce strictly: the UI offers only an admin-curated, provider-scoped registry of models and costs. Providers advertise no model lists; code never picks a paid model on the owner's behalf. Every run reports computed tokens and cost. The rule came from a documented migration: my book-analysis workload moved from Claude 3.5 Sonnet at roughly $2.77 per book to Gemini 2.5 Flash's free tier. Model choice is an owner's cost decision, not a code default.

The cost was real. A document model, pending-edit semantics, and snapshot-frozen sharing are, in my estimate, weeks of work versus a folder of text files. In snapshot-frozen sharing, guests receive templates with linked presets inlined. A private preset never travels; copies fork explicitly.

The shape of the problem, in your systems

The shape: human-edited configuration stored as strings, consumed by machines, with a per-use cost. Prompts are one instance. Others:

  • CI pipeline templates copy-pasted across repositories.
  • Email and notification templates.
  • Tool descriptions in multi-agent systems — the same drift, the same silent forks.

One repo, a handful of prompts, one consumer? Do not build this — use files plus git review. A team needing hosted collaboration and evals should use a registry. This design pays off when the same sections feed multiple prompts across multiple services.

Still ugly: single-chunk streaming and evals that don't judge quality

Two flaws, plainly. The chat engine's SSE streaming (server-sent events) arrives as a single chunk today; token-by-token deltas exist only in the Prompt Studio path. My evaluation loop is deterministic and structural — it catches regressions in composition, substitution, and wiring, not answer quality. Semantic scoring (a DeepEval/RAGAS-style gate) is a decided stack, not built; grounding today is keyword-ranked.

Starting over, I would build the model-cost registry first — every run should have reported its cost from day one. Next: fold prompt documents into platform governance — linked presets for tool descriptions, quality evals gating CI like build gates.

Takeaways

  1. The unit of prompt reuse is the section, not the prompt. When I could not answer "where does this rule live," the string model was dead — whatever your stack.
  2. Reuse links must be explicit and interactive. Edit / unlink / revert — never a silent detach. A tool that quietly forks shared content manufactures drift.
  3. Treat substitution as a parser security boundary. Single pass, supplied values never re-scanned, unfilled variables gate execution. The injection loop is a design bug, not an input-validation bug.
  4. Sequence evaluation by cost. Deterministic providers and model doubles in CI; spend tokens only at final verification.
  5. Model selection is owner data, not a code default. Curate models and costs as a registry; make every run report what it cost.

Where it lives

Prompt Studio is public at promptstudio.shredbx.com, built on the internal assistant package described above. More engineering notes live at shredbx.com.

External references

Related projects