# Spec Kit, BMAD and OpenSpec, and the Cost of Planning First
Published 2026-09-13. Updated 2026-09-17.
Canonical URL https://log.delaa.dev/blog/2026-09-13-spec-kit-bmad-openspec-compared
> Three spec-driven-development tools compared for real, with what a controlled benchmark says about their cost and what their own review mechanisms reveal.
---
The last five parts of this series were about a harness watching an agent
*act*. [Retry logic and a ten-persona
pipeline](/blog/2026-08-28-ai-harness-tmux-panes-to-ten-agents), [the allowlist that let a shell
operator through](/blog/2026-09-03-agent-breaks-rules-before-context-fills), [the rule an
agent read and broke anyway](/blog/2026-09-11-agent-took-forbidden-shortcut) once the
budget got tight. This part
is about a different kind of harness, one that runs before any code
exists at all, gating what gets built rather than how it gets built. Three
tools claim to do that job the same way. They don't. Reading all three end
to end turned up a controlled benchmark that says the disagreement has a
real cost, and a mechanism worth keeping regardless of which tool it came
from.
## What three tools claiming the same thing actually do differently
[GitHub's Spec Kit](https://github.blog/ai-and-ml/generative-ai/spec-driven-development-with-ai-get-started-with-a-new-open-source-toolkit/ "GitHub's own announcement, published 2 September 2025"),
[BMAD-METHOD](https://github.com/bmad-code-org/BMAD-METHOD "Agile AI-Driven Development, MIT licensed"),
and [OpenSpec](https://github.com/Fission-AI/OpenSpec "Spec-driven development for AI coding assistants, MIT licensed")
all sit in the same category and turn a request into a written spec before
an agent starts writing code. Installing and reading all three, not
skimming READMEs but the actual skill definitions each tool hands an
agent, turned up three different answers to the same design question of
how much should stand between a request and the first line of code.
Spec Kit runs a fixed sequence. Constitution, then specify, then plan,
then tasks, then implement. Each stage gates the next; a checklist has
to pass before the pipeline advances. BMAD-METHOD routes adaptively. Its
own entry-point workflow checks whether a change is small enough to skip
straight to implementation, and only falls back to full multi-stage
planning when the request doesn't fit that fast path. OpenSpec drops the
gate model entirely. Its own README calls this "fluid not rigid,"
and its implementation skill can run at any point, before or after other
artifacts exist, artifacts editable anytime.
Three teams, the same stated problem, three different answers
about how much ceremony a spec deserves before code gets written.
## The benchmark that should worry anyone about to adopt one of these
[Scott Logic ran a controlled comparison](https://blog.scottlogic.com/2025/11/26/putting-spec-kit-through-its-paces-radical-idea-or-reinvented-waterfall.html "Putting Spec Kit Through Its Paces: Radical Idea or Reinvented Waterfall?")
that's worth taking seriously precisely because it isn't a hit piece.
The same engineer rebuilt the same ~1,000-line feature (a deleted
circuit-management system for a go-kart tracking app) twice, once
through Spec Kit's full pipeline, once through plain iterative prompting
with immediate verification. The numbers.
| Metric | Spec Kit | Iterative prompting |
|---|---|---|
| Agent execution time | 57 minutes | 8 minutes |
| Code generated | ~989 lines | 1,000 lines |
| Markdown artifacts created | 4,839 lines | 0 |
| Review/testing time | 5.5 hours | 24 minutes |
| Bugs introduced | 1 | 0 |
Source: Scott Logic, "Putting Spec Kit Through Its Paces", November 2025.
The execution-time gap is 7x. The review-time gap is 13.75x, larger,
not smaller, which is the number that actually matters. A pipeline whose
whole design goal is producing enough upfront clarity to make
implementation and verification cheaper produced *more* markdown and
*more* review burden for the same feature, with a bug the iterative
approach didn't introduce. In the author's own words, "For now, the
fastest path is still iterative prompting and review, not industrialised
specification pipelines."
## Where the three tools actually earned their differences
Cost isn't the only axis these tools differ on. Each ships its own
mechanism for deciding whether a review finding is real, and reading
all three side by side showed a spread in rigor that has nothing to do
with gate model or ceremony level.
Spec Kit's review skill assigns severity by a short heuristic. A finding
is Critical if it violates a stated principle or leaves a requirement
with zero coverage, High if it's a duplicate or conflicting requirement,
and so on down the list. Nothing in that skill checks whether the
underlying claim is actually true before grading how bad it would be if
it were.
OpenSpec's verification skill states its own method plainly. Search the
codebase for keywords related to a requirement, then "assess if
implementation likely exists." Its own words include "don't require
perfect certainty." And its own stated bias under uncertainty is to
downgrade. Prefer a Suggestion over a Warning, a Warning over a Critical,
whenever unsure. That's a defensible default for a tool trying to avoid
false alarms. It's also the opposite of what you'd want from a step
whose entire job is catching real problems before they ship.
### The one that verifies before it grades
BMAD-METHOD's review protocol does neither. Before a finding gets a
severity, it has to be verified. Read the cited line, follow the
callers and the guards upstream, until the claimed bad outcome can be
confirmed or refuted. Its own stated principle is that code failing
loudly on a situation you never showed it could reach is correct
behavior, not a defect. Verified findings get one of five verdicts, not
three. `false` (checked, doesn't happen, state what disproves it) and
`maybe-false` (couldn't settle it, state what would) sit alongside
`high`/`medium`/`low`, so an unverifiable claim doesn't get silently
folded into "probably fine" the way OpenSpec's downgrade bias would
treat it. Verified findings that survive route to exactly one of three
outcomes. Patch it now if the fix is trivial and adds nothing new, halt
and ask a human if it isn't, or log it for later if it's a pre-existing
issue rather than something this change caused.
## Taking the mechanism, not the tool
None of that made a case for installing BMAD-METHOD. Its own review
protocol lives inside a workflow that needs two separate runtimes
(Python for its rendering step, Node for its CLI wrapper), plus five
agent personas and a dozen skills that have nothing to do with reviewing
findings. Installing the whole framework to keep one page of
classification logic would mean carrying a build pipeline, a second
language runtime, and a foreign project structure into every repo that
wanted the one thing actually worth having.
So we didn't install it. We read the mechanism, verified it against the
source, and rewrote it as a standalone skill. Verify a claim before
classifying it, keep the five verdicts including the two that record
uncertainty honestly instead of hiding it, route every survivor to
patch, halt, or defer instead of leaving a flat severity list someone
has to re-triage by hand. A second skill does the same extraction from
OpenSpec, taking not its artifact graph or its CLI but the one
discipline worth keeping. When two claims about the same code disagree,
don't trust whichever one is closer at hand. Search the actual code for
evidence before deciding which claim is true.
That's the tradeoff, and it isn't one to resolve too cleanly.
What got left behind is real. BMAD's escalation ramp for surprises
mid-implementation, OpenSpec's dependency-graph resolution across a
whole planning artifact set. The bet is that the two mechanisms we kept
were the load-bearing part, and everything around them was the
particular way each tool chose to package it.
## What this changes about the wiki-and-mechanism thesis from parts 2-5
Every mechanism this series has described so far ran at execution time.
`.git/info/exclude` keeping AI context out of a project's own repo (see
[part 2](/blog/2026-08-19-built-harness-before-it-had-name)), the allowlist fix that
rejects a shell operator before a model ever gets to decide whether to
use one, the gate that only lets a persona advance once its artifact is
actually committed. This audit found the same underlying
principle holding one layer earlier, in the planning stage, in three
tools built by three separate teams who arrived at three different
answers about how much of it to apply.
It's worth naming the boundary rather than overselling the parallel.
Scott Logic's own numbers, and the same complaint independently voiced
in [Spec Kit's own issue tracker](https://github.com/github/spec-kit/issues/75 "\"SpecKit creates the illusion of work, generating a bunch of text\"")
(*"it's more a tool for a product owner than for a developer"*), both
point at something the execution-side mechanisms in this series don't
share. A planning pipeline's cost is measured in hours of human review,
paid every time it runs. The allowlist fix runs in milliseconds and
never asks anyone to read 4,839 lines of markdown to find out if it
worked. A mechanism that's obviously worth it at execution time, where
the alternative is a security hole or a silently dropped requirement,
isn't automatically worth the same ceremony at planning time, where the
alternative might just be a shorter conversation with the agent.
What survived the audit either way is two rules. Verify a claim before
deciding how bad it is, and when two claims disagree, go read the code.
Those two rules don't care whether they're running before the first line
gets written or after the last one ships.
---
[The first article in this series](/blog/2026-07-28-what-is-harness-engineering)
defined the discipline this one keeps returning to. The model provides
reasoning, the harness provides everything that makes the reasoning
trustworthy enough to act on. That's true one layer up from where this
series started looking.
---
— Delaa