The test factory
How a Bandwise test is made, and why you can trust its answer key
A practice book holds a handful of tests. Bandwise writes new ones with a language model, which is only useful if the questions are fair and the answers are right. So every test is planned by code, written to a published standard, and then attacked: by validators, by a second model sitting it blind, and by an adjudicator. This page shows each step, what it catches, what it costs, and where the approach still falls short.
- tests in the library
- 21
- passed every structural gate just now
- 21 of 21
- marked questions, each with evidence and an explanation
- 1,611
- of rendered Listening audio
- 501 min
Nine stages
From a seed number to a test you can sit
Stages with a solid number run in this prototype. Stages with a dashed outline are designed and documented, but still done by hand or waiting for real candidates.
Stage 1: Blueprint
CodePlans the testBefore any model writes a word, a seeded sampler plans the whole test: module, difficulty, a topic for every part drawn from 126 subjects and 88 everyday situations, the question types, the numbering from 1 to 40, and a cast of speakers with different accents.
What it prevents
- Sameness: a model asked to “write a varied test” keeps returning to the same few topics and formats
- Illegal papers: parts that do not add up to 40 questions, or question types that never appear in that part
- A topic turning up twice in one test
In the code: src/lib/generation/blueprint.ts
Stage 2: Author
Language modelWrites the draftA language model writes one unit at a time to a written item-writing standard: the text first, then the questions. Every question must carry the exact words of the text that prove its answer, and an explanation that names the trap.
What the standard demands
- Original texts only: nothing copied or recalled from real exam papers
- Invented names, dates and figures, so that questions cannot be answered from general knowledge
- Answers in the order of the text, distractors built into the script, paraphrased stems
In the code: src/lib/generation/prompts.ts · docs/design/item-writing-spec.md
Stage 3: Structural gates
CodeBlocks publicationOrdinary code, with no judgement involved, checks the draft. Anything it finds goes back to the author with the exact findings, at most twice. A draft that still fails is discarded.
What it catches
- An evidence quote that is not word for word in the text, or an answer that never appears in it
- A key that breaks its own word limit, or a gap with no matching question
- Listening answers heard out of order on a recording that plays once
- Wrong numbering, a missing group, or a question type the blueprint did not ask for
- Warnings for review: seven words lifted from the text, a correct option much longer than the rest, a lopsided TRUE / FALSE / NOT GIVEN set
In the code: src/lib/content/validate.ts · src/lib/generation/checks.ts
Stage 4: Blind solver
Language modelBlocks publicationA second pass, blind: a solver that never sees the answer key sits the unit exactly as you would, because the same code that hides answers from your browser hides them from the solver. Its answers are marked by the same marker that scores your tests. The solver can be a different model from the author (one setting); in this prototype it is the same one.
What it catches
- A wrong answer key
- Two options that can both be defended
- A NOT GIVEN statement that can in fact be worked out from the text
- A gap-fill whose key leaves out a perfectly good answer
In the code: src/lib/generation/pipeline.ts · src/lib/content/candidate.ts · src/lib/scoring/objective.ts
Stage 5: No-text solver
Language modelFlags for reviewAnother model is given the questions with the passage or recording withheld, and told to guess. Questions it gets right more often than luck allows do not test reading or listening at all.
What it catches
- Questions answerable from general knowledge
- Options that give themselves away by their wording
- Gaps that can be filled by grammar alone
In the code: src/lib/generation/pipeline.ts (flagAnswerableWithoutText)
Stage 6: Adjudication
Language modelBlocks publicationEvery disagreement between solver and key is ruled on. Either the key stands, with the decisive words of the text quoted, or the question is repaired with the smallest possible change. A repaired unit goes back through the structural gates and a fresh blind solve; anything still disputed is held for a person.
What comes out
- A verdict and a reason for every disputed question, stored in the test’s QA report
- Only units with no unresolved dispute are assembled into a test
In the code: src/lib/generation/pipeline.ts · content/drafts/<id>/qa-report.json
Stage 7: Audio
Code + speech engineProduces audioListening scripts are turned into speech line by line, with a separate voice for each speaker chosen by the accent and gender in the blueprint. Silences for reading time are inserted and each part is mixed into one track. Once audio exists, the transcript is withheld from your browser until you submit.
What it guarantees
- You hear the recording; you never read it during the test
- Each speaker keeps one distinct voice throughout a part
In the code: scripts/render-audio.ts
Stage 8: Human spot review
PeopleBlocks publicationPlanned, not automated yetA person reads a sample of every batch, and every question that an earlier gate flagged. In this prototype that means reading the QA report by hand; a review screen is part of the production plan.
What only people catch
- Dialogue that is correct but does not sound like people talking
- Cultural missteps and topics that could distress a candidate
- A plausible “fact” that two models agreed on and is still wrong
In the code: docs/design/generation-pipeline.md
Stage 9: Live calibration
Live dataAfter publicationPlanned, not automated yetOnce enough people have sat a test, their answers show how hard each question really is. Questions almost everyone or almost no one gets right are retired, as are questions that strong candidates miss more often than weak ones. The raw-score-to-band table is then adjusted for that test.
What it will catch
- Questions that are harder or easier than they look
- Questions that reward the wrong candidates
- A test whose 30 out of 40 is not worth the same as another’s
In the code: docs/design/generation-pipeline.md
The bank today
Every published test, re-checked as this page loaded
These figures are not stored anywhere. The server read each test, counted its questions and words, and ran the structural gates again a moment ago.
- Marked questions
- 80
- Question formats
- 12
- Reading words
- 2,627
- Script words
- 3,428
- Audio
- 26 min 30 s (4/4 parts)
- Review warnings
- 5
Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch
- Marked questions
- 80
- Question formats
- 11
- Reading words
- 2,619
- Script words
- 3,317
- Audio
- 24 min 28 s (4/4 parts)
- Review warnings
- 1
Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation
- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,603
- Script words
- 3,549
- Audio
- 26 min 37 s (4/4 parts)
- Review warnings
- 3
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 80
- Question formats
- 10
- Reading words
- 2,645
- Script words
- 3,395
- Audio
- 24 min 53 s (4/4 parts)
- Review warnings
- 3
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 80
- Question formats
- 10
- Reading words
- 2,575
- Script words
- 3,259
- Audio
- 25 min 23 s (4/4 parts)
- Review warnings
- 0
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 80
- Question formats
- 10
- Reading words
- 2,665
- Script words
- 3,291
- Audio
- 25 min 29 s (4/4 parts)
- Review warnings
- 0
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,569
- Script words
- 3,272
- Audio
- 26 min 15 s (4/4 parts)
- Review warnings
- 0
Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch
- Marked questions
- 80
- Question formats
- 10
- Reading words
- 2,630
- Script words
- 3,161
- Audio
- 23 min 29 s (4/4 parts)
- Review warnings
- 0
Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation
- Marked questions
- 80
- Question formats
- 10
- Reading words
- 2,614
- Script words
- 3,225
- Audio
- 24 min 21 s (4/4 parts)
- Review warnings
- 1
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,639
- Script words
- 3,188
- Audio
- 23 min 37 s (4/4 parts)
- Review warnings
- 0
Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation
- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,609
- Script words
- 3,231
- Audio
- 24 min 46 s (4/4 parts)
- Review warnings
- 0
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,557
- Script words
- 3,217
- Audio
- 25 min 08 s (4/4 parts)
- Review warnings
- 0
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 80
- Question formats
- 11
- Reading words
- 2,834
- Script words
- 3,124
- Audio
- 24 min 06 s (4/4 parts)
- Review warnings
- 0
Written by claude-fable-5-1 · pipeline 0.1.0-seed · difficulty standard
- Marked questions
- 80
- Question formats
- 11
- Reading words
- 2,848
- Script words
- 3,124
- Audio
- 23 min 58 s (4/4 parts)
- Review warnings
- 0
Written by claude-fable-5-1 · pipeline 0.1.0-seed · difficulty stretch
General Training Practice Test 1
General TrainingStructural gates: 0 errorsBlind-solved and adjudicated- Marked questions
- 80
- Question formats
- 12
- Reading words
- 2,418
- Script words
- 3,120
- Audio
- 24 min 03 s (4/4 parts)
- Review warnings
- 0
Written by claude-fable-5-1 · pipeline 0.1.0-seed · difficulty standard
General Training Practice Test 000201
General TrainingStructural gates: 0 errorsBlind-solved and adjudicated- Marked questions
- 80
- Question formats
- 10
- Reading words
- 2,279
- Script words
- 3,268
- Audio
- 25 min 01 s (4/4 parts)
- Review warnings
- 1
Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation
General Training Practice Test 000202
General TrainingStructural gates: 0 errorsBlind-solved and adjudicated- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,420
- Script words
- 3,441
- Audio
- 24 min 48 s (4/4 parts)
- Review warnings
- 3
Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation
General Training Practice Test 000203
General TrainingStructural gates: 0 errorsBlind-solved and adjudicated- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,694
- Script words
- 3,524
- Audio
- 26 min 24 s (4/4 parts)
- Review warnings
- 3
Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch
General Training Practice Test 000204
General TrainingStructural gates: 0 errorsBlind-solved and adjudicated- Marked questions
- 80
- Question formats
- 9
- Reading words
- 2,514
- Script words
- 3,307
- Audio
- 25 min 16 s (4/4 parts)
- Review warnings
- 2
Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch
General Training Practice Test 000205
General TrainingStructural gates: 0 errorsBlind-solved and adjudicated- Marked questions
- 80
- Question formats
- 8
- Reading words
- 2,551
- Script words
- 3,341
- Audio
- 24 min 54 s (4/4 parts)
- Review warnings
- 3
Written by claude-code-cli · pipeline 0.2.0 · difficulty standard
- Marked questions
- 11
- Question formats
- 4
- Reading words
- 320
- Script words
- 287
- Audio
- 2 min 00 s (1/1 parts)
- Review warnings
- 2
Written by hand-authored · pipeline 0.1.0-seed · difficulty foundation
Try it
Blueprint explorer
Type a seed, or shuffle. Your browser runs the same sampler the server uses and plans a complete test in a few milliseconds: topics, question types, numbering and voices. No model is involved at this stage, which is why the plan is free, instant and repeatable.
Any whole number. The same seed always plans the same test, here and on the server.
Showing the plan for Academic Practice Test 000042, Academic, standard difficulty.
Academic Practice Test 000042
AcademicDifficulty: standardid ac-000042Listening
Part 1
a new member joining a public library and reserving items
the caller changes one detail partway through (a date, a time or a quantity)
- 1–10Form completion (ONE WORD AND/OR A NUMBER)
Voices: Tamsin Rasmussen (librarian; female, australian) · Liam Abernethy (new member; male, british)
Show the brief the author receivesHide the brief
UNIT l1 — Listening Part 1 (Academic test). Fragment id "l1", part 1, Questions 1–10. What is heard: a new member joining a public library and reserving items. Build the distractors around this: the caller changes one detail partway through (a date, a time or a quantity). Speakers — use exactly these ids, names, genders and accents (plus the narrator): - id "tamsin": Tamsin Rasmussen, librarian — female, australian accent - id "liam": Liam Abernethy, new member — male, british accent Question groups, in this order and with exactly these numbers: - Questions 1–10 — Form completion (group id "l1-g1"; type "completion", variant "form"): wordLimit {"words":1,"number":true} — the instructions must say "ONE WORD AND/OR A NUMBER" Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them. Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.Part 2
a tour introduction at a botanical garden
rules, safety and what to bring
- 11–16Multiple choice
- 17–20Multiple choice (choose TWO)
Voices: Sofia Whitcombe (head gardener; female, british)
Show the brief the author receivesHide the brief
UNIT l2 — Listening Part 2 (Academic test). Fragment id "l2", part 2, Questions 11–20. What is heard: a tour introduction at a botanical garden. Angle: rules, safety and what to bring. Speakers — use exactly these ids, names, genders and accents (plus the narrator): - id "sofia": Sofia Whitcombe, head gardener — female, british accent Question groups, in this order and with exactly these numbers: - Questions 11–16 — Multiple choice (group id "l2-g1"; type "multiple-choice"): 6 single-answer questions (select: 1) - Questions 17–20 — Multiple choice (choose TWO) (group id "l2-g2"; type "multiple-choice"): 2 "Choose TWO" question(s) with select: 2 and five options A–E, numbered 17 and 19; each occupies two question numbers Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them. Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
Part 3
a student discussing a work-placement report with a tutor, on machine translation
- 21–24Matching features
- 25–30Multiple choice
Voices: Tomas Kowalczyk (student; male, british) · Imogen Eastwick (tutor; female, south african)
Show the brief the author receivesHide the brief
UNIT l3 — Listening Part 3 (Academic test). Fragment id "l3", part 3, Questions 21–30. What is heard: a student discussing a work-placement report with a tutor, on machine translation. Speakers — use exactly these ids, names, genders and accents (plus the narrator): - id "tomas": Tomas Kowalczyk, student — male, british accent - id "imogen": Imogen Eastwick, tutor — female, south-african accent Question groups, in this order and with exactly these numbers: - Questions 21–24 — Matching features (group id "l3-g1"; type "matching", variant "features") - Questions 25–30 — Multiple choice (group id "l3-g2"; type "multiple-choice"): 6 single-answer questions (select: 1) Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them. Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
Part 4
a lecture on detecting forgeries in the art market
an overview lecture tracing how understanding of the subject has changed
- 31–37Notes completion (NO MORE THAN TWO WORDS)
- 38–40Sentence completion (NO MORE THAN TWO WORDS)
Voices: Grace Iverson (lecturer; female, british)
Show the brief the author receivesHide the brief
UNIT l4 — Listening Part 4 (Academic test). Fragment id "l4", part 4, Questions 31–40. What is heard: a lecture on detecting forgeries in the art market. Angle: an overview lecture tracing how understanding of the subject has changed. Speakers — use exactly these ids, names, genders and accents (plus the narrator): - id "grace": Grace Iverson, lecturer — female, british accent Question groups, in this order and with exactly these numbers: - Questions 31–37 — Notes completion (group id "l4-g1"; type "completion", variant "notes"): wordLimit {"words":2,"number":false} — the instructions must say "NO MORE THAN TWO WORDS" - Questions 38–40 — Sentence completion (group id "l4-g2"; type "completion", variant "sentence"): wordLimit {"words":2,"number":false} — the instructions must say "NO MORE THAN TWO WORDS" Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them. Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
Reading
Text 1
the history of the university lecture
descriptive: its origins and historical development, period by period
- 1–7TRUE / FALSE / NOT GIVEN
- 8–13Table completion (ONE WORD ONLY)
Show the brief the author receivesHide the brief
UNIT r1 — Academic Reading, Passage 1. Fragment id "r1", section 1, Questions 1–13. Subject: the history of the university lecture. Treatment (descriptive): its origins and historical development, period by period. Length: 700–850 words. Question groups, in this order and with exactly these numbers: - Questions 1–7 — TRUE / FALSE / NOT GIVEN (group id "r1-g1"; type "judgement", variant "true-false-not-given") - Questions 8–13 — Table completion (group id "r1-g2"; type "completion", variant "table"): wordLimit {"words":1,"number":false} — the instructions must say "ONE WORD ONLY" Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.Text 2
procrastination
discursive: differing expert forecasts and what each assumes
- 14–20Matching headings
- 21–26Summary completion (NO MORE THAN TWO WORDS)
Show the brief the author receivesHide the brief
UNIT r2 — Academic Reading, Passage 2. Fragment id "r2", section 2, Questions 14–26. Subject: procrastination. Treatment (discursive): differing expert forecasts and what each assumes. Length: 750–900 words. Question groups, in this order and with exactly these numbers: - Questions 14–20 — Matching headings (group id "r2-g1"; type "matching", variant "headings"): 10 headings with roman-numeral ids - Questions 21–26 — Summary completion (group id "r2-g2"; type "completion", variant "summary"): wordLimit {"words":2,"number":false} — the instructions must say "NO MORE THAN TWO WORDS" Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.Text 3
amateur astronomy clubs
argumentative: the writer calls for a change of policy and answers the objections
- 27–30Multiple choice
- 31–35YES / NO / NOT GIVEN
- 36–40Matching sentence endings
Show the brief the author receivesHide the brief
UNIT r3 — Academic Reading, Passage 3. Fragment id "r3", section 3, Questions 27–40. Subject: amateur astronomy clubs. Treatment (argumentative): the writer calls for a change of policy and answers the objections. Length: 800–950 words. Question groups, in this order and with exactly these numbers: - Questions 27–30 — Multiple choice (group id "r3-g1"; type "multiple-choice"): 4 single-answer questions (select: 1) - Questions 31–35 — YES / NO / NOT GIVEN (group id "r3-g2"; type "judgement", variant "yes-no-not-given") - Questions 36–40 — Matching sentence endings (group id "r3-g3"; type "matching", variant "sentence-endings") Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
Writing
Tasks 1 and 2
two pie charts for two years or two groups (one "pie" visual with two series, each summing to 100): tea and coffee consumption in four countries
Essay (opinion): young children cared for by grandparents or by nurseries
Data story: the rank order at the end is the reverse of the start.
Show the brief the author receivesHide the brief
UNIT writing — Academic Writing. Task 1 (Academic): two pie charts for two years or two groups (one "pie" visual with two series, each summing to 100), showing tea and coffee consumption in four countries. Visual kinds, in order: pie. The invented data must tell this story: the rank order at the end is the reverse of the start. Task 2: essayType "opinion", on young children cared for by grandparents or by nurseries (Family and society). Question frame: To what extent do you agree or disagree? Timing, fixed by the exam: minutes 60; Task 1 minWords 150 and suggestedMinutes 20; Task 2 minWords 250 and suggestedMinutes 40. Difficulty: Standard — prompts as abstract and as worded as those of a typical live paper.
Speaking
Parts 1–3
Describe a person who taught you something useful
Part 1: the home you live in, giving gifts, weekends. Part 3: role models, influence and relationships between generations.
Show the brief the author receivesHide the brief
UNIT speaking — Speaking (identical for both modules). Part 1 topics, in this order, four questions each: the home you live in; giving gifts; weekends. Part 2 cue card (A person): "Describe a person who taught you something useful". Part 3 theme: role models, influence and relationships between generations. Five or six questions, moving from concrete comparison to abstract speculation. Timing, fixed by the exam: minutes 14; Part 2 prepSeconds 60 and speakSeconds 120. Difficulty: Standard — prompts as abstract and as worded as those of a typical live paper.
Scale and cost
Thousands of tests is arithmetic, not a slogan
Two questions decide whether a test factory scales: how many genuinely different tests can be planned, and what each one costs to write and check.
How many different tests can be planned?
at least 1036
distinct Academic blueprints. This is a lower bound: it multiplies only the choices of topic, treatment and question mix below, and ignores names, accents, word limits, question-count splits and difficulty.
| Choice | Options |
|---|---|
| Listening Part 1: situation × twist | 384 |
| Listening Part 2: talk × focus | 204 |
| Five long-text subjects, each from a different domain | 14.2 billion |
| Discussion format, lecture angle and three passage treatments | 24,576 |
| Question-type mixes across the seven objective units | 483,840 |
| Writing Task 1: visual × data story × subject, or a process | 1,938 |
| Writing Task 2: essay type × issue | 240 |
| Speaking: opener × two Part 1 topics × cue card | 227,040 |
The honest reading of that number
A different blueprint is not automatically a test that feels different. What a keen learner notices is a subject coming round again. The tightest pool today is “long-text subjects (five used per test)”: after about 25 tests a subject must reappear, though in a different role (a lecture rather than a passage) and with a different treatment. The pools are plain data — 126 subjects in 14 domains, 88 listening situations, 50 short-text kinds, 190 writing prompts, 80 cue cards — so widening them is a one-line change, not an engineering project.What does a test cost to make?
One full test is about 33 model calls: nine units written, seven of them sat blind, probed without the text and, where the solver disagrees, adjudicated. Prices are per million tokens in and out; the Batch API halves them for work that can wait up to a day.
Claude Opus 5
$5 in / $25 out per million tokens
- Per test, live
- $3.36
- Per test, batch
- $1.90
- 1,000 tests, live
- $3,356
- 1,000 tests, batch
- $1,904
Claude Sonnet 5
$2 in / $10 out per million tokens
- Per test, live
- $1.53
- Per test, batch
- $0.95
- 1,000 tests, live
- $1,527
- 1,000 tests, batch
- $946
Claude Haiku 4.5
$1 in / $5 out per million tokens
- Per test, live
- $0.92
- Per test, batch
- $0.63
- 1,000 tests, live
- $917
- 1,000 tests, batch
- $627
Every figure includes $0.31 per test for neural text-to-speech (19,200 characters at $0.016 per thousand), which the batch discount does not touch. The prototype’s own tests were voiced with ElevenLabs (the turbo model, half a credit per character: about 9,600 credits a test), and every spoken line is cached so a script edit re-renders only the lines that changed.
Show the stage-by-stage breakdown and assumptionsHide the breakdown
| Stage | Calls | Tokens in | Tokens out | Cost |
|---|---|---|---|---|
| Author | 9.0 | 49,500 | 42,395 | $1.13 |
| Repair rounds | 4.5 | 38,847 | 19,437 | $0.59 |
| Blind solver | 7.0 | 14,905 | 14,960 | $0.45 |
| No-text probe | 7.0 | 6,680 | 4,460 | $0.14 |
| Adjudicator | 3.5 | 26,723 | 20,248 | $0.58 |
| Confirming solve | 2.5 | 5,217 | 5,236 | $0.16 |
- Fractional calls are averages: 0.6 repair rounds per marked unit, and an adjudication for 50% of them. These are planning assumptions, to be replaced by measured token usage from real batches.
- The item-writing standard is a single unchanging system prompt of about 5,050 tokens, so it is cached: a cache read costs a tenth of the normal input price. Inside a batch, cache hits are best-effort, so the batch columns assume 60% rather than 90%.
- Thinking tokens are billed as output and are included. Human review time is not.
Limits
What this approach cannot promise yet
Generated tests are practice material. Here is what stands between this prototype and material you could calibrate an exam on.
A language model’s question is a draft until another model has failed to break it
Models write fluent questions with confident answer keys, and some of those keys are wrong or arguable. That is why the pipeline publishes nothing on the author’s word: a generated draft has to survive the structural gates and a blind solve before it earns the validated badge. The solver can be a different model from the author (one setting), but in this prototype it is the same one, and two passes by one model can share a blind spot — which is what the human spot review is for. The four seed tests in the library were written by hand and pass the structural gates, but no blind solver has sat them yet, so none of them carries the badge.
Difficulty is a target, not a measurement
“Foundation”, “standard” and “stretch” are instructions to the author. How hard a question really is can only be measured from the answers of real candidates. Until a test has been sat a couple of hundred times, its band estimate rests on the published raw-score tables for real papers, and a generated test may run a little easier or harder than those.
Band scores are estimates
Listening and Reading bands come from raw-score tables; Writing and Speaking bands come from a model examiner working to the public band descriptors (or, offline, a clearly labelled heuristic), always shown with a likely range. None of them is an official result, and Bandwise is independent of the organisations that own IELTS.
The prototype’s voices are neural text-to-speech, cast by accent and gender
Each speaker gets one ElevenLabs voice, chosen by the accent and gender in the blueprint from the voices on the account; when the account has no voice for an accent the nearest one stands in, and every substitution is written to the audio manifest — today’s library sounds more British than its blueprints asked for. The voices are clear, but they do not overlap or interrupt the way people do. The free macOS voices remain the offline fallback, and the costs above price the same service at list rates.
Human review and live calibration are designed, not yet running
The last two stages of the pipeline need reviewers and real attempt data. In this prototype, review means a person reading the QA report, and no question has yet been retired on evidence from candidates.
Follow the calibration study