Skip to content
Bandwise

The test factory

How a Bandwise test is made, and why you can trust its answer key

A practice book holds a handful of tests. Bandwise writes new ones with a language model, which is only useful if the questions are fair and the answers are right. So every test is planned by code, written to a published standard, and then attacked: by validators, by a second model sitting it blind, and by an adjudicator. This page shows each step, what it catches, what it costs, and where the approach still falls short.

tests in the library
21
passed every structural gate just now
21 of 21
marked questions, each with evidence and an explanation
1,611
of rendered Listening audio
501 min

Nine stages

From a seed number to a test you can sit

Stages with a solid number run in this prototype. Stages with a dashed outline are designed and documented, but still done by hand or waiting for real candidates.

  1. Stage 1: Blueprint

    CodePlans the test

    Before any model writes a word, a seeded sampler plans the whole test: module, difficulty, a topic for every part drawn from 126 subjects and 88 everyday situations, the question types, the numbering from 1 to 40, and a cast of speakers with different accents.

    What it prevents

    • Sameness: a model asked to “write a varied test” keeps returning to the same few topics and formats
    • Illegal papers: parts that do not add up to 40 questions, or question types that never appear in that part
    • A topic turning up twice in one test

    In the code: src/lib/generation/blueprint.ts

  2. Stage 2: Author

    Language modelWrites the draft

    A language model writes one unit at a time to a written item-writing standard: the text first, then the questions. Every question must carry the exact words of the text that prove its answer, and an explanation that names the trap.

    What the standard demands

    • Original texts only: nothing copied or recalled from real exam papers
    • Invented names, dates and figures, so that questions cannot be answered from general knowledge
    • Answers in the order of the text, distractors built into the script, paraphrased stems

    In the code: src/lib/generation/prompts.ts · docs/design/item-writing-spec.md

  3. Stage 3: Structural gates

    CodeBlocks publication

    Ordinary code, with no judgement involved, checks the draft. Anything it finds goes back to the author with the exact findings, at most twice. A draft that still fails is discarded.

    What it catches

    • An evidence quote that is not word for word in the text, or an answer that never appears in it
    • A key that breaks its own word limit, or a gap with no matching question
    • Listening answers heard out of order on a recording that plays once
    • Wrong numbering, a missing group, or a question type the blueprint did not ask for
    • Warnings for review: seven words lifted from the text, a correct option much longer than the rest, a lopsided TRUE / FALSE / NOT GIVEN set

    In the code: src/lib/content/validate.ts · src/lib/generation/checks.ts

  4. Stage 4: Blind solver

    Language modelBlocks publication

    A second pass, blind: a solver that never sees the answer key sits the unit exactly as you would, because the same code that hides answers from your browser hides them from the solver. Its answers are marked by the same marker that scores your tests. The solver can be a different model from the author (one setting); in this prototype it is the same one.

    What it catches

    • A wrong answer key
    • Two options that can both be defended
    • A NOT GIVEN statement that can in fact be worked out from the text
    • A gap-fill whose key leaves out a perfectly good answer

    In the code: src/lib/generation/pipeline.ts · src/lib/content/candidate.ts · src/lib/scoring/objective.ts

  5. Stage 5: No-text solver

    Language modelFlags for review

    Another model is given the questions with the passage or recording withheld, and told to guess. Questions it gets right more often than luck allows do not test reading or listening at all.

    What it catches

    • Questions answerable from general knowledge
    • Options that give themselves away by their wording
    • Gaps that can be filled by grammar alone

    In the code: src/lib/generation/pipeline.ts (flagAnswerableWithoutText)

  6. Stage 6: Adjudication

    Language modelBlocks publication

    Every disagreement between solver and key is ruled on. Either the key stands, with the decisive words of the text quoted, or the question is repaired with the smallest possible change. A repaired unit goes back through the structural gates and a fresh blind solve; anything still disputed is held for a person.

    What comes out

    • A verdict and a reason for every disputed question, stored in the test’s QA report
    • Only units with no unresolved dispute are assembled into a test

    In the code: src/lib/generation/pipeline.ts · content/drafts/<id>/qa-report.json

  7. Stage 7: Audio

    Code + speech engineProduces audio

    Listening scripts are turned into speech line by line, with a separate voice for each speaker chosen by the accent and gender in the blueprint. Silences for reading time are inserted and each part is mixed into one track. Once audio exists, the transcript is withheld from your browser until you submit.

    What it guarantees

    • You hear the recording; you never read it during the test
    • Each speaker keeps one distinct voice throughout a part

    In the code: scripts/render-audio.ts

  8. Stage 8: Human spot review

    PeopleBlocks publicationPlanned, not automated yet

    A person reads a sample of every batch, and every question that an earlier gate flagged. In this prototype that means reading the QA report by hand; a review screen is part of the production plan.

    What only people catch

    • Dialogue that is correct but does not sound like people talking
    • Cultural missteps and topics that could distress a candidate
    • A plausible “fact” that two models agreed on and is still wrong

    In the code: docs/design/generation-pipeline.md

  9. Stage 9: Live calibration

    Live dataAfter publicationPlanned, not automated yet

    Once enough people have sat a test, their answers show how hard each question really is. Questions almost everyone or almost no one gets right are retired, as are questions that strong candidates miss more often than weak ones. The raw-score-to-band table is then adjusted for that test.

    What it will catch

    • Questions that are harder or easier than they look
    • Questions that reward the wrong candidates
    • A test whose 30 out of 40 is not worth the same as another’s

    In the code: docs/design/generation-pipeline.md

The bank today

Every published test, re-checked as this page loaded

These figures are not stored anywhere. The server read each test, counted its questions and words, and ran the structural gates again a moment ago.

  • Academic Practice Test 000100

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    12
    Reading words
    2,627
    Script words
    3,428
    Audio
    26 min 30 s (4/4 parts)
    Review warnings
    5

    Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch

  • Academic Practice Test 000101

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    11
    Reading words
    2,619
    Script words
    3,317
    Audio
    24 min 28 s (4/4 parts)
    Review warnings
    1

    Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation

  • Academic Practice Test 000102

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,603
    Script words
    3,549
    Audio
    26 min 37 s (4/4 parts)
    Review warnings
    3

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Academic Practice Test 000103

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    10
    Reading words
    2,645
    Script words
    3,395
    Audio
    24 min 53 s (4/4 parts)
    Review warnings
    3

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Academic Practice Test 000104

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    10
    Reading words
    2,575
    Script words
    3,259
    Audio
    25 min 23 s (4/4 parts)
    Review warnings
    0

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Academic Practice Test 000105

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    10
    Reading words
    2,665
    Script words
    3,291
    Audio
    25 min 29 s (4/4 parts)
    Review warnings
    0

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Academic Practice Test 000106

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,569
    Script words
    3,272
    Audio
    26 min 15 s (4/4 parts)
    Review warnings
    0

    Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch

  • Academic Practice Test 000107

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    10
    Reading words
    2,630
    Script words
    3,161
    Audio
    23 min 29 s (4/4 parts)
    Review warnings
    0

    Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation

  • Academic Practice Test 000108

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    10
    Reading words
    2,614
    Script words
    3,225
    Audio
    24 min 21 s (4/4 parts)
    Review warnings
    1

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Academic Practice Test 000109

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,639
    Script words
    3,188
    Audio
    23 min 37 s (4/4 parts)
    Review warnings
    0

    Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation

  • Academic Practice Test 000110

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,609
    Script words
    3,231
    Audio
    24 min 46 s (4/4 parts)
    Review warnings
    0

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Academic Practice Test 000111

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,557
    Script words
    3,217
    Audio
    25 min 08 s (4/4 parts)
    Review warnings
    0

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Academic Practice Test 1

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    11
    Reading words
    2,834
    Script words
    3,124
    Audio
    24 min 06 s (4/4 parts)
    Review warnings
    0

    Written by claude-fable-5-1 · pipeline 0.1.0-seed · difficulty standard

  • Academic Practice Test 2

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    11
    Reading words
    2,848
    Script words
    3,124
    Audio
    23 min 58 s (4/4 parts)
    Review warnings
    0

    Written by claude-fable-5-1 · pipeline 0.1.0-seed · difficulty stretch

  • General Training Practice Test 1

    General TrainingStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    12
    Reading words
    2,418
    Script words
    3,120
    Audio
    24 min 03 s (4/4 parts)
    Review warnings
    0

    Written by claude-fable-5-1 · pipeline 0.1.0-seed · difficulty standard

  • General Training Practice Test 000201

    General TrainingStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    10
    Reading words
    2,279
    Script words
    3,268
    Audio
    25 min 01 s (4/4 parts)
    Review warnings
    1

    Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation

  • General Training Practice Test 000202

    General TrainingStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,420
    Script words
    3,441
    Audio
    24 min 48 s (4/4 parts)
    Review warnings
    3

    Written by claude-code-cli · pipeline 0.2.0 · difficulty foundation

  • General Training Practice Test 000203

    General TrainingStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,694
    Script words
    3,524
    Audio
    26 min 24 s (4/4 parts)
    Review warnings
    3

    Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch

  • General Training Practice Test 000204

    General TrainingStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    9
    Reading words
    2,514
    Script words
    3,307
    Audio
    25 min 16 s (4/4 parts)
    Review warnings
    2

    Written by claude-code-cli · pipeline 0.2.0 · difficulty stretch

  • General Training Practice Test 000205

    General TrainingStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    80
    Question formats
    8
    Reading words
    2,551
    Script words
    3,341
    Audio
    24 min 54 s (4/4 parts)
    Review warnings
    3

    Written by claude-code-cli · pipeline 0.2.0 · difficulty standard

  • Quick Diagnostic

    AcademicStructural gates: 0 errorsBlind-solved and adjudicated
    Marked questions
    11
    Question formats
    4
    Reading words
    320
    Script words
    287
    Audio
    2 min 00 s (1/1 parts)
    Review warnings
    2

    Written by hand-authored · pipeline 0.1.0-seed · difficulty foundation

Try it

Blueprint explorer

Type a seed, or shuffle. Your browser runs the same sampler the server uses and plans a complete test in a few milliseconds: topics, question types, numbering and voices. No model is involved at this stage, which is why the plan is free, instant and repeatable.

Any whole number. The same seed always plans the same test, here and on the server.

Showing the plan for Academic Practice Test 000042, Academic, standard difficulty.

Academic Practice Test 000042

AcademicDifficulty: standardid ac-000042

Listening

  • Part 1

    a new member joining a public library and reserving items

    the caller changes one detail partway through (a date, a time or a quantity)

    • 110Form completion (ONE WORD AND/OR A NUMBER)

    Voices: Tamsin Rasmussen (librarian; female, australian) · Liam Abernethy (new member; male, british)

    Show the brief the author receives
    UNIT l1 — Listening Part 1 (Academic test). Fragment id "l1", part 1, Questions 1–10.
    What is heard: a new member joining a public library and reserving items.
    Build the distractors around this: the caller changes one detail partway through (a date, a time or a quantity).
    Speakers — use exactly these ids, names, genders and accents (plus the narrator):
    - id "tamsin": Tamsin Rasmussen, librarian — female, australian accent
    - id "liam": Liam Abernethy, new member — male, british accent
    Question groups, in this order and with exactly these numbers:
    - Questions 1–10 — Form completion (group id "l1-g1"; type "completion", variant "form"): wordLimit {"words":1,"number":true} — the instructions must say "ONE WORD AND/OR A NUMBER"
    Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them.
    Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
  • Part 2

    a tour introduction at a botanical garden

    rules, safety and what to bring

    • 1116Multiple choice
    • 1720Multiple choice (choose TWO)

    Voices: Sofia Whitcombe (head gardener; female, british)

    Show the brief the author receives
    UNIT l2 — Listening Part 2 (Academic test). Fragment id "l2", part 2, Questions 11–20.
    What is heard: a tour introduction at a botanical garden.
    Angle: rules, safety and what to bring.
    Speakers — use exactly these ids, names, genders and accents (plus the narrator):
    - id "sofia": Sofia Whitcombe, head gardener — female, british accent
    Question groups, in this order and with exactly these numbers:
    - Questions 11–16 — Multiple choice (group id "l2-g1"; type "multiple-choice"): 6 single-answer questions (select: 1)
    - Questions 17–20 — Multiple choice (choose TWO) (group id "l2-g2"; type "multiple-choice"): 2 "Choose TWO" question(s) with select: 2 and five options A–E, numbered 17 and 19; each occupies two question numbers
    Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them.
    Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
  • Part 3

    a student discussing a work-placement report with a tutor, on machine translation

    • 2124Matching features
    • 2530Multiple choice

    Voices: Tomas Kowalczyk (student; male, british) · Imogen Eastwick (tutor; female, south african)

    Show the brief the author receives
    UNIT l3 — Listening Part 3 (Academic test). Fragment id "l3", part 3, Questions 21–30.
    What is heard: a student discussing a work-placement report with a tutor, on machine translation.
    Speakers — use exactly these ids, names, genders and accents (plus the narrator):
    - id "tomas": Tomas Kowalczyk, student — male, british accent
    - id "imogen": Imogen Eastwick, tutor — female, south-african accent
    Question groups, in this order and with exactly these numbers:
    - Questions 21–24 — Matching features (group id "l3-g1"; type "matching", variant "features")
    - Questions 25–30 — Multiple choice (group id "l3-g2"; type "multiple-choice"): 6 single-answer questions (select: 1)
    Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them.
    Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
  • Part 4

    a lecture on detecting forgeries in the art market

    an overview lecture tracing how understanding of the subject has changed

    • 3137Notes completion (NO MORE THAN TWO WORDS)
    • 3840Sentence completion (NO MORE THAN TWO WORDS)

    Voices: Grace Iverson (lecturer; female, british)

    Show the brief the author receives
    UNIT l4 — Listening Part 4 (Academic test). Fragment id "l4", part 4, Questions 31–40.
    What is heard: a lecture on detecting forgeries in the art market.
    Angle: an overview lecture tracing how understanding of the subject has changed.
    Speakers — use exactly these ids, names, genders and accents (plus the narrator):
    - id "grace": Grace Iverson, lecturer — female, british accent
    Question groups, in this order and with exactly these numbers:
    - Questions 31–37 — Notes completion (group id "l4-g1"; type "completion", variant "notes"): wordLimit {"words":2,"number":false} — the instructions must say "NO MORE THAN TWO WORDS"
    - Questions 38–40 — Sentence completion (group id "l4-g2"; type "completion", variant "sentence"): wordLimit {"words":2,"number":false} — the instructions must say "NO MORE THAN TWO WORDS"
    Script: 650–950 words including narrator lines; answers are heard in question order with two or three sentences between them.
    Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.

Reading

  • Text 1

    the history of the university lecture

    descriptive: its origins and historical development, period by period

    • 17TRUE / FALSE / NOT GIVEN
    • 813Table completion (ONE WORD ONLY)
    Show the brief the author receives
    UNIT r1 — Academic Reading, Passage 1. Fragment id "r1", section 1, Questions 1–13.
    Subject: the history of the university lecture.
    Treatment (descriptive): its origins and historical development, period by period.
    Length: 700–850 words.
    Question groups, in this order and with exactly these numbers:
    - Questions 1–7 — TRUE / FALSE / NOT GIVEN (group id "r1-g1"; type "judgement", variant "true-false-not-given")
    - Questions 8–13 — Table completion (group id "r1-g2"; type "completion", variant "table"): wordLimit {"words":1,"number":false} — the instructions must say "ONE WORD ONLY"
    Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
  • Text 2

    procrastination

    discursive: differing expert forecasts and what each assumes

    • 1420Matching headings
    • 2126Summary completion (NO MORE THAN TWO WORDS)
    Show the brief the author receives
    UNIT r2 — Academic Reading, Passage 2. Fragment id "r2", section 2, Questions 14–26.
    Subject: procrastination.
    Treatment (discursive): differing expert forecasts and what each assumes.
    Length: 750–900 words.
    Question groups, in this order and with exactly these numbers:
    - Questions 14–20 — Matching headings (group id "r2-g1"; type "matching", variant "headings"): 10 headings with roman-numeral ids
    - Questions 21–26 — Summary completion (group id "r2-g2"; type "completion", variant "summary"): wordLimit {"words":2,"number":false} — the instructions must say "NO MORE THAN TWO WORDS"
    Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.
  • Text 3

    amateur astronomy clubs

    argumentative: the writer calls for a change of policy and answers the objections

    • 2730Multiple choice
    • 3135YES / NO / NOT GIVEN
    • 3640Matching sentence endings
    Show the brief the author receives
    UNIT r3 — Academic Reading, Passage 3. Fragment id "r3", section 3, Questions 27–40.
    Subject: amateur astronomy clubs.
    Treatment (argumentative): the writer calls for a change of policy and answers the objections.
    Length: 800–950 words.
    Question groups, in this order and with exactly these numbers:
    - Questions 27–30 — Multiple choice (group id "r3-g1"; type "multiple-choice"): 4 single-answer questions (select: 1)
    - Questions 31–35 — YES / NO / NOT GIVEN (group id "r3-g2"; type "judgement", variant "yes-no-not-given")
    - Questions 36–40 — Matching sentence endings (group id "r3-g3"; type "matching", variant "sentence-endings")
    Difficulty: Standard — aimed at candidates working towards Band 6–7. The density, vocabulary and paraphrasing of a typical live paper; distractors are plausible and need careful reading or listening to reject.

Writing

  • Tasks 1 and 2

    two pie charts for two years or two groups (one "pie" visual with two series, each summing to 100): tea and coffee consumption in four countries

    Essay (opinion): young children cared for by grandparents or by nurseries

    Data story: the rank order at the end is the reverse of the start.

    Show the brief the author receives
    UNIT writing — Academic Writing.
    Task 1 (Academic): two pie charts for two years or two groups (one "pie" visual with two series, each summing to 100), showing tea and coffee consumption in four countries. Visual kinds, in order: pie.
    The invented data must tell this story: the rank order at the end is the reverse of the start.
    Task 2: essayType "opinion", on young children cared for by grandparents or by nurseries (Family and society). Question frame: To what extent do you agree or disagree?
    Timing, fixed by the exam: minutes 60; Task 1 minWords 150 and suggestedMinutes 20; Task 2 minWords 250 and suggestedMinutes 40.
    Difficulty: Standard — prompts as abstract and as worded as those of a typical live paper.

Speaking

  • Parts 1–3

    Describe a person who taught you something useful

    Part 1: the home you live in, giving gifts, weekends. Part 3: role models, influence and relationships between generations.

    Show the brief the author receives
    UNIT speaking — Speaking (identical for both modules).
    Part 1 topics, in this order, four questions each: the home you live in; giving gifts; weekends.
    Part 2 cue card (A person): "Describe a person who taught you something useful".
    Part 3 theme: role models, influence and relationships between generations. Five or six questions, moving from concrete comparison to abstract speculation.
    Timing, fixed by the exam: minutes 14; Part 2 prepSeconds 60 and speakSeconds 120.
    Difficulty: Standard — prompts as abstract and as worded as those of a typical live paper.

Scale and cost

Thousands of tests is arithmetic, not a slogan

Two questions decide whether a test factory scales: how many genuinely different tests can be planned, and what each one costs to write and check.

How many different tests can be planned?

at least 1036

distinct Academic blueprints. This is a lower bound: it multiplies only the choices of topic, treatment and question mix below, and ignores names, accents, word limits, question-count splits and difficulty.

Independent choices the blueprint sampler makes, and how many options each has
ChoiceOptions
Listening Part 1: situation × twist384
Listening Part 2: talk × focus204
Five long-text subjects, each from a different domain14.2 billion
Discussion format, lecture angle and three passage treatments24,576
Question-type mixes across the seven objective units483,840
Writing Task 1: visual × data story × subject, or a process1,938
Writing Task 2: essay type × issue240
Speaking: opener × two Part 1 topics × cue card227,040

The honest reading of that number

A different blueprint is not automatically a test that feels different. What a keen learner notices is a subject coming round again. The tightest pool today is “long-text subjects (five used per test)”: after about 25 tests a subject must reappear, though in a different role (a lecture rather than a passage) and with a different treatment. The pools are plain data — 126 subjects in 14 domains, 88 listening situations, 50 short-text kinds, 190 writing prompts, 80 cue cards — so widening them is a one-line change, not an engineering project.

What does a test cost to make?

One full test is about 33 model calls: nine units written, seven of them sat blind, probed without the text and, where the solver disagrees, adjudicated. Prices are per million tokens in and out; the Batch API halves them for work that can wait up to a day.

  • Claude Opus 5

    $5 in / $25 out per million tokens

    Per test, live
    $3.36
    Per test, batch
    $1.90
    1,000 tests, live
    $3,356
    1,000 tests, batch
    $1,904
  • Claude Sonnet 5

    $2 in / $10 out per million tokens

    Per test, live
    $1.53
    Per test, batch
    $0.95
    1,000 tests, live
    $1,527
    1,000 tests, batch
    $946
  • Claude Haiku 4.5

    $1 in / $5 out per million tokens

    Per test, live
    $0.92
    Per test, batch
    $0.63
    1,000 tests, live
    $917
    1,000 tests, batch
    $627

Every figure includes $0.31 per test for neural text-to-speech (19,200 characters at $0.016 per thousand), which the batch discount does not touch. The prototype’s own tests were voiced with ElevenLabs (the turbo model, half a credit per character: about 9,600 credits a test), and every spoken line is cached so a script edit re-renders only the lines that changed.

Show the stage-by-stage breakdown and assumptions
One Academic test on Claude Opus 5, live API, shared system prompt cached
StageCallsTokens inTokens outCost
Author9.049,50042,395$1.13
Repair rounds4.538,84719,437$0.59
Blind solver7.014,90514,960$0.45
No-text probe7.06,6804,460$0.14
Adjudicator3.526,72320,248$0.58
Confirming solve2.55,2175,236$0.16
  • Fractional calls are averages: 0.6 repair rounds per marked unit, and an adjudication for 50% of them. These are planning assumptions, to be replaced by measured token usage from real batches.
  • The item-writing standard is a single unchanging system prompt of about 5,050 tokens, so it is cached: a cache read costs a tenth of the normal input price. Inside a batch, cache hits are best-effort, so the batch columns assume 60% rather than 90%.
  • Thinking tokens are billed as output and are included. Human review time is not.

Limits

What this approach cannot promise yet

Generated tests are practice material. Here is what stands between this prototype and material you could calibrate an exam on.

  • A language model’s question is a draft until another model has failed to break it

    Models write fluent questions with confident answer keys, and some of those keys are wrong or arguable. That is why the pipeline publishes nothing on the author’s word: a generated draft has to survive the structural gates and a blind solve before it earns the validated badge. The solver can be a different model from the author (one setting), but in this prototype it is the same one, and two passes by one model can share a blind spot — which is what the human spot review is for. The four seed tests in the library were written by hand and pass the structural gates, but no blind solver has sat them yet, so none of them carries the badge.

  • Difficulty is a target, not a measurement

    “Foundation”, “standard” and “stretch” are instructions to the author. How hard a question really is can only be measured from the answers of real candidates. Until a test has been sat a couple of hundred times, its band estimate rests on the published raw-score tables for real papers, and a generated test may run a little easier or harder than those.

  • Band scores are estimates

    Listening and Reading bands come from raw-score tables; Writing and Speaking bands come from a model examiner working to the public band descriptors (or, offline, a clearly labelled heuristic), always shown with a likely range. None of them is an official result, and Bandwise is independent of the organisations that own IELTS.

  • The prototype’s voices are neural text-to-speech, cast by accent and gender

    Each speaker gets one ElevenLabs voice, chosen by the accent and gender in the blueprint from the voices on the account; when the account has no voice for an accent the nearest one stands in, and every substitution is written to the audio manifest — today’s library sounds more British than its blueprints asked for. The voices are clear, but they do not overlap or interrupt the way people do. The free macOS voices remain the offline fallback, and the costs above price the same service at list rates.

  • Human review and live calibration are designed, not yet running

    The last two stages of the pipeline need reviewers and real attempt data. In this prototype, review means a person reading the QA report, and no question has yet been retired on evidence from candidates.

    Follow the calibration study