A test question is one specific question a test must answer to tell whether a concept works. It's small, direct and answerable from what happens in a session. It's written in Hunches, alongside the hunch it checks, and answered in Testing.
For example, the rewards team is testing a new progress bar with new cardholders. One test question is: "Can participants say what 5,000 points buys?" If they can, the concept is making point value clear. If they can't, it isn't.
Why it matters
Tests often go wrong because nobody agreed on what they were meant to answer. Sessions wander, the team collects interesting but unrelated feedback, and at the end it's hard to say whether the concept worked.
Good test questions keep a test on track. Each one points to a single thing the team needs to learn, so every task and every observation has a purpose. They also make results easy to share, because each question gets a clear answer.
Write a few sharp questions before the test starts, make sure each one can be answered by what participants do or say, and tie each one back to the hunch being checked.
How we write one
Every test question is written as one question that starts with Can, Do or Which:
Can / Do / Which a question that the test can answer?
Can participants say what 5,000 points buys?
Starting with Can, Do or Which keeps the question answerable by what people actually do in a session, rather than by opinion or debate.
How it differs from similar ideas
-
An ask is a question that enters the whole decision loop, such as whether to redesign the rewards dashboard. A test question lives inside one test and checks one thing about a concept.
-
A test is the whole plan: what's checked, with whom, how, and the expected result. A test question is one piece of that plan.
-
A hunch is a belief. A test question is how the team finds out whether part of that belief holds.
-
An observation is what was seen or heard. It's the raw answer to a test question.
A good test question
-
Starts with Can, Do or Which
-
Asks one thing
-
Can be answered by what participants do or say
-
Belongs to one test
-
Ties back to the hunch or assumption being checked
-
Doesn't lead participants toward an answer
What happens to it
-
Drafted. The team writes it with the hunch it checks, in Hunches, or Ray finds it in a research plan or discussion guide.
-
Attached. It is linked to a test and the concept that test evaluates.
-
Asked. The test runs, and observations are captured against it.
-
Answered. Observations and UX metrics give it a clear answer, which feeds a signal.
-
Unanswered or dropped. The test couldn't answer it, or it was removed; the reason is kept.
For Ray
This part tells Ray how to find a test question in an input, such as a discussion guide, a research plan, a test brief, a usability script or a test readout, and how to store it.
How to find one
-
Mark questions inside a test. Read the input and mark every question a test is meant to answer, including success criteria phrased as tasks.
-
Check the scope. If the question is about whether the team should do something, store it as an Ask. If it's about how a concept performs in a test, it's a test question.
-
Rewrite as Can, Do or Which. Turn tasks and goals into a question, such as "Task: name what 5,000 points buys" into "Can participants say what 5,000 points buys?"
-
Find the test. Link it to the test it belongs to. If no test exists yet, flag it.
-
Find the concept. Identify the concept the question evaluates.
-
Find the answer, if any. If the input includes results, link the observations and UX metrics that answer it.
-
Write the sentence. Use the format: Can / Do / Which [question]?
-
Fill the record, score it, link it and store it using the rules below.
Cues to look for
-
Questions in discussion guides and usability scripts
-
Task lists with success criteria: "participant should be able to..."
-
Research goals: "we need to learn whether..."
-
Readout headings that pose a question and then answer it
What doesn't count
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 50px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>If the statement...</p></th><th colspan="1" rowspan="1"><p>Store it as</p></th></tr><tr><td colspan="1" rowspan="1"><p>Asks whether the team should take an action</p></td><td colspan="1" rowspan="1"><p>Ask</p></td></tr><tr><td colspan="1" rowspan="1"><p>Is an interview prompt with no link to a concept, such as "tell me about your card"</p></td><td colspan="1" rowspan="1"><p>Leave it out; note it on the Input only</p></td></tr><tr><td colspan="1" rowspan="1"><p>States a belief to check</p></td><td colspan="1" rowspan="1"><p>Hunch or Assumption</p></td></tr><tr><td colspan="1" rowspan="1"><p>Describes the full test plan</p></td><td colspan="1" rowspan="1"><p>Test</p></td></tr><tr><td colspan="1" rowspan="1"><p>Reports what a participant did</p></td><td colspan="1" rowspan="1"><p>Observation</p></td></tr></tbody></table>
The record
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 75px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>Field</p></th><th colspan="1" rowspan="1"><p>What to fill in</p></th><th colspan="1" rowspan="1"><p>Values</p></th></tr><tr><td colspan="1" rowspan="1"><p>Statement</p></td><td colspan="1" rowspan="1"><p>The one-sentence question</p></td><td colspan="1" rowspan="1"><p>Can / Do / Which [question]?</p></td></tr><tr><td colspan="1" rowspan="1"><p>Test</p></td><td colspan="1" rowspan="1"><p>The test it belongs to</p></td><td colspan="1" rowspan="1"><p>Link to Test</p></td></tr><tr><td colspan="1" rowspan="1"><p>Concept</p></td><td colspan="1" rowspan="1"><p>The concept it evaluates</p></td><td colspan="1" rowspan="1"><p>Link to Concept</p></td></tr><tr><td colspan="1" rowspan="1"><p>Checks</p></td><td colspan="1" rowspan="1"><p>The part of the hunch or assumption it addresses</p></td><td colspan="1" rowspan="1"><p>Link to Hunch or Assumption</p></td></tr><tr><td colspan="1" rowspan="1"><p>Answer</p></td><td colspan="1" rowspan="1"><p>The answer, once known</p></td><td colspan="1" rowspan="1"><p>Yes, No, Partly, or the option chosen</p></td></tr><tr><td colspan="1" rowspan="1"><p>Evidence</p></td><td colspan="1" rowspan="1"><p>What answered it</p></td><td colspan="1" rowspan="1"><p>Links to Observation and UX Metric</p></td></tr><tr><td colspan="1" rowspan="1"><p>Drafted by</p></td><td colspan="1" rowspan="1"><p>The person or role who wrote it</p></td><td colspan="1" rowspan="1"><p>Name or role</p></td></tr><tr><td colspan="1" rowspan="1"><p>Source</p></td><td colspan="1" rowspan="1"><p>The input and exact passage</p></td><td colspan="1" rowspan="1"><p>Link to Input</p></td></tr><tr><td colspan="1" rowspan="1"><p>Status</p></td><td colspan="1" rowspan="1"><p>Where it stands</p></td><td colspan="1" rowspan="1"><p>Drafted, Attached, Asked, Answered, Unanswered, Dropped</p></td></tr><tr><td colspan="1" rowspan="1"><p>Confidence</p></td><td colspan="1" rowspan="1"><p>How clearly it was stated and tied to a test</p></td><td colspan="1" rowspan="1"><p>High, medium, low</p></td></tr></tbody></table>
New test questions start as Drafted, with source block Hunches. A question moves to Answered only when observations or metrics are linked to it.
Scoring confidence
Confidence measures how clearly the question was stated and how firmly it's tied to a test, not how important it is.
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 50px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>Level</p></th><th colspan="1" rowspan="1"><p>When to use it</p></th></tr><tr><td colspan="1" rowspan="1"><p>High</p></td><td colspan="1" rowspan="1"><p>Written as a clear question inside a named test, tied to a concept</p></td></tr><tr><td colspan="1" rowspan="1"><p>Medium</p></td><td colspan="1" rowspan="1"><p>Clear question, but Ray had to rewrite it from a task, or the concept is unclear</p></td></tr><tr><td colspan="1" rowspan="1"><p>Low</p></td><td colspan="1" rowspan="1"><p>Inferred from a loose research goal, or no test can be found</p></td></tr></tbody></table>
If any part of the record can't be filled from the source, leave it blank and lower confidence by one level.
Linking
-
Check for duplicates first. If the same test already has a question asking the same thing, add the new source to it instead of creating another.
-
Link to its test and concept and its source input.
-
Link to the hunch or assumption it checks.
-
Link the answer. When results arrive, link the observations and UX metrics that answer it, and the signal they feed.
-
Note reuse. If the same question is used in a later test, store it again for that test and link the two as related.
When to store and when to ask
-
Store automatically when confidence is high.
-
Add to the review digest when confidence is medium or low, when it has no test, or when the question looks leading.
When to flag
-
A test question has no test linked to it
-
A test has no test questions
-
A test is Complete but one of its questions has no answer
-
A question leads participants toward an answer, such as "Don't you find the progress bar clearer?"
-
A question really asks whether the team should act, and belongs as an Ask
Worked example
Input: a moderated usability script for the rewards concept test.
"Task 3: Show the participant the home screen with the progress bar. Ask them to tell you what 5,000 points would get them. Success = participant names a reward or a dollar value without help."
What Ray finds:
-
"Tell you what 5,000 points would get them" is the question, rewritten as "Can participants say what 5,000 points buys?"
-
"The home screen with the progress bar" is the concept being evaluated.
-
"Success = participant names a reward or a dollar value without help" tells Ray how the answer will be judged, and links to task success as the metric.
-
The script belongs to the planned abstract-points concept test, so the question is attached to it.
Stored record:
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 50px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>Field</p></th><th colspan="1" rowspan="1"><p>Value</p></th></tr><tr><td colspan="1" rowspan="1"><p>Statement</p></td><td colspan="1" rowspan="1"><p>Can participants say what 5,000 points buys?</p></td></tr><tr><td colspan="1" rowspan="1"><p>Test</p></td><td colspan="1" rowspan="1"><p>Test the abstract-points hunch with 10 new cardholders using a concept test; expect 70% task success on the point value task.</p></td></tr><tr><td colspan="1" rowspan="1"><p>Concept</p></td><td colspan="1" rowspan="1"><p>Points progress bar</p></td></tr><tr><td colspan="1" rowspan="1"><p>Checks</p></td><td colspan="1" rowspan="1"><p>Hunch: We think people ignore rewards because points feel abstract.</p></td></tr><tr><td colspan="1" rowspan="1"><p>Answer</p></td><td colspan="1" rowspan="1"><p>Blank: not yet asked</p></td></tr><tr><td colspan="1" rowspan="1"><p>Evidence</p></td><td colspan="1" rowspan="1"><p>Blank</p></td></tr><tr><td colspan="1" rowspan="1"><p>Drafted by</p></td><td colspan="1" rowspan="1"><p>UX researcher (Hunches)</p></td></tr><tr><td colspan="1" rowspan="1"><p>Source</p></td><td colspan="1" rowspan="1"><p>Usability script, Task 3, October 2026</p></td></tr><tr><td colspan="1" rowspan="1"><p>Status</p></td><td colspan="1" rowspan="1"><p>Attached</p></td></tr><tr><td colspan="1" rowspan="1"><p>Confidence</p></td><td colspan="1" rowspan="1"><p>Medium: clear question, but rewritten from a task</p></td></tr></tbody></table>
Next step: added to the review digest at medium confidence, with a prompt to the researcher to confirm the rewritten wording.
Edge cases
-
Written as a task. "Find what 5,000 points buys" is a task. Rewrite it as a Can, Do or Which question and keep the original wording in the source.
-
Mistaken for an ask. "Should we show points in dollars?" is about whether to act. Store it as an Ask, even if it appears in a test plan.
-
Two questions in one. "Can they find rewards and say what points buy?" is two questions. Split them so each gets its own answer.
-
Open-ended prompts. "What do you think of this screen?" isn't a test question. Note it on the Input and capture what people said as Observations.