A method is a named technique the team uses inside a block. In Focus, it is the comparison design chosen in Comparing to repeat one measurement across a variable for an initiative. It answers the practical question: how will we compare the options fairly? Methods also appear in other blocks, as test types in Testing and as prioritizing tools such as RICE in Bets. This page covers comparison methods in Focus.
For example, the team has an initiative to improve rewards visibility for new cardholders. The variable is the version, so they run an A/B test of two progress bar variants. That A/B test is the method.
Why it matters
When the method isn't chosen on purpose, teams fall back on habit. They run the same kind of study every time, or they ship straight to everyone and hope, and it becomes hard to tell what actually caused a change.
A named method makes the plan clear. Everyone knows what will be tried, how the options will be compared, and which UX metrics will decide the winner. It also makes results easier to trust, because the approach was set before the numbers came in.
Name the variable first, pick a method that fits it, name the metric it will be judged by, and write it down before the work starts.
How we write one
Every method is written as one sentence that names the initiative and the approach:
Pursue an initiative through an approach.
Pursue improving rewards visibility for new cardholders through an A/B test of two progress bar variants.
Writing it this way ties every method to the initiative it serves, so no approach floats free of the work it is meant to move forward.
How it differs from similar ideas
-
A test checks whether a hunch or assumption holds, usually with a small group and a test question. A comparison method repeats one measurement across a variable for an initiative that testing already supports, often at larger scale.
-
Comparing is the Focus block where the same measurement is repeated across one variable. A method is one of its techniques, such as an A/B test, a visual iteration, a before-and-after test or a benchmark.
-
An initiative is the body of work. A method is how that work gets done, and one initiative can use several methods.
-
A concept is what is being tried, such as the progress bar. A method is how it is being tried, such as an A/B test, and the variable is what changes between the options, such as the version.
A good method
-
Serves one named initiative and names its variable
-
Names a specific approach, such as an A/B test or a phased rollout
-
Says which UX metric will judge it
-
Names the audience or traffic it runs on
-
Is chosen before results come in
-
Has an owner
What happens to it
-
Proposed. Someone suggests an approach for an initiative.
-
Planned. The owner sets the metric, the audience and the length of the run.
-
Running. The approach is live and data is being collected.
-
Evaluated. UX metrics are read against benchmarks and the options are compared.
-
Completed or abandoned. It ends with a result that informs a bet, or it stops early if the setup proves unsound.
For Ray
This part tells Ray how to find a method in an input, such as an experiment plan, a sprint plan, a rollout schedule, an analytics dashboard note or a team standup, and how to store it.
How to find one
-
Mark statements about approach. Read the input and mark every statement about how a piece of work will be carried out.
-
Find the approach. Name it plainly, such as A/B test, multivariate test, visual iteration, before-and-after test, benchmark or phased rollout.
-
Find the initiative. Identify which initiative it serves. If none is named, leave it blank and lower confidence.
-
Find the metric. Look for the UX metric that will judge it. If none is named, leave it blank.
-
Find the setup. Note the variable (one of the 12), the audience or traffic share, the variants or phases, and how long it runs.
-
Note the owner. Record the person or role running it.
-
Write the sentence. Use the format: Pursue [initiative] through [approach].
-
Fill the record, score it, link it and store it using the rules below.
Cues to look for
-
Approach language: "we'll run", "split traffic", "roll out to 10% first", "two-week sprint"
-
Named techniques: A/B test, multivariate test, phased rollout, holdout, iteration sprint
-
Experiment plans with variants, traffic splits and a primary metric
-
Rollout schedules with phases and dates
What doesn't count
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 50px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>If the statement...</p></th><th colspan="1" rowspan="1"><p>Store it as</p></th></tr><tr><td colspan="1" rowspan="1"><p>Checks a hunch with a small group and a test question</p></td><td colspan="1" rowspan="1"><p>Test</p></td></tr><tr><td colspan="1" rowspan="1"><p>Describes what is being tried, such as a design</p></td><td colspan="1" rowspan="1"><p>Concept</p></td></tr><tr><td colspan="1" rowspan="1"><p>Names the body of work itself</p></td><td colspan="1" rowspan="1"><p>Initiative</p></td></tr><tr><td colspan="1" rowspan="1"><p>Reports a comparison point, such as a baseline</p></td><td colspan="1" rowspan="1"><p>Benchmark</p></td></tr><tr><td colspan="1" rowspan="1"><p>Commits to a change with a goal, workflows and a metric</p></td><td colspan="1" rowspan="1"><p>Bet</p></td></tr></tbody></table>
The record
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 75px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>Field</p></th><th colspan="1" rowspan="1"><p>What to fill in</p></th><th colspan="1" rowspan="1"><p>Values</p></th></tr><tr><td colspan="1" rowspan="1"><p>Statement</p></td><td colspan="1" rowspan="1"><p>The one-sentence method</p></td><td colspan="1" rowspan="1"><p>Pursue [initiative] through [approach].</p></td></tr><tr><td colspan="1" rowspan="1"><p>Initiative</p></td><td colspan="1" rowspan="1"><p>The initiative it serves</p></td><td colspan="1" rowspan="1"><p>Link to Initiative</p></td></tr><tr><td colspan="1" rowspan="1"><p>Approach</p></td><td colspan="1" rowspan="1"><p>The kind of approach</p></td><td colspan="1" rowspan="1"><p>A/B test, Multivariate test, Visual iteration, Before-and-after test, Benchmark, Phased rollout, Other</p></td></tr><tr><td colspan="1" rowspan="1"><p>Variable and variants</p></td><td colspan="1" rowspan="1"><p>The one thing that changes, and the options compared or rolled out</p></td><td colspan="1" rowspan="1"><p>One of the 12 variables, plus short text with links to Concept</p></td></tr><tr><td colspan="1" rowspan="1"><p>Audience</p></td><td colspan="1" rowspan="1"><p>Who it runs on, and how much traffic</p></td><td colspan="1" rowspan="1"><p>Link to Audience, with share if given</p></td></tr><tr><td colspan="1" rowspan="1"><p>UX metric</p></td><td colspan="1" rowspan="1"><p>The metric that will judge it</p></td><td colspan="1" rowspan="1"><p>Link to UX Metric</p></td></tr><tr><td colspan="1" rowspan="1"><p>Benchmark</p></td><td colspan="1" rowspan="1"><p>The comparison point</p></td><td colspan="1" rowspan="1"><p>Link to Benchmark</p></td></tr><tr><td colspan="1" rowspan="1"><p>Duration</p></td><td colspan="1" rowspan="1"><p>How long it runs</p></td><td colspan="1" rowspan="1"><p>Dates or period</p></td></tr><tr><td colspan="1" rowspan="1"><p>Owner</p></td><td colspan="1" rowspan="1"><p>The person or role running it</p></td><td colspan="1" rowspan="1"><p>Name or role</p></td></tr><tr><td colspan="1" rowspan="1"><p>Source</p></td><td colspan="1" rowspan="1"><p>The input and exact passage</p></td><td colspan="1" rowspan="1"><p>Link to Input</p></td></tr><tr><td colspan="1" rowspan="1"><p>Status</p></td><td colspan="1" rowspan="1"><p>Where it stands</p></td><td colspan="1" rowspan="1"><p>Proposed, Planned, Running, Evaluated, Completed, Abandoned</p></td></tr><tr><td colspan="1" rowspan="1"><p>Confidence</p></td><td colspan="1" rowspan="1"><p>How clearly it was specified</p></td><td colspan="1" rowspan="1"><p>High, medium, low</p></td></tr></tbody></table>
New methods start as Proposed. A method can't move to Running until it has a UX metric linked.
Scoring confidence
Confidence measures how fully the method was specified, not how likely it is to succeed.
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 50px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>Level</p></th><th colspan="1" rowspan="1"><p>When to use it</p></th></tr><tr><td colspan="1" rowspan="1"><p>High</p></td><td colspan="1" rowspan="1"><p>Written in a plan with an approach, an initiative, a UX metric and an audience</p></td></tr><tr><td colspan="1" rowspan="1"><p>Medium</p></td><td colspan="1" rowspan="1"><p>The approach and initiative are clear, but the metric, audience or duration is missing</p></td></tr><tr><td colspan="1" rowspan="1"><p>Low</p></td><td colspan="1" rowspan="1"><p>Mentioned in passing, such as "maybe we A/B it", with no initiative or metric</p></td></tr></tbody></table>
If any part of the record can't be filled from the source, leave it blank and lower confidence by one level.
Linking
-
Check for duplicates first. If an existing method uses the same approach for the same initiative and variants, add the new source to it instead of creating another.
-
Link to its initiative and its source input.
-
Link to the UX metrics that evaluate it and the benchmark it is compared against.
-
Link to the concepts being compared as variants.
-
Link to any bet its results inform, once evaluated.
When to store and when to ask
-
Store automatically when confidence is high.
-
Add to the review digest when confidence is medium or low, when it has no UX metric, or when it isn't linked to an initiative.
When to flag
-
A method is Running with no UX metric linked
-
A method has no initiative or variable behind it
-
A method's metric was changed after it started running
-
A method has passed its end date without being evaluated
-
A method's results contradict an open test or signal
Worked example
Input: an experiment plan in the team's shared drive.
"Rewards visibility, experiment 1. Variant A: progress bar toward next reward. Variant B: progress bar with dollar value. 50/50 split of new cardholders for three weeks. Primary metric: task success. Owner: product designer."
What Ray finds:
-
"Experiment 1" with two variants and a 50/50 split is an A/B test.
-
"Rewards visibility" points to the initiative it serves.
-
"Task success" is the UX metric, and "new cardholders for three weeks" is the audience and duration.
-
The two variants differ only in design version, so the variable is Iteration; each variant is stored as a Concept and linked as the things being compared.
Stored record:
<table xmlns="http://www.w3.org/1999/xhtml" style="min-width: 50px;"><colgroup><col style="min-width: 25px;"><col style="min-width: 25px;"></colgroup><tbody><tr><th colspan="1" rowspan="1"><p>Field</p></th><th colspan="1" rowspan="1"><p>Value</p></th></tr><tr><td colspan="1" rowspan="1"><p>Statement</p></td><td colspan="1" rowspan="1"><p>Pursue improving rewards visibility for new cardholders through an A/B test of two progress bar variants.</p></td></tr><tr><td colspan="1" rowspan="1"><p>Initiative</p></td><td colspan="1" rowspan="1"><p>Improve rewards visibility for new cardholders by end of Q1</p></td></tr><tr><td colspan="1" rowspan="1"><p>Approach</p></td><td colspan="1" rowspan="1"><p>A/B test</p></td></tr><tr><td colspan="1" rowspan="1"><p>Variable and variants</p></td><td colspan="1" rowspan="1"><p>Iteration. A: progress bar toward next reward; B: progress bar with dollar value</p></td></tr><tr><td colspan="1" rowspan="1"><p>Audience</p></td><td colspan="1" rowspan="1"><p>Customers: new cardholders, 50/50 split</p></td></tr><tr><td colspan="1" rowspan="1"><p>UX metric</p></td><td colspan="1" rowspan="1"><p>Task success (Behavioral, Leading, Design KPI)</p></td></tr><tr><td colspan="1" rowspan="1"><p>Benchmark</p></td><td colspan="1" rowspan="1"><p>Task success 42% from the value test</p></td></tr><tr><td colspan="1" rowspan="1"><p>Duration</p></td><td colspan="1" rowspan="1"><p>Three weeks</p></td></tr><tr><td colspan="1" rowspan="1"><p>Owner</p></td><td colspan="1" rowspan="1"><p>Product designer</p></td></tr><tr><td colspan="1" rowspan="1"><p>Source</p></td><td colspan="1" rowspan="1"><p>Experiment plan, November 2026</p></td></tr><tr><td colspan="1" rowspan="1"><p>Status</p></td><td colspan="1" rowspan="1"><p>Planned</p></td></tr><tr><td colspan="1" rowspan="1"><p>Confidence</p></td><td colspan="1" rowspan="1"><p>High: written plan with an approach, initiative, metric and audience</p></td></tr></tbody></table>
Next step: stored automatically and linked to the initiative; Ray will check back after three weeks for the evaluation.
Edge cases
-
Called a test. Teams often say "test" for any experiment. If it checks a hunch with a small group, store it as a Test; if it pursues an initiative at scale, store it as a Method.
-
Several methods at once. "A/B test, then roll out in phases" is two methods. Store each one and link them in order.
-
No metric named. Store the method at lower confidence and flag it, since results can't be judged without one.
-
Method changes mid-run. Don't edit the running method. Mark it Abandoned, store the new one, and link the two.