- name
When the user wants to plan, run, or read A/B tests, or asks a judgement call on one: is the result real, can we stop, which version won, is this worth testing at all: sample size, duration, significa
SKILL.md
The skill file stays in the publisher's repository.
Paste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add coreyhaines31/marketingskills --skill ab-testing -a cursor -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add coreyhaines31/marketingskills --skill ab-testing -a claude-code -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add coreyhaines31/marketingskills --skill ab-testing -a codex -yPaste the copied command in a terminal. It installs this SKILL.md into that agent.
npx skills add coreyhaines31/marketingskills --skill ab-testing -a grok -ySend the copied message in a Grok Bot chat. The Bot fetches this SKILL.md from GitHub and adds it to the skills all your Bots share.
Install this skill: https://github.com/coreyhaines31/marketingskills/tree/HEAD/skills/ab-testingFrom the publisher
You are an expert in experimentation and A/B testing. Your goal is to help design tests that produce statistically valid, actionable results.
Check for product marketing context first:
If .agents/product-marketing.md exists (or .claude/product-marketing.md, or the legacy product-marketing-context.md filename, in older setups), read it before asking questions. Use that context and only ask for information not already covered or specific to this task.
Before designing a test, understand:
Because [observation/data],
we believe [change]
will cause [expected outcome]
for [audience].
We'll know this is true when [metrics].
Weak: "Changing the button color might increase clicks."
Strong: "Because users report difficulty finding the CTA (per heatmaps and feedback), we believe making the button larger and using contrasting color will increase CTA clicks by 15%+ for new visitors. We'll measure click-through rate from page view to signup start."
Per variant, for 95% confidence (two-sided) and 80% power. Lift is relative (3% → 3.6% is a 20% lift).
Calculators:
For detailed sample size tables and duration calculations: See references/sample-size-guide.md
When a change only reaches visitors who interact with it (a hero video, a calculator, a chat widget), a test on overall conversion is underpowered: if 10% of visitors click play, the other 90% add noise and no signal. Either run a triggered analysis (log the same trigger point in both arms, such as the play button coming into view, and compare only visitors who reached it; don't filter on clicks, since the variant can change who clicks), or skip the test, ship the better version, and compare versions on engagement (watch-through) and viewer conversion, tagged by version ID. The second is a directional comparison, not a controlled test.
Considerations:
DO:
Avoid:
Looking at results before reaching sample size and stopping early leads to false positives and wrong decisions. Pre-commit to sample size and trust the process.
These distinctions follow the American Statistical Association's p-value statement. Statistical significance alone does not establish business value or justify shipping a variant.
Document every test with:
For templates: See references/test-templates.md
Individual tests are valuable. A continuous experimentation program is a compounding asset. This section covers how to run experiments as an ongoing growth engine, not just one-off tests.
1. Generate hypotheses (from data, research, competitors, customer feedback)
2. Prioritize with ICE scoring
3. Design and run the test
4. Analyze results with statistical rigor
5. Promote winners to a playbook
6. Generate new hypotheses from learnings
→ Repeat
Feed your experiment backlog from multiple sources:
Score each hypothesis 1-10 on three dimensions:
ICE Score = (Impact + Confidence + Ease) / 3
Run highest-scoring experiments first. Re-score monthly as context changes.
Track your experimentation rate as a leading indicator of growth:
When a test wins, don't just implement it — document the pattern:
## [Experiment Name]
**Date**: [date]
**Hypothesis**: [the hypothesis]
**Sample size**: [n per variant]
**Result**: [winner/loser/inconclusive] — [primary metric] changed by [X%] (95% CI: [range], p=[value])
**Guardrails**: [any guardrail metrics and their outcomes]
**Segment deltas**: [notable differences by device, segment, or cohort]
**Why it worked/failed**: [analysis]
**Pattern**: [the reusable insight — e.g., "social proof near pricing CTAs increases plan selection"]
**Apply to**: [other pages/flows where this pattern might work]
**Status**: [implemented / parked / needs follow-up test]
Over time, your playbook becomes a library of proven growth patterns specific to your product and audience.
Weekly (30 min): Review running experiments for technical issues and guardrail metrics. Don't call winners early — but do stop tests where guardrails are significantly negative.
Bi-weekly: Conclude completed experiments. Analyze results, update playbook, launch next experiment from backlog.
Monthly (1 hour): Review experiment velocity, win rate, cumulative lift. Replenish hypothesis backlog. Re-prioritize with ICE.
Quarterly: Audit the playbook. Which patterns have been applied broadly? Which winning patterns haven't been scaled yet? What areas of the funnel are under-tested?
© 2026 Skills Rack.
---name: ab-testingdescription: When the user wants to plan, run, or read A/B tests, or asks a judgement call on one: is the result real, can…---# skills/ab-testing/SKILL.mdOverview
When the user wants to plan, run, or read A/B tests, or asks a judgement call on one: is the result real, can we stop, which version won, is this worth testing at all: sample size, duration, significance, hypotheses, and experiment backlogs. For tracking setup, see analytics. For page-level conversion fixes, see cro.
FAQ
Download skills/ab-testing/SKILL.md from the publisher, or copy the path and open the repository. PlugMCP lists the skill and does not host the file.
The publisher keeps it at https://github.com/coreyhaines31/marketingskills.
Claude, Cursor, Codex, and Grok Build can read a SKILL.md you add from the publisher's repository. Grok Bot installs it when you send the skill's GitHub folder link in a chat.