Email A/B Testing Framework

TESTING STRATEGY

Test Prioritization Matrix

Test Type	Impact	Effort	Priority
Subject line variations	High	Low	P1 - Test weekly
Send time optimization	Medium	Low	P1 - Test monthly
Preheader text	Medium	Low	P2 - Test bi-weekly
CTA button text	Medium	Low	P2 - Test monthly
Email length	Medium	Medium	P3 - Test quarterly
Design/layout changes	Medium	High	P3 - Test quarterly
Personalization elements	High	High	P4 - Test when resources allow
Segmentation strategies	High	High	P4 - Test when resources allow

Testing Roadmap Template

QUARTERLY TESTING ROADMAP

Q_: _______________

MONTH 1:
Week 1: Subject line test - [Hypothesis]
Week 2: Subject line test - [Hypothesis]
Week 3: Send time test - [Hypothesis]
Week 4: Analyze + document learnings

MONTH 2:
Week 1: CTA test - [Hypothesis]
Week 2: Subject line test - [Hypothesis]
Week 3: Preheader test - [Hypothesis]
Week 4: Analyze + document learnings

MONTH 3:
Week 1: Content/design test - [Hypothesis]
Week 2: Subject line test - [Hypothesis]
Week 3: Personalization test - [Hypothesis]
Week 4: Quarterly analysis + planning

QUARTERLY TARGETS:
• Tests run: ___
• Winning variations: ___
• Lift achieved: ___%

---

TEST DESIGN

Hypothesis Framework

TEST HYPOTHESIS TEMPLATE

TEST NAME: _______________
TEST ID: _______________
DATE: _______________

HYPOTHESIS:
If we change [variable]
From [current state] to [new variation]
We expect [metric] to [increase/decrease] by [amount]
Because [reasoning]

VARIABLE BEING TESTED:
☐ Subject line
☐ Preheader
☐ Send time
☐ CTA
☐ Content
☐ Design
☐ Personalization
☐ Other: _______________

VARIANTS:
Control (A): _______________
Treatment (B): _______________
Treatment (C): _______________ (if applicable)

SUCCESS CRITERIA:
Primary metric: _______________
Target lift: ___%
Minimum detectable effect: ___%

Sample Size Calculator

List Size	Min per Variant	Test Duration	Confidence
1,000	500	Full send	80%
5,000	1,000	20% test	90%
10,000	2,000	20% test	95%
25,000+	2,500	10% test	95%

Statistical Significance Rules

STATISTICAL SIGNIFICANCE CHECKLIST

BEFORE DECLARING WINNER:
☐ Minimum sample size reached
☐ 95% confidence level achieved
☐ Test ran for minimum duration
☐ Results are practically significant (not just statistically)

MINIMUM REQUIREMENTS:
• Open rate tests: 500+ per variant minimum
• Click rate tests: 100+ clicks per variant minimum
• Conversion tests: 30+ conversions per variant minimum

COMMON MISTAKES TO AVOID:
✗ Ending test early when result "looks good"
✗ Cherry-picking metrics that show wins
✗ Testing too many variables at once
✗ Using small samples for low-frequency events

---

TEST TYPES

Subject Line Testing

Test Element	Variations	Example
Length	Short vs. long	"New guide" vs. "Our comprehensive guide to B2B marketing"
Personalization	With/without name	"[Name], your report" vs. "Your report is ready"
Emoji	With/without	"🚀 New feature launch" vs. "New feature launch"
Number	With/without	"5 ways to grow" vs. "How to grow faster"
Question	Statement vs. question	"Improve your pipeline" vs. "Want a better pipeline?"
Urgency	With/without	"Last chance" vs. "Don't miss this"

Subject Line Test Protocol

SUBJECT LINE A/B TEST

CONTROL (A): _______________
Characters: ___
Key element: _______________

VARIATION (B): _______________
Characters: ___
Key element: _______________

WHAT'S DIFFERENT:
☐ Length
☐ Personalization
☐ Tone
☐ Specific words
☐ Format (question/statement)

HYPOTHESIS:
Variation B will [increase/decrease] open rate by ___%
because _______________

RESULTS:
Control open rate: ____%
Variation open rate: ____%
Lift: +/- ____%
Winner: ☐ A ☐ B ☐ No difference

LEARNING:
_______________

Send Time Testing

Time Slot	Day	Audience Type	Best For
7-8am	Tue-Thu	Executives	Early risers, commute readers
9-11am	Tue-Thu	General B2B	Standard work hours
12-1pm	Tue-Wed	SMB	Lunch break readers
2-4pm	Tue-Thu	General B2B	Afternoon productivity lull
6-8pm	Tue-Wed	Executives	Evening catch-up

CTA Testing Framework

Element	Test Variations
Button text	Action verb differences: "Get" vs. "Download" vs. "Access"
Button color	Primary brand vs. contrasting vs. urgency (red/orange)
Button size	Standard vs. larger tap target
Button position	Above fold only vs. above + below
Link vs. button	Text link vs. button CTA
First person	"Get my guide" vs. "Get your guide"

---

LEARNING DOCUMENTATION

Test Result Template

TEST RESULT DOCUMENTATION

TEST NAME: _______________
DATE RUN: _______________
AUDIENCE: _______________
SAMPLE SIZE: _______________

HYPOTHESIS:
_______________

VARIANTS:
A (Control): _______________
B (Treatment): _______________

RESULTS:
             A           B          Lift
Opens:      ____%       ____%      +/-____%
Clicks:     ____%       ____%      +/-____%
CTOR:       ____%       ____%      +/-____%
Conv:       ____%       ____%      +/-____%

WINNER: ☐ A ☐ B ☐ No significant difference

CONFIDENCE LEVEL: ___%

KEY LEARNING:
_______________

ACTION TAKEN:
☐ Implement winner as default
☐ Test further with different segment
☐ Test different variation of concept
☐ No action (inconclusive)

APPLY TO:
☐ This email type only
☐ All marketing emails
☐ Specific segment: _______________

Testing Knowledge Base

Date	Test Type	Hypothesis	Result	Learning	Applied
	Subject line		Win/Loss/Tie		Y/N
	Send time		Win/Loss/Tie		Y/N
	CTA		Win/Loss/Tie		Y/N

Quarterly Testing Review

QUARTERLY TESTING REVIEW

PERIOD: _______________

TESTS CONDUCTED:
Subject line tests: ___
Send time tests: ___
CTA tests: ___
Other tests: ___
Total: ___

RESULTS SUMMARY:
Winning tests: ___
Losing tests: ___
Inconclusive: ___
Win rate: ___%

IMPACT:
Average open rate change: +/-___%
Average click rate change: +/-___%
Estimated additional conversions: ___
Estimated revenue impact: $___

TOP LEARNINGS:
1. _______________
2. _______________
3. _______________

NEXT QUARTER PRIORITIES:
1. _______________
2. _______________
3. _______________

---