After running an A/B Experiment or a Holdout Experiment, you will receive a report. All reports vary slightly depending on what you choose as your goal. As a marketer, you want repeatable results. You want to be confident that if the test was run again, the results would be the same. These reports will give you a better insight into the effectiveness of your communication.
Based on the goal, here are the 8 options of reports one may receive.
- Holdout – Optimize revenue – AOV
- Holdout – Optimize revenue – Spend per customer
- Holdout – Optimize revenue – Total number of purchases
- A/B – Optimize revenue – AOV
- A/B – Optimize revenue – Spend per customer
- A/B – Optimize revenue – Total number of purchases
- A/B – Optimize engagement – Opens
- A/B – Optimize engagement – Click throughs
Holdout Only
You are comparing the group who did receive the email with the group who did not receive the email. For example, if 18 out of 50 people who received the email made a purchase online and if 20 out of 50 people who did not receive the email made a purchase online, you can conclude that your email is ineffective. On the other hand, if 40 out of 50 people who received the email made a purchase and 2 out of 50 people who did not receive the email made a purchase, you can conclude that the email you sent out was effective.
A/B Only
There are two different outcomes of A/B testing, winner or insignificant. For example, if 30 out of 50 people opened email A, and 32 out of 50 people opened email B, those results are insignificant. On the other hand, if 40 out of 50 people opened email A, and 4 out of 50 people opened email B, then we can conclude that email A is the winner, and therefore have more confidence that email A will be more successful.
