A Holdout Experiment determines the effectiveness of an email. When running a Holdout Experiment, Clutch will “holdout” a group of customers who will not receive an email and compare their purchase behavior against an equal group of customers who will receive an email. Customers not selected for Holdout or for comparison will also receive the email.
Clutch provides the tools for brands to go a step further in their communication. The results and reports from holdout experiments allow marketers to easily identify improvements and implement change that drives ROI. For example, if people who did not receive an email spend just as much money as people who did receive an email, then you can conclude that your email is not as effective as it could be.
By running a Holdout test, you will be able to see if your email is effective toward achieving your goal. The main goal is to optimize revenue and marketers can select which KPI they want to focus on.
Optimizing Revenue
- AOV – does my email drive up the average order value?
- Total Number of Purchases – does my email increase the total number of purchases?
- Spend per Customer – does my email increase the spend per customer?
- People who get my email make 10 purchases per year
- People who do not get my email make 2 purchases per year
Example
If you have 1,000 total people, and you are testing 10% of these people, 100 people will be in the experiment. 50 people will receive an email, and 50 people will not receive an email. The 900 people will get the email as normal and are not included in the reporting.
