We at Silverpop often hear email subject line questions such as, “What’s the sample size needed for subject line testing?” in conversations with clients and other marketers. My colleague Stephen Guerra wrote an excellent explanation ("Email Testing: Determining the Right Sample Size") in a previous blog post, and he and I came up with these additional thoughts on subject line testing after receiving yet another inquiry on the topic:
1. Testing is almost always a good idea. Just remember, though, that with a small sample size, there will be less confidence in the results (Is the winner really the winner? Is it repeatable?).
2. Don’t completely discount small sample sizes, though. Smaller samples can be meaningful if you see results that vary by a wide-enough margin.
3. With a small sample size, test only two subject lines. This will keep the sample size as large as possible, as opposed to testing three or four options.
4. Test concepts and direction. Pick completely different types of subject lines you want to test, such as discount versus benefits, humor versus serious, etc. In the initial tests you’ll want to test larger concepts that you can build on and refine with continuous testing.
5. Minimize variables. If you want to test discount versus benefits, for example, try to keep the number of subject-line characters the same or close to remove the variable and impact of length. Make sure that the call to action or key content is in the same location in the subject line (e.g., at the beginning).
6. Test the types of subject lines three times to increase confidence. However, don't test the same subject line over and over. Instead, test the style, with each subject line as close to the core/initial test as possible.
What you’re testing is not necessarily a specific subject line but a type (in essence a template) of subject line that you can use again and again with different offer copy.
7. Think about the goal of your email program. If your subject line has a call to action, use click-through rate and/or conversion rate/revenue and not just open rate to determine the winner.
If the creative is identical in both emails, then this shows the impact of the subject line on actual conversion, which is more important than just number of opens.
I've seen this firsthand with tests I ran for a retailer in which the subject line with the lower open rate actually produced more revenue than the subject line with the higher open rate. There are lots of tricks and approaches to increasing opens of subject lines, but this doesn't mean they produce the action you want.
8. Make sure you do a true apples-to-apples split test. Don’t vary a single aspect of the two emails, send them at the same time and do an every nth split of your list.
9. Allow enough time for results. What’s the life expectancy of your messages? You may find that your messages still garner opens and conversions days after they were sent.
If so, you may want to run your tests over a few days to see how they perform over their true lifetime. Of course, you may need to make a decision and send to the remainder of the list before then.
In that case, use the data available to you. However, don't forget that those tests are still running. Examine the results in a week or two, and see if the final results differ from your earlier reading.
10. Consider segmentation. Examine your results as they apply to different segments in your list to see if specific subject line types resonate better with certain subscriber groups.
11. Take credit! Make sure to internally publicize your testing successes. Let others know the numbers on how your testing improved revenue by detailing (in dollar figures) how much your test improved results.
To do this, extrapolate the revenue that would have been earned if the losing version had been sent to the entire list, and take the difference between that and the actual results.
Share what you learned with your creative team and others (merchandising, sales, brand managers) and make sure they understand the benefits and can offer suggestions for future tests.
More resources on email testing: