By Alan Sharpe, CFREDirect mail fundraising is a soccer game where the opposing team keeps moving the goal posts.
A premium that worked last year doesn’t work today. A package design that worked at your last charity doesn’t work at your new one. A proven way to acquiring new donors gradually fails.
How can a tested, proven tactic stop working?
When you test one thing against another in the mail and Thing A outperforms Thing B, you know what works, right? The key to knowing what works in direct mail fundraising is testing, right?
Well, sort of.
There are at least three times when you should ignore your test results.
When Charities are in Different Sectors
I know of two charities. One is a national animal welfare group that relies heavily on premiums (greeting cards, address labels, note pads) to acquire and retain donors. The other charity is a national human rights organization where one in four of its donors hate premiums, never respond to them, and ask the charity to never mail them.
As you can see, a tested tactic that works for one charity will not necessarily work for another. When you move from one charity to another, ignore your test results from your former charity, or at least re-test them to make sure they are valid at your new charity.
When the Test Results are Dated
What worked in the mail once may not work again. When was the last time you received a CD in the mail from America Online (AOL) promoting their dial-up Internet service? There was a time when half of all CDs produced worldwide had an AOL logo on them. In the late 1990s, AOL was signing up new subscribers at the rate of one every six seconds.
No more.
Outer envelopes are another good example. Every charity used to mail plain #10 envelopes. Then one charity put teaser copy on its envelopes and saw a lift in response.
Other charities heard about it, copied the tactic, and soon every charity that put teaser copy on its outer envelopes saw a lift in response. Years later, when everyone was using teasers, a charity mailed a plain #10 envelope and saw a lift in response. Plain envelopes were suddenly different.
Be prepared to ignore your test results if they are from tests conducted a while ago. Technology changes. Donors change. Test results change.
When Your Sample Size is Too Small
To get test results that you can trust you need to receive at least 50 responses. To get 50 responses at a 1% response rate, you must mail at least 5,000 pieces. If you are mailing two packages and measuring the difference in results between the two of them (response rate, average gift, cost to raise a dollar, for example), then you must mail at least 5,000 of each package for a total mailing quantity of 10,000. If your test sample is smaller than 5,000 pieces of mail, don’t trust your results.
Another time you should not trust direct mail fundraising test results is when they are someone else’s results. Mine, for example. If you want to be confident that what I’m saying is valid, test my advice for yourself. I always tell the truth, but what is true for me may not be true for you. There are no absolute truths in direct mail fundraising, except this one: There are no absolute truths in direct mail fundraising.
These are great tips that tell us to do more appropriate and timely testing.