Testing,1,2,3...
From GrassrootsPedia
Slides and audio from the "Testing, 1,2,3.." webinar, presented by Jeff Regen, Defenders of Wildlife
Some Q&A from the webinar:
1. When testing donations, because the donation sample will likely look nothing like a normal distribution, is the t-value the right table to use, or should i be looking at a different value?
- Answer below from Nirmal Mankani, Fall 2009 NOI Fellow*
I'm gonna be honest, I've never stringently checked the normality assumption when I've done significance tests, but it's a good point and I probably should. Most of the testing I've done in the past involve binary outcomes (someone votes or they don't, someone takes an action on an e-mail or they don't, etc), so I've been curious about how to deal with donations myself.
First I'd eyeball / do a test to make sure your sample data differs significantly from a normal distribution. there are a couple tests to do this that can be found in most stats programs (ex http://en.wikipedia.org/wiki/Shapiro%E2%80%93Wilk_test ). There's also something called a Normal Q-Q plot (also found in most stats programs http://en.wikipedia.org/wiki/Q-Q_plot ) that should look like a straight diagonal line if your data are normally distributed.
If the assumption of normality is violated, it's possible to correct it with a transformation of your data (as long as you apply the transformation across all of your data, so the relationships between variables remain intact, you're just changing the scale that things are measured). So taking the log, reciprocal, square root, etc might give you a normal distribution.
Finally, if your data aren't normal, and there's no way to transform it to become normal, there are non-parametric tests that allow you to do significance tests that have counterparts among the parametric statistics. I have never used them but worth checking out: http://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U
2. On multivariate testing, what are good programs to use, and resources to learn about the process?
From Jeff:
- Optimost
- Omniture's multivariate testing tool
- Google Web Optimizer (free)
From Nirmal:
- I'm not really familiar with the web tools out there for doing testing with e-mail, but i think it's a good idea to get a stats program like SPSS or STATA -- there are lots of great e-mail lists and tutorials on the web to learn how to use them, if you're comfortable with excel or databases you should be able to pick it up. Then import/export from your vendor's e-mail tool to conduct and analyze the test.
