The statistical software on this page allows researchers to analyze experiments in which the outcome variable is dichotomous (e.g., voting versus nonvoting). The statistical analysis is designed to accommodate situations in which only some of the subjects assigned to the treatment group actually receive the treatment, as well as situations in which some of the control group is treated inadvertently.
You supply six numbers: the number of people that you (1) assigned to the treatment group, (2) assigned to the control group, (3) successfully treated in the treatment group, (4) inadvertently treated in the control group, (5) found to have voted in the treatment group, and (6) found to have voted in the control group.
Melissa Michelson's door-to-door canvassing experiment in Dos Palos, California provides a nice illustration. Prior to the 2001 election, she assigned 466 people with Latino surnames to the treatment group and 298 to the control group. 342 of the people in the treatment group were successfully contacted. No one in the control group was contacted. In the treatment group, 86 people voted, whereas 41 people voted in the control group. The six numbers are therefore: 466, 298, 342, 0, 86, 41.
After entering these numbers in the appropriate boxes, click the "Submit" button. You will see output that summarizes the research findings and estimates the size and precision of the treatment effects. Check the statistical summary in order to ensure that you have entered the data correctly. The computer will summarize the voting rates and contact rates in the treatment and control groups. Next, examine the "intent-to-treat" estimate. This number is calculated by subtracting the voting rate in the control group from the voting rate in the group assigned to the treatment. In this example, the intent-to-treat estimate is 4.7, suggesting that assignment to the treatment group raised turnout by 4.7 percentage-points. Beneath this figure is the standard error of the estimated intent-to-treat effect. The larger this number, the more uncertainty surrounds the intent-to-treat estimate. The "treatment effect" is estimated by dividing the intent-to-treat estimate (4.7) by the contact rate (.73), which produces the number 6.4. Those who were actually treated became 6.4 percentage-points more likely to vote. The uncertainty of this estimate is measured by its standard error, 3.7.
Finally, the statistical software makes two useful calculations. The first is the "1-tailed significance" of the estimated treatment effect. When conducting GOTV experiments, it is conventional to expect turnout to rise as the result of the treatment. The "null hypothesis" is that the treatment failed to increase turnout. The 1-tailed significance level states the probability of obtaining an estimate as large the estimated treatment effect (in this case 20) by chance. When this probability is below .05, the estimate is said to be "statistically significant." In this case, the estimate is not statistically significant. Naturally, if the experiment were repeated, the results might come out differently. The "power" of an experiment describes the probability that it produces statistically significant estimates. In this case, the probability is low (23.8%), so it is not surprising that the estimate we obtained fell short of statistical significance. In order to increase the power in subsequent research, increase your contact rate and the number of observations in your study.
See: Michelson, Melissa R. 2003. Getting Out the Latino Vote: How Door-to-Door Canvassing Influences Voter Turnout in Rural Central California. Political Behavior 25(3):247-263, Table 1.
Yale Institution for Social and Policy Studies Publications
Don Green and Alan Gerber, Department of Political Science, Yale University, with technical support from Pete Emerson and Steve Citron-Pousty, Social Science Research Services, Yale University.
Angrist, Joshua D., Imbens, Guido W. and Donald B. Rubin. 1996. Identification of Causal Effects Using Instrumental Variables. Journal of the American Statistical Association 91(June): 444-455.
Gerber, Alan S. and Donald P. Green. 2000. "The Effects of Personal Canvassing, Telephone Calls, and Direct Mail on Voter Turnout: A Field Experiment." American Political Science Review, 94 (3): 653-64.