Pete and Repeat: Dealing with Pretests and Posttests

Read Chapter 10 in your TopHat text. This is a good presentation on the general topic of issues with repeated measures. But it does not specifically address the pretests and posttests we so often see in schools. So to add that, take a look at the logic of the Solomon 4 Group Design. Notice that now you need extra students who do things like only take the posttest, in order to be certain that there was no effect of taking the pretest itself.

Why do we give pre-tests?

• To set students to for learning, activating their prior knowledge
• To determine a baseline of their understanding
• To enable us to measure change/growth over the course of the lesson

We would do that first one, if we the pretest were an integral part of our lesson. We would do the other 2 to enable research/assessment. And therein lies the issue with pretest.

Once you give both a pretest and posttest on the same topic, now you not only have the problem of determining the impact of the pretest (solved by the Solomon 4 Group Design), but you also have to figure out how to use the values on the pretest to adjust the posttest scores. If you want the full analysis of the options, there is the  Demitrov and Rumrill (2003) 6 page explanation of your options.

OK.. so let’s say that for your Fall practicum study, you decided to give a pretest to both a Tech and a Comparison (No Tech) class. Note: the pretest and posttest should cover the same material with with the identical or equivalent questions and content, with the hope students will score low on the pretest and show growth on the posttest.  Now, with your educational research hat on, let’s consider a few things.

1. If you have both a pretest score and a posttest score for each student in the 2 groups, Technology and Comparison groups, what score do you analyze to see if the Technology group was better?
   1. subtract the pretest score from the posttest score to get a “Gain Score” also called a “Difference Score” or a “Change Score.”
   2. use something like a (repeated measures) t-test to see if the 2 groups are different on the pretest. If they are not different, then just use the posttest scores to see if the technology intervention worked.
   3. Use some fancy statistical tool, like ANCOVA to partial out the variance associated with the pretest, from the posttest before you test for difference on those posttest scores.
2. Not to belabor the point, but now consider this 1 page review of the impact of the school cafeteria on the weight gain of girls vs boys. In this brief description, you can get the idea that not all statistical analyses address exactly the same question about the data. The choice of precisely which way to analyze your pre-post data is complicated, and relates to precisely what question you have about the technology (or cafeteria) intervention you did.

And what about if you don’t have a Comparison Class? Well, that’s OK too. If you only have pre- post scores from one class, you can still use statistics to see if the change (if any) is bigger than could be explained by random simply fluctuations or measurement error (see Estrada et al. 2019 for more details for those who love stats!).

Essential Understanding