[Note: there are no TopHat readings for this module]

Good Science, Bad Science

At this point in our semester’s work, we can agree that good science looks like careful thinking that controls most factors and varies the one that theory suggests is effective. The best example of this would be medical science, right? But education is similar, it is a clinical activity where individuals are treated. So let’s take 12 minutes to examine medical science with a Ben Goldacre TED talk. Think about your practicum data from the Fall once again… and how you felt about any results that were anything less that completely supporting your idea– and then think how important negative trial results are to making decisions. Scholarly work is about refuting or disconfirming our hypotheses, not proving them.

Novelty Effect (“ooooh shiny”)

Novelty is good right? Try a new tech in your class and it’s exciting and eye-catching, stimulating and rewarding, most students will naturally love it… true? Surely some brain science can confirm this simple concept once and for all. Or not. Many things beside novelty can make conclusions about applying new technology difficult. Novelty is said to improve student achievement, behavior, attitudes, even memory, at least initially and for a short while. But does it? Certainly with something like VR you might expect a substantial novelty effect (See Tsay et al. 2020 Novelty and VR) Let’s check the research on memory too (Poppenk 2010 pdf).

Let’s consider this 2018 study supporting the use of smart phones in use in the Physics classroom. Hochberg 2018 pdf. This study had an experimental and control group. Was it the same teacher? Did the students know who was the target of the study? Is Hochberg right that tech may be more novel for girls, so we should worry more about it having a gender bias simply due to novelty? Certainly the researchers thought they should worry about a novelty effect, but they only reported research on Hawthorne to support that belief. The Adair 1989 pdf meta analysis (cited in Hochberg to justify their discussion of novelty) certainly believed that novelty was a separate factor from the attention people get from being in a study (Hawthorne see below). Confusing? Perhaps there is more study to be done here.

Hawthorne Effect (Experimental demands or any attention is good attention)

Do your students tend to act different in front of a camera in your classroom? A topic related to Novelty is the Hawthorne effect. Basically that when people know they are the focus of research (attention) they tend to act different and sometimes change just to meet the expectations (or the perceived expectations) of the experimenters. The basic technology with how bright the lights were illuminating an electric plant and how that might help improve worker productivity. The basics of the findings at the Hawthorne Works of Western Electric were that it didn’t matter whether they turned the lights up or down, productivity always improved. What? that doesn’t fit the theory at all! A simple explanation was that the workers understood when the lighting changed that they were being studied, and so they improved because they appreciated the attention they were getting. At least someone cares.

In 1939, the Harvard researchers were widely criticized, and even told that they were fools not to understand the Marxist description of workers and their class struggle. Strange how even politics could enter in to the discussion of basic research, right? Glad we’re over that!

Some notes on the Hawthorne effect.

Placebo Effect

From most drug studies we know that it is important for research to be at least “single blind,” meaning the patients don’t know if they are getting the active drug or a sugar pill, and better still “double blind,” meaning neither the researchers/doctors collecting the data nor the patients know who is getting the real thing. Our beliefs and expectations can be quite powerful for influencing our behavior, and this is why the placebo effect (using a fake intervention) has been so well documented. Reflecting back to the TED talk at the start of this section, you should consider that a Control group (who gets nothing) is easy to beat, while a Control group that gets the best current practices is harder to beat.. and a Control group that receives a fake (placebo) treatment is really hard to beat.

Teacher Expectations of children (Pygmalion effect).

In many studies where one of the variables is the teacher, the effect of different teachers can often be bigger than the effect of different treatments or even bigger that factors often considered to be critical to good outcomes, like class size or Socio-economic-status (SES). Teachers can have up to a 1 standard deviation effect on measure of student learning– but interestingly, it is not well understood how that effect actually takes place. In fact Hattie has listed 252 possible aspects to this effect. This may be due to teacher expectations and subtle differences in how they treat students for whom they have high or low expectations (whether related to stereotypes or not). But there is solid evidence from at least one study Nye 2004 pdf that the impact of a good teacher can overcome many things. A recent blog post summarizes this older study, concluding, as many do, that teachers matter more than most other factors.

Essential Understanding:

You might think you know what a Novelty effect is. But research is about careful thinking, and looking to disprove things you think you know (theories): Questioning everything. From this module you should see how careful thinking leads to doubting many common sense explanations and not overlooking alternative simple explanations. And it should also demonstrate how looking carefully at something often leads to more questions, rather than simple answers. That’s the nature of scholarly work and the result of a commitment to relying on scholarly evidence rather than intuition or opinion.