SEGway August 2014

Five Important Things to Look for When Reading Efficacy Research

The educational community at large and specifically funding sources (e.g., government, foundations) are insisting on scientifically-based research to support the products and services used in schools.  But, how do you know if the research examining the product or service you are considering is any good?  Ultimately, you need to rely on the review of others in the know through the peer review process (see below).  But, as with most purchasing these days, it pays to be an educated consumer.  So, here are five important things to look for when reviewing effectiveness research.
 

• Mistaking testimonials and case studies for efficacy research- Educational publishers and technology providers often include testimonials from happy users or case studies of successful implementations, in their product literature.  While feedback from colleagues who like the product and are successful is helpful, testimonials and case studies are not the same thing as scientific effectiveness research.  They are based on a single point of view or instance of use, and you are likely hearing only one side of the story. Only a properly designed and carefully executed study, with a sufficiently large number of participants, comparing users of the product or service to a similar or randomly assigned group of non-users can provide you with the evidence you need to be comfortable that the product effectively achieves desired outcomes.

      

• Research should be conducted by an independent research organization-It is difficult to judge objectively your own work; the same is true for product and service providers. To help ensure against a confirmatory bias--the tendency to want to prove your own position--an effectiveness research study should be conducted by a third-party, credible research organization.

• Inclusion of a Control Group-  
  Research that only reports on a group that used the product or service without reference to a comparison group who did not use the product or service is very limited.  Without a comparison group, we cannot know whether any effect shown (e.g., academic growth) is really due to the use of the product or service.  Sure, the group using the product may have seen academic growth, but if the amount of growth is no greater than seen with non-users the product may not be actually contributing anything.

• Inclusion of a sufficiently large and representative sample-
  You want to make sure that the researchers have based their results on a sufficient number of study participants that are reasonably similar in make up to the group with whom you plan to use the product or service.  It is hard to give an exact number, but small numbers of students in a limited number of classes/schools should give you pause. And, make sure that the demographic profile and other characteristics you think are important are consistent with your implementation.  For example, a study conducted in an urban high school may be less applicable to a planned implementation in a suburban middle school.
   
• Mistaking significance for magnitude-
  Most of us know to look for statistically significant results--to make sure that any relationships or differences found were unlikely to occur by chance alone.  That is all well and good, but it does not tell us how large the relationship or differences are.  The magnitude of the relationship or differences are often reported as an effect size.  Ways of reporting the effect size vary, but in most applications these effect sizes tend to be between 0 and 1, or are reported as a percentage ranging from 0 to 100. Larger values show a greater effect.  But, remember: There are many influences in education and even small effects may be enough to make product purchase worthwhile.

Measurement Moment 

Earthquakes and the Richter Scale

The Napa Earthquake last week reminds us of the destructive power of earthquakes.  But, how many of us really understand just how powerful they are and how we measure the magnitude of these earthquakes?  What is a 6.0 earthquake (like Napa)?  What is a 7.0 earthquake (like the Bay Area Quake in the late 80's)?  How do these differ?

First, a little history.  In 1935, Charles Francis Richter "invented" the Richter scale to measure the magnitude of earthquakes.  The "Richter Scale" ranges from 1-10 and measures the magnitude of the seismological waves associated with earthquakes recorded on a device designed for this purpose. 

Many people assume that an earthquake one unit greater in magnitude on the scale is simply one small unit larger and that each unit increase is the same size  (i.e., the scale is linear).  But, the Richter scale is a logarithmic scale (Base 10).  Each unit of increase is ten times greater than the previous unit.  So, for example, the Bay Area quake (6.9) was about 10 times the magnitude of the Napa quake.

Why use a logarithmic scale? Richter noticed that when using a linear scale, the scale was not sensitive enough at the bottom and too sensitive at the top.  A simple linear scale did not provide a good model nor meaningful information about the strength of earthquakes.  So Richter took the log of the values to create a scale that better modeled the meaningful differences between earthquakes of different magnitudes.  Earthquakes could be more meaningfully compared using the logarithmic scale. A detailed discussion of logs is beyond the scope of this newsletter, but this is an interesting introduction to logarithmic scales.
 

 SEG At Upcoming Conferences

Let's Meet!

What would you like to read about in upcoming SEGway issues?
Please let us know by clicking on the link below:  
 

SEGway Topics Poll
 

Thank you.