SEGWay January 2014

Top Questions Asked by Educators Who Are

Considering The Use of an Efficacy Study

What educators ask most often about Effectiveness Research

Over the past year, we've had the opportunity to visit with literally thousands of educators across the country.  Not surprisingly, we've found tremendous interest in Effectiveness Research and evidence-based educational practice.  Among the many questions we are asked, the following questions are by far the most common.  

• Why do I need a formal study; all of our customers tell us our product/service is effective?- Positive comments and testimonials are useful, but this does not constitute "scientific evidence."  A small set of client opinions is not considered "scientific evidence."  Scientifically acceptable proof of effectiveness requires a sound research design accepted by the scientific community.  School buyers and government funding agencies want to see a scientific study supporting the effectiveness of your product or service before buying or providing funds for purchase.  Buyers of school products and services are more savvy and are no longer willing to settle for products and services backed by unsubstantiated claims and testimonials.  They are looking for hard evidence that will give them confidence in solutions before buying. Many communities, still strapped for cash based on the recent economy, are shopping "smarter," by limiting purchases to those with research-based efficacy support.

  A strong efficacy study typically includes a scientific sample of users (treatment group) that are tested using reliable outcome measures both before using the product and after using the product.  And, it is best if we compare the results of those users to a comparison group (control group) that is not using the product. Proper statistical techniques need to be applied to evaluate whether the results obtained are greater than would be expected by chance alone.  Scientific effectiveness studies vary in scope (big-small, long-short, etc.) but tend to share these common features. 

  SEG can help you work through an effectiveness study design that makes sense for your product or service.  

• When should I begin planning for an efficacy study?- Often we hear from educational providers that they are ready to conduct a study and wish to start as soon as possible.  While we very much want to support everyone and admire their drive, careful planning and preparation is critical for effectiveness study success.  We encourage product and service providers to begin planning about 6 months before executing a study, or at least 3 months before starting the study.

  Planning and preparation include, first and foremost, designing the study.  Second, we also encourage clients to begin preliminary identification and recruitment of the sample 3-6 months ahead of time.  Third, those conducting an effectiveness study should leave plenty of time for the selection or identification of appropriate measures for the study.  While planning involves much more than this, these three upfront activities will dramatically influence the success of the study.

  Thinking about conducting an effectiveness study? SEG can help you begin planning to maximize your success.

What questions do you have?  Let us know what questions you want answered.  Contact hrickert@segmeasurement to share your feedback. 

Listening to our clients

What our clients have to say about us

At SEG, we like to think that we practice what we preach.  So, recently, we asked our current and past customers for feedback.  We were pleased to hear such positive feedback from them.  Here is a sampling of what our clients had to say....

SEG was a wonderful partner in our latest research project.  They were always a partner in the project, on time for every task, provided the appropriate follow-up, and communicated with a frequency that far exceeded my expectations.  Not only would we use them as an external evaluator again, we have relied on them to help us with grant applications for future work we hope to be involved in.  I can't recommend them highly enough. Marc Liebman, Imagine Learning     As a professional who is assessment/data savvy, but who certainly is not at the expert/scholar level, I especially appreciate working with SEG and the way they make the assessment process collegial. There is no substitute for having access to your provider to have brief, but meaningful and necessary, conversations about your project. [They] are simply THE BEST! Karen Faircloth, Northwest RESA

SEG Measurement Awards SXSWedu Scholarship

Teacher Melissa Contreras to Attend Conference

As one of our goals, SEG Measurement is committed to developing a greater understanding of educational assessment and efficacy research.  To that end, we are sponsoring a scholarship for a teacher to attend the upcoming SXSWedu Conference.  Melissa Contreras, a 16 year veteran Social Studies teacher in Texas, was selected.   

 

Melissa has worked in two recognized districts that have varied from small to large districts and currently works in Austin ISD.  Her teaching experience has been with student backgrounds from mixed socio-economic areas, first generation high school graduates and first to attend college.  Her mantra, "make the connection" has been her goal in her teaching experience; guiding her students to expand their knowledge and skills using the new resources and technology that arise today. We are pleased to support Mrs. Contreras' attendance at SWSXedu. 

Efficacy Task Force Continues to Move forward
Defines Areas to be Addressed by the Efficacy Research Guidelines

As most of you know, in 2013 SEG formed a Task Force to develop much-needed guidelines for teachers and administrators to use in understanding efficacy research from educational product and service providers.  The Task Force is making progress.

At this point we have made a preliminary identification of the specific areas to be covered by the guidelines.  Once the coverage is determined, we will move to develop the actual guidelines.  The Task Force will be looking at this in February and March.  

We welcome your feedback as to what should be included in the guidelines. What questions do you have when you are presented with efficacy research from educational product providers?  Please email Hilary Rickert at hrickert@segmeasurement.com with your thoughts and input on the efficacy research guidelines.

Jane Loevinger: Unsung leader in Test Validity Research

Anyone who has studied Measurement is very familiar with the leaders in the area of test validity--Thorndike, Cronbach and Messick, to name a few, that have contributed to our understanding of this subject.  But, few of us recognize the name Jane Loevinger, despite her superior work in this area.     

 

Dr. Loevinger's 1957 59 page monograph "Objective Tests as Instruments of Psychological Theory" focusing on test validity provides insightful information.   Her extension of the classical validity argument anticipates our more recent focus on validity claims and an evidential approach to validity. You can find Loevinger's complete Monograph here.   

 

Jane Loevinger (1918-2008) was academically strong at an early age.  Despite being counseled, like many women of the time, not to pursue a career in Psychology, because it was "too mathematical" went on to graduate Magna Cum Laude in Psychology at age 19, and then went on to achieve her Master's Degree in Psychometrics from the University of Minnesota in 1939, and a PhD from UC Berkeley in 1944.  Her dissertation, completed under Erik Erikson, criticized existing Psychometric Theories.  After receiving her degree, she moved to Los Alamos to work on the Atomic Bomb Project, but found that job opportunities for women after the war were limited.   After her work at Los Alamos, she took on  part time teaching and research positions at Washington University in Saint Louis, but was not accepted as a full-time professor until 1973 (emeritus as of 1988).    

Technical Corner

Aligning Your Goals With The Effectiveness Research Design
 

About SEG Measurement

 

 SEG At Upcoming Conferences

Let's Meet!