September 2017 SEGway

2013 Logo 3d gradient
SEGway               September 2017
Research and Assessment News from SEG Measurement   
In This Issue

Follow us on Twitter


View our profile on LinkedIn


31 Pheasant Run
New Hope Pennsylvania 18938
It has been interesting to watch the increasing attention to product effectiveness research over my long career (40 years).  While most organizations now recognize the importance of proof of effectiveness, the passage of ESSA has certainly sealed the deal. Schools demand it, and vendors know they can't sell without it.
ESSA (and its predecessors to a lesser extend) require that products funded with federal money be supported by scientifically based research.  But, the reasons for conducting product extend beyond codified law.

First, it's the right thing to do. Our children deserve to receive the highest quality instruction supported by educational products that work. This lies at the heart of SEG Measurement's mission (and is reflected in our mission statement.)

Second, we live in a consumer culture that "looks for" proof.  As consumers we want to make sure that we are buying products that work.  We are highly influenced by research evidence when we make purchases.  Whether searching for a microwave oven or a home service, we often seek out evidence that we are making the right choice.

Third, schools want to make sure they are spending their limited resources wisely.  Schools have limited resources to spend and they want to get the most "bang for the buck".  Naturally, they look to proof before buying.

Fourth, companies want every advantage to sell their products and services. Simply put, sound, scientific evidence of effectiveness from a credible, third-party research firm sells. Companies recognize that third party evidence of effectiveness wins deals.

Whatever your reasons for wanting to demonstrate educational product effectiveness, we want to help.  Let SEG Measurement help you conduct that research in a technically sound, credible and cost-effective manner. 
Ta ke a look at our website at , as it is frequently updated with developments in the field, and  email me at  with any questions about item banks, assessments, or how to create your own marketing campaign with an effectiveness study . I always look forward to hearing from you.
Scott Signature

Scott Elliot 
SEG Measurement 

Questions Most Often Asked by Educators Considering Conducting an Efficacy Study
What educators ask most often about effectiveness research


Over the past decade, we have had the opportunity to visit with literally thousands of educators across the country.  Not surprisingly, we have found tremendous interest in effectiveness research and evidence-based educational practice.  Among the many questions we are asked, some questions are by far the most common. Here are the most common questions we hear.

 Why do I need a formal study; all of our customers tell us our product/service is effective?-


Positive comments and testimonials are useful, but this does not constitute "scientific evidence."  A small set of client opinions is not considered "scientific evidence."  Scientifically acceptable proof of effectiveness requires a sound research design accepted by the scientific community.  School buyers and government funding agencies want to see a scientific study supporting the effectiveness of your product or service before buying or providing funds for purchase.  Buyers of school products and services are savvier and are no longer willing to settle for products and services backed by unsubstantiated claims and testimonials.  They are looking for hard evidence that will give them confidence in solutions before buying. Many communities, still strapped for cash based on the recent economy, are shopping "smarter," by limiting purchases to those with research-based efficacy support.

SEG can help you work through an effectiveness study design that makes sense for your product or service. 
  A strong efficacy study typically includes a scientific sample of users (treatment group) that are tested using reliable outcome measures both before using the product and after using the product.  And, it is best if we compare the results of those users to a comparison group (control group) that is not using the product. Proper statistical techniques need to be applied to evaluate whether the results obtained are greater than would be expected by chance alone.  Scientific effectiveness studies vary in scope (big-small, long-short, etc.) but tend to share these common features.  

When should I begin planning for an efficacy study?- 

Often we hear from educational providers that they are ready to conduct a study and wish to start as soon as possible.  While we very much want to support everyone and admire their drive, careful planning and preparation is critical for effectiveness study success.  We encourage product and service providers to begin planning about 6 months before executing a study, or at least 3 months before starting the study.

Planning and preparation include, first and foremost, designing the study.  Second, we also encourage clients to begin preliminary identification and recruitment of the sample 3-6 months ahead of time.  Third, those conducting an effectiveness study should leave plenty of time for the selection or identification of appropriate measures for the study.  While planning involves much more than this, these three upfront activities will dramatically influence the success of the study.

Thinking about conducting an effectiveness study? SEG can help you begin planning to maximize your success.

What questions do you have?  Let us know what questions you want answered.  Contact hrickert@segmeasurement to share your feedback. 

Technical Corner 
Aligning Your Goals With The Effectiveness Research Design 

With all that has been written about research designs and statistical analyses, you might be surprised to learn that the most common problem we see is a failure to make sure that the research design components are aligned with the goals of the research. 

There is no substitute for careful planning of an effectiveness study.  The first component of proper research design planning is to have a clear sense of your goals and a clear sense of what you are trying to measure.   Ask yourself, what is my product designed to achieve? If your product is designed to improve student learning, then that is what you should measure--student learning; you would certainly not want to measure student attitudes toward school or how teachers feel about the product.  Be clear on the desired outcomes, document it clearly and succinctly, and make sure you are measuring the desired outcomes.

Second, you should be clear on your purpose for the research.  Why are you conducting the study? Your choices of design, measurement, analysis, etc. will be very different if your purpose is to collect formative information for ongoing improvement of the product than if your purpose is to prove that your product is effective by looking at summative outcomes. For example, you would not want to solely conduct a posttest at the conclusion of the study, if you are looking for formative information; ongoing observation, surveys or interviews might be more aligned to your desire for ongoing improvement information. Again, state your purpose clearly and succinctly, and make sure your study is aligned with your goals.

There is far more to aligning the research design with your goals. To see how others have aligned their goals with the research questions and design, check out SEG's study summaries here on our website.

SEG Measurement can help you design your efficacy research study to maximize success.  Call us today to find out how we can help you with your assessment and research needs.


With nearly 40 years of experience in research, we know what it takes to conduct sound efficacy research.    Call or email us today at 800-254-7670 ext. 102 or for a cost-effective efficacy study by SEG Measurement.

SEG At Upcoming Conference s
Let's Meet!
We are looking forward to seeing our colleagues and meeting new friends at the upcoming conferences.  We are participating in several sessions & we invite you to join us. 
 Look for us at these upcoming conferences:
  • EdNET Education Networking Conference, September 17 - 19, Scottsdale, Arizona
  • Advancing Improvement in Education Conference (AIE), October 4 - 6, San Antonio, TX
  • EDUCAUSE 2017 Annual Conference, October 31 - November 3, 2017, Philadelphia, PA
We would love to meet with you and discuss how we can help you build strong assessments and get the proof of effectiveness you need for success.  
If you would like to meet with a representative from SEG Measurement to discuss how we might help you with your assessment and research needs, please contact us at

About SEG Measurement 
Building Better Assessments and Evaluating Product Efficacy
SEG Measurement conducts technically sound product efficacy research for educational p ublishers, technology providers, government agencies and other educational organizations, and helps organizations build better assessments. We have been meeting the research and assessment needs of organizations since 1979. SEG Measurement is located in New Hope, Pennsylvania and can be accessed on the web at