SEGway October 2016

2013 Logo 3d gradient
SEGway     October 2016
Research and Assessment News from SEG Measurement  
 
In This Issue
 

Follow us on Twitter

 

View our profile on LinkedIn

 


31 Pheasant Run
New Hope Pennsylvania 18938
 
800.254.7670
 
 267.759.0617

Greetings!
 
We are often challenged when it comes to interpreting government regulations. In this SEGway issue, we offer assistance with unraveling the Every Student Succeeds Act (ESSA).  What evidence is needed to enable schools to purchase products with ESSA funding?  Can older efficacy studies be used?  Is a case study valuable?

Recently we attended the EdNET and WIDA conferences.  It is always enjoyable to get together with folks and meet new colleagues while learning about updates in the education, technology and publishing spheres.  We savor the collective energy and interest generated at these conferences, and look forward to working with many of you. 
 
We encourage you to learn more about our work in assessment and efficacy research by connecting with us at conferences. We also look forward to continuing the discussion with you through the newsletter, on Twitter(@segmeasure) and on LinkedIn (SEG Measurement).  
 
Take a look at our website at www.segmeasurement.com, as it is frequently updated with developments in the field.  And, feel free to email me at selliot@segmeasurement.com. I always look forward to hearing from you.
 
Sincerely,
 
Scott Signature
 
 

Scott Elliot
SEG Measurement 

 
Understanding ESSA
How Much Evidence Do You Need?
 
The Every Student Succeeds Act (ESSA) reauthorizes the 50-year-old Elementary and Secondary Education Act (ESEA) and the No Child Left Behind Act (NCLB).

The term "evidence-based" appears on almost 60 occasions throughout ESSA. This term is used to govern the use of funds and selection of activities and interventions throughout nearly all the major programs of ESEA, including professional development, school improvement, literacy activities and Title I and Title II interventions.  

ESSA is the first federal education law to define the term "evidence-based."Administrators and teachers are instructed by ESSA to purchase products that are proven to be effective, in order to receive state, local and federal funding which pays for a lot of education products, even at the district level. 
 
ESSA is defining the following levels of evidence from education publishers and providers:
 
Level 4 - Strong Evidence. To be supported by strong evidence, there must be at least one well-designed and well-implemented experimental study with a large representative sample across multiple sites.
 
Level 3 - Moderate Evidence. To be supported by moderate evidence, there must be at least one well-designed and well-implemented quasi-experimental study with a large representative sample across multiple sites.
 
Level 2 - Promising Evidence. To be supported by promising evidence, there must be at least one well-designed and well-implemented correlational study with statistical controls for selection bias or differences between the intervention and comparison groups.
 
Level 1 - Demonstrates a Rationale. To demonstrate a rationale, the intervention should include a well-specified logic model that is informed by research or an evaluation that suggests how the intervention is likely to improve relevant outcomes; and an effort to study the effects of the intervention, ideally producing promising evidence of the expected outcomes.
 
SEG Measurement offers a range of efficacy and effectiveness research services to support educational publishers and other educational service providers for use in obtaining proof that their products and services are effective, and to be "ESSA ready." These services include collecting evidence at all of these levels through case studies, white papers, qualitative studies, quasi-experimental studies, and experimental studies.
We can help to evaluate your previously conducted research, determine the types of evidence you may have already collected, and work with you to develop a plan to achieve the highest level of evidence for your products. Once the plan is in place, we will execute the study or series of studies in the agreed upon time frame and budget.

Call or email us today at 800-254-7670 ext. 102 or hrickert@segmeasurement.com for a cost-effective efficacy study by SEG Measurement, and be ESSA ready!

Visit our website for more details at: www.segmeasurement.com.
 

Research Success Story:  EdTechLens
 
During spring 2016, we conducted a third-party evaluation of Rainforest Journey, an e-learning life science program from EdTechLens.
 
Our research was focused on learning more about whether 4th and 5th grade students who used Rainforest Journey showed greater gains in life science knowledge and skills than their peers, whether they showed greater engagement or interest in science after completing life science instruction than their peers, and whether the teachers found Rainforest Journey to be an effective resource for teaching life science. We also wanted to measure/gauge the program's effectiveness in literacy support.
 
We worked with EdTechLens to create the design, the assessments, and the surveys that would be used as part of the study. EdTechLens helped to identify the participants and provided typical training and support for the teachers. The teachers used EdTechLens as prescribed between January and June 2016.
 
The results of this research provided evidence of the effectiveness of Rainforest Journey across multiple facets including cognitive and affective outcomes. These findings were especially commendable since the study was conducted on the first full production release of the product. In addition to evaluating the effectiveness, the study helped to collect a deeper understanding of how teachers were implementing the program, the practices that were most successful, and ideas that could be used to inform new releases. We also found that the program supports learning goals in two subject areas - life science and literacy.
 
"Literacy rocks! RF Journey efficacy report shows we are not just science - 100% of teachers agree on quality of written content and that program works well for reading informational text."- Ellen Senisi, Founder, EdTechLens
We provided a full technical report at the completion of the study. EdTechLens used that information to create a short research brief to share with potential customers. In addition to making this research available on their website, they are spreading the word through social media and other channels.
 
EdTechLens has indicated that they have already experienced increased sales as a result of these efforts. We are very honored to have been selected to complete this research for EdTechLens, to help them get the most out of their research budget, and to be a part of their continued success.
 
For more information regarding Rainforest Journey, please visithttps://www.edtechlens.com/rainforest-journey.

For more information about conducting a study, please call or email us today at 800-254-7670 ext. 102 or hrickert@segmeasurement.com.

 

 
Recent Questions:  Propensity Score Matching  
student-participants.jpg
For this installment of frequently asked questions, we are focusing on Propensity Score Matching (PSM).  If you have specific questions that are not covered here or if you are interested in exploring any of these questions further, please contact us.  We are here to help.
What is Propensity Score Matching?
Propensity Score Matching (PSM) is a statistical technique that researchers can use to match treatment and control group participants to create balanced groups for comparison. Since randomly assigning individual students to treatment and control groups is typically not possible in a classroom/school setting, propensity score matching is a sophistical approach that comes close to having randomly assigned the students to the groups. After the full treatment and control groups are established, PSM is used to pick matched sets of treatment and control students. PSM helps to ensure that the students in the groups are as similar as possible, which provides us more confidence that when we see differences at the end of the study that we are focusing on differences related to the treatment rather than other confounding variables. There are multiple approaches to PSM including the required similarity of the matched students, the variables included in the model to match students, and whether there is one-to-one matching or one-to-many matching allowed.
 
What kinds of factors are used in the matching?
Selecting the variables to include for matching is a mixture of art and science and is limited to the availability and accuracy of data. Variables that correspond to state, district, school, teacher, and student levels can be included in the model. The goal is to match students on initial ability, demographics, class and school environment, and any other factor that may potentially be related to whether or not the student happened to be in the treatment or control group. If multiple levels of attributes are not available, PSM can be run on as few as one factor such as initial ability level.
 
How do we know when we should use Propensity Score Matching?
PSM is very helpful in cases in which there are large treatment and control groups and there is room to fine tune the final groups for comparison. It is also appropriate in cases in which the outcome measure of interest is the state test, a treatment group exists (i.e., users of a product), and the possible control group is everyone else in the state. In that example, PSM can be used to create balanced treatment and control groups that are similar on all of the important variables.
 

What questions do you have?  Let us know what questions you want answered.  Contactinfo@segmeasurement.com to share your feedback.

SEG At Upcoming Conferences
Let's Meet!
 
We are looking forward to seeing our colleagues and meeting new friends at the upcoming conferences.  We are participating in several sessions & we invite you to join us.
 
   Look for us at these upcoming conferences:
  • NCTM Regional Conference, October 31 - November 2, Philadelphia, PA
  • NJEA Convention, November 10 - 11, Atlantic City, NJ
  • E-Learn World Conference on E-Learning, November 14 - 16, Washington DC
  • SIIA, December 8- 9, The Princeton Club, NYC
We would love to meet with you and discuss how we can help you build strong assessments and get the proof of effectiveness you need for success.  
 
If you would like to meet with a representative from SEG Measurement to discuss how we might help you with your assessment and research needs, please contact us at info@segmeasurement.com.

About SEG Measurement
Building Better Assessments and Evaluating Product Efficacy
medical-team-portrait.jpg
SEG Measurement conducts technically sound product efficacy research for educational publishers, technology providers, government agencies and other educational organizations, and helps organizations build better assessments. We have been meeting the research and assessment needs of organizations since 1979. SEG Measurement is located in New Hope, Pennsylvania and can be accessed on the web at