Many of you were interested in the research questions we covered in last month's newsletter, so we are answering additional questions in this newsletter.
If you have specific questions that are not covered here or if you are interested in exploring any of these questions further, please contact us. We are here to help.
How do you account for differences in teacher quality?
In most effectiveness studies, there are classes assigned to use the product and classes assigned to not use the product so you can fairly compare. It is well known that teachers have a great impact on the amount of student learning achieved in the classroom throughout a school year. As such, it is important to ensure that the teachers included in an effectiveness study are similar across the classes that are using the product and not using the product. If more experienced teachers are all clustered in the classes that are using the product, it gives those classrooms an unfair advantage. There are a number of things that can be done to help control for this. The first is to randomly assign teachers to treatment and control. If random assignment is not possible, then the control/comparison group teachers should be selected to be as similar to the treatment teachers as possible and well matched statistically. Another approach is to remove or control for the effect of any differences between the teachers using a procedure called Analysis of Covariance.
What units/lessons do we pick to investigate?
For products that range from K-12 and have multiple subject areas, it can be a challenge to determine where to focus for a study. For a full year study, we typically recommend focusing on 1 - 3 grades in 1 or 2 subjects. For a shorter study, it would be helpful to select one grade and a couple of lessons/units/features to focus on in a short period of time. When making decisions on areas of focus, consider your goals for the study. Do you want to explore successes in areas where you have not had much usage to date, or are you interested in studying grades/features/subjects/units where you have the greatest amount of use? Based on your goals, the optimal scope of the study (as well as necessary implementation time frame) can be determined.
We have a product that is completely "free style" and are not sure how to control the usage for the study?
Products that have a very specific curriculum are more straightforward to define the expected usage to be included in the study. For those products that do not have any specific curriculum or recommended use, time or engagement become more critical than the actual features used. A minimum number of logins, resources used, time spent on task, or number of games played are a few examples of how usage can be monitored. When recruiting for a study, it is important to define the minimum level of use requested over the study time frame. It is also important to collect information from participants about how they used the product so that usage information can be shared along with the results. From this, best practices or recommendations for successful implementation can be presented.
How do we know if classes are actually using our product as expected for the study?
For online products, usage is typically tracked through the product itself. On a regular basis, the usage data should be evaluated to determine whether the participants are being compliant in their use of the product. In some cases it may be necessary to remind participants about the minimum level of usage required to participate. For offline products or products that do not have sufficient usage tracking/reporting, it is helpful to have the teachers check-in on a regular basis with us to attest to their usage. We provide an online survey tool for the teachers to confirm their use of certain features or the length of time they used the product in the classroom (based on the specifics of the study). In other cases, we talk with the teachers to confirm their use of the product. The goal here is to be involved enough through online tracking or connecting with teachers to ensure that classes are using the product as expected while also leaving teachers the autonomy for an authentic implementation. It is also important to be flexible with the usage expectations and understand that the same level of usage is not going to be maintained each week throughout the study.
What level of training or support is needed for participants?
For an effectiveness study, it is important to have the conditions for using the product be similar to what would happen under normal conditions. If there is a web-based training webinar for new users, then the users for the study should complete the same online training. If there is in-person training or coaching as part of a typical implementation, then that should be included for study participants as well. If there is no training typically provided, then no training should be provided for the study participants. In terms of technical support, the study participants should receive the same level of support as other users. Technical issues or questions should be addressed in a timely manner.
What questions do you have? Let us know what questions you want answered. Contactinfo@segmeasurement.com to share your feedback.
