Debuting a new EdTech system is an exciting and stressful time for most school administrators. At first, you can exhale because the new system is finally here and your teachers and staff are fully onboarded to the tools and apps.
However, your work is just beginning. As an administrator, you are responsible for proving that your EdTech investment is as valuable as you hoped it would be. You need to make sure students are benefiting from these systems and teachers are using them effectively. This requires a mixture of qualitative and quantitative results tracking – especially if you need to report the impact of the system to the district at the end of the semester.
Start by developing a plan to track student learning outcomes. You can establish a few key metrics and processes that can guide your evaluation to help you understand how the technology improves your school. Use this guide to develop a results tracking plan.
Track Key Performance Indicators Over Time
Key performance indicators (KPIs) refer to important metrics that reflect the use of processes and programs. As an administrator, you can establish your own set of KPIs when looking for an EdTech system or you can ask software providers for a list of KPIs that their systems are supposed to target.
One example of a target KPI might be the average student reading level in each class. Let’s say an EdTech system will use games and activities to teach vocabulary, grammar, sentence structure, and the elements of a story. You would track its effectiveness by looking at the average reading level of different classes and students. Your goal is to see students who enter fourth grade below their target reading level catch up and become confident readers by the end of the year.
Choose a set of KPIs to track throughout the year and set windows to evaluate them. You might check in monthly or every other month depending on your goals. Be careful when you set KPIs to avoid vanity metrics. These are indicators that only tell you good things and don’t showcase the quality of the software used. A vanity metric might include the number of total logins by teachers by month. This only shows that your teachers have used the software, not necessarily that it’s actually useful.
Observe How Teachers Use the Product
Your KPIs will help you track student learning outcomes as a result of your new EdTech tools, but qualitative observations will tell you if teachers are using the systems correctly and effectively. While the educators in your school might have received training sessions on the systems, they might have forgotten about key features. They might also ignore the system if they think it’s ineffective.
For example, it is commonly assumed that most EdTech systems are games that are used as rewards for students. If teachers can get through the education part of the lesson plan, the students will be rewarded with the apps or quests. EdTech apps are often sold as reinforcement tools, reducing the amount of information that students forget. However, effective EdTech technology should be able to introduce new material to students and challenge them to learn. It should serve as an assistive device for teachers, not just a reward or distraction on a rainy day.
Spend time in various classrooms observing how teachers use the technology you are providing them. If you notice a pattern, you may need to provide additional training opportunities so educators can get the most out of these systems.
Make Sure Students Can Fully Master Concepts
There’s a difference between completing a task and mastering a concept. Oftentimes, students can complete tasks in EdTech systems by following basic directions or playing “process of elimination” games. However, this doesn’t mean they have mastery. When a student really masters a concept, they can apply it in a variety of different situations. They know when it should be used and how to apply it effectively.
Your EdTech systems should be able to help students master concepts. Your teachers should also be able to evaluate students outside of the technology to make sure they fully understand the material.
Students learning at a mastery level is more important than justifying your EdTech investment. So many core concepts taught to young learners are used in higher grades and in more advanced subjects. They are used in real life and in the careers of your future students. You need the students in your school to master fractions, comma use, how government works, and the systems of the body if they are going to be successful learners and well-rounded adults.
Gather Teacher, Students, and Parent Opinions
Finally, spend some time actually talking with your team members to learn what they like and dislike about the systems you provide. You can set up quick student interviews in the classroom or request 10-minute meetings with some of the teachers on staff. Some schools also use feedback forms to collect ideas and insights – which is useful for parents who want to share their experiences but work during the day.
The best software systems will adapt to the needs of your teachers and students. At eSpark, we request student feedback after every activity in the form of a thumbs-up or thumbs-down. If any game or challenge drops below an 80% thumbs-up rate, we pull it from the system and make improvements to appeal to students.
The best way to solicit feedback is to let your teachers know that you are always available to listen. If your educators feel like they can share honest experiences and ideas for improvement, they won’t be afraid to knock on your door.
Beware of Goodhart’s Law
Before you can launch your plan to track student learning outcomes in your new EdTech system, heed a word of caution against obsessing over data points. Goodhart’s Law is an economic concept that boils down to, “when a measure becomes a target, it ceases to be a good measure.” It essentially tells people to be careful what data points and targets they wish for, as they might have unintended consequences.
Goodhart’s Law is often cited when teachers are pushed to focus on standardized testing and have to “teach to the test,” almost exclusively in their curricula. While some schools may increase their test scores, students also become disengaged with learning and teachers have to remove fun or useful lesson plans that inspire passion but don’t directly relate to standardized tests.
If you are trying to evaluate a new EdTech system, you can avoid the consequences of Goodhart’s Law by evaluating the tools with an open mind. Track progress with the goal of seeing if students improve, not to push them to do better. Collect feedback from teachers, parents, and students to see what they like and don’t like about the system. Look to evaluate effectiveness beyond tracking a few KPIs.
It’s Worth Your Time to Develop a Strong Evaluation Process
Launching a school-wide EdTech system can be an exhausting process. All of your staff members need to be experts in the technology and should be able to use it effectively and share it with the students. However, using an EdTech system doesn’t automatically mean it’s useful for your school. Take time to evaluate the software and ensure it’s being used effectively and making an impact on your students. You need to know that your tools drive results beyond hitting vanity numbers or irrelevant benchmarks.
Comments