Science process/inquiry skills are notoriously difficult to assess in isolation. As a result, we often see assessments that “test” process skills through multiple choice questions: “choose the best hypothesis based on the sample experiment shown above.” Despite the ubiquity of multiple choice, we know that these types of questions are a two-dimensional means of assessing most knowledge, particularly process skills. Even if a student selects the correct answer, he or she may still have difficulty creating his or her own hypothesis if the answer choices were removed. Multiple choice assessments create good test takers but not good real-world problem solvers. Worchester Polytechnic Institute’s Janice D. Gobert, Ph.D., an associate professor of psychology and learning sciences, might have the answer to this dilemma.
Dr. Gobert and her team developed a method to assess middle school students’ process skills using “microworlds,” a virtual lab simulation that uses open-ended response to capture student understanding in a way that multiple choice cannot. Thus far, the team’s results have been promising – initial findings show that students could still demonstrate the skills they acquired six months prior.
The crux of Gobert’s work rests on shifting how we approach concept of knowledge: rather than using a top-down approach (also known as knowledge engineering), where a predefined pathway is laid out to reach a specific point of knowledge according to pedagogical theory, Gobert believes that a bottom-up approach using educational data mining and machine learning is more effective. This approach aims to find the best way to learn a topic, based on how thousands of students interact with the concept in the microworld.

One challenge is the manual input required by this approach, which can complicate scalability. Each student’s open-ended responses need to be hand scored and coded and then added to a database in order to increase the systems’ accuracy in automatically scoring a student’s process skills knowledge.
The potential for these systems is huge. Several companies and organizations, including the Rice Center for Digital Learning and Scholarship, are diligently working to make such systems a reality, but approaches differ. Dr. Richard Baraniuk, Victor E. Cameron Professor of electrical engineering and computer engineering at Rice University, argues that hand coding is not a feasible option for making large scale machine learning platforms. The amount of data processed by these systems is immense – having slews of programmers grading and hand coding the relationships between the data sets would make any resulting product prohibitively expensive.
Still, the power of machine learning systems might justify the expense. Teachers need not fear though: even if large, robust machine learning systems were available for your students tomorrow, the power of hands-on problem based learning, authentic teacher interventions, and full scale science projects would not disappear. In fact, learning experiences such as these may amplify the teacher's role, as increasing granular data may show the exact nature of a student's misconception, giving teachers more clarity on how to offer support. In this way, harnessing the power of machine learning might be more analogous to a stethoscope than penicillin – we’ll be able to better diagnose and then act on problems, rather than simply cure them in one sweep.