BrainSpark Assumption Testing

Assumption Test 1: Narrative-based learning is effective in improving students’ accuracy and speed with assignments related to the material.

Full Notes + Process: https://docs.google.com/document/d/1PbL8gO0BCHQK8qIGAhzVtqVVMIIyqYwVLhYUeufwCG4/edit?usp=sharing

Notes:

After conducting the split test, we observed some differences in the performance of the two groups, though the variations were modest, suggesting a slight impact of instructional methods on comprehension and application of the product rule in derivatives.

• Narrative-Based Explanation Group (n=4):

Average Score: 4.125 out of 5 Completion Time: 8.45 minutes on average

• Expository Explanation Group (n=3):

Average Score: 3.625 out of 5 Completion Time: 8.95 minutes on average Differences Mean Score: 0.5 points higher for the narrative-based group. Completion Time: 0.5 minutes faster on average for the narrative-based group.

The results indicate that a narrative-based instructional approach may offer a slight advantage in comprehension and retention of the product rule in derivatives. However, due to the small sample size, these findings are preliminary and should be interpreted with caution. Further research with a larger sample size could help clarify these effects and better assess the potential benefits of narrative-based learning for mathematical concepts.

 

Assumption Test 2: STEM TAs would be interested in using BrainSpark as a supplementary tool for their course

Full Notes + Process: https://docs.google.com/document/d/1R-P5RZehjXT1GL9bWLUgi-sMeueuTeBhgssukgM0HRM/edit?usp=sharing

Notes:

STEM TAs not only liked the personalized and dynamic aspect of BrainSpark Games, but also thought that this was a very creative way to keep students engaged. Therefore, I have concluded that these 3 TAs would be interested in implementing BrainSpark in their curriculum, with an average score of 6/10. Nevertheless, this is an extremely small sample to draw significant conclusions with, so it would be valuable to continue surveying STEM TAs and maybe even professors. 

Nevertheless, ⅔ TAs pushed back and mentioned that time was their most significant constraint. They mentioned that students are already overwhelmed with work, and assigning additional assignments for them to complete could overload them. However, we can solve this issue by making BrainSpark an additional tool that TAs can use if they have time, or they can choose which module and lessons they want to assign for students to complete, which is a key learning from this assumption testing. 

Assumption Test 3: Using an RPG-style storyline can make learning STEM concepts engaging and enjoyable for students

Full Notes + Process: https://docs.google.com/document/d/1874ZRHkY-q2avPwfELuwvF1KnARUV0cGFKPhJi2mpNQ/edit?tab=t.0

Notes:

Testee 1: DD

• Excited and engaged by the story; loved the RPG elements and enjoyed the dialogue with the sage. 
• Would pretend to be a character when answering questions, used a “character voice”
• Did a dance once all three questions were answered, very enthused with the storyline
• Although interested when listening to the question description/answering, would go through answering the questions with a serious expression. 
• Got all three questions right, so did not feel the need to “skip” or “reduce” math tasks without stalling story progression.
• Wished there was more of a payoff with questions (rather than just a comment)

 

Testee 2: AL

• Testee was under stress as they had a pset due at midnight, so wanted to speed up storyline dialogue to get to the problems
• When requesting for a refresher on trigonometric functions from the sage, wanted just the information without the theatrics (as they were in a rush)
• Stated they would enjoy/would prefer the full experience if they had more time. 
  • Did note that if this were “cram time” during an exam, would close out the game and go towards finding just the information without any fluff for the sake of time.
• Laughed and clapped at parts of the storyline. Did seem amused when they allowed it to go on.
• Appreciated the structured math tasks as they progressed, seeing them as a valuable way to apply trigonometry, even if they “weren’t in it for the story.”
• Would likely not choose a storyline-based format for learning under a time crunch but saw the value in it as a novel study approach. In an actual game, would prefer a toggle to skip through dialogue to get to the actual content.
• Said they wished they had more time to be able to see the full storyline

 

Testee 3: CC

• Very enthusiastic about the storyline; immediately drawn into the RPG elements and ready to “save the village.” Would say comments like, “Noo, the villagers!”
• Wanted 
• Found the math mildly annoying and felt it disrupted the story’s pacing. Needed help with some trigonometry concepts (haven’t done so in ~6 years) and didn’t enjoy having to relearn these to move forward in the story. 
• Felt the learning aspect “diluted the fun” of the RPG format and expressed a preference for more storyline as a greater payoff before being asked to do another question.
•  “I loved the game part but didn’t like that I had to do math to continue. Although Ido prefer this to regular learning, in practice I might be annoyed if I want to get to the story but don’t understand a concept/if I just want to learn and I have to get through storylines to do so.”