User Testing with Kids: Lessons from the Field

Successful user testing requires a delicate balance of focus and openness – that was one of our greatest takeaways from the classes we took last year at the d.School (d.Bootcamp, d.Media, and the LDT d.Seminar, all awesome). Throughout the process of creating Motion Math, our learning game for the iPhone, iPad, and iPod Touch, we’ve user tested our game with kids with a narrow set of questions in mind while trying to maintain a beginner’s mind, open to all their explorations. What a user does with your product is more important than what they say; this is especially true with kids, who are not always able to articulate their experiences.  With Motion Math’s release this week on the App Store (http://bit.ly/motionmath), we wanted to share five key take-aways from dozens of rounds of user testing with kids aged 6-14 in homes and schools, starting with the humble paper prototype shown here.

1. KIDS GRASP TECHNOLOGY
One danger with creating educational games is that we all have some common sense about how to teach; this intuition can blind us to the ways that kids (and especially today’s digital natives) are different. To take one striking example, many young kids now expect all screens to be touchscreens; we’ve heard from many parents that their young children now pinch and tap the TV!

Adults are more articulate about what they want in an interface, but often struggle in ways that kids never do; we’ve learned to be wary of adding features that adults ask for.  A great example is text instructions – kids get the symbolic hints whereas adults often prefer explicit language.

Also, kids are fantastic bug finders! If you want your game to be played upside down, sideways, and in ways you never intended, give it to a child. We encourage this creative testing by asking students to find bugs. In the rare ;-) cases that they do, we reward them with a collection of hand-picked bug stickers.  It’s charming to watch them wear around a sparkly bug sticker on their shirts as a badge of their technical prowess.

2. THE DEEPER YOU LOOK, THE MORE COMPLEX YOUR SUBJECT MATTER
We decided to tackle fractions in our first game: we knew it was notoriously difficult for many learners, but didn’t comprehend its complexity before talking to kids. The fact that 1/2, 50%, 0.5, and 3/6 all mean the same thing is odd; at the same time “1/2” is very different from “1/2 of 2.”  Fractions can mean a distance on a number line, selecting parts of a group, parts of a whole, breaking something into parts, and many other physical metaphors, and competence in one metaphor doesn’t necessarily mean understanding (or even awareness) of another. For example, when first exposed to fractions in the classroom, kids don’t necessarily come home and start converting measurements in the kitchen. Games are unique device to show patterns among concepts because the game environment can be constructed in a way that surfaces these relationships.  As we start on new math topics, we’ll assume there are more wrinkles in the idea than we expect; the challenge is to surface this complexity and demonstrate patterns while preventing beginners from being overwhelmed.

3. SHOW, DON’T INSTRUCT (AND ITERATE!)

The part of our game that has undergone the most profound change is the introduction.  Even when testing our original paper prototype, we realized our unique game interaction would require some explaining: our game doesn’t rely on the more common interaction of tapping, but rather tilting.  In addition to novel gameplay, we also introduce a new form of educational content: placing a bouncing fraction-star on a number line. It’s potentially overwhelming.

Initially, we wrote carefully-crafted instructions: “Move the fraction star to the right place on the number line by tilting.”  We added italics on the last word to really drive home the game mechanic, and we were pretty proud of ourselves. One day of classroom testing proved we had perhaps the most useless sentence ever written. Most kids didn’t read the instructions;  even those that did forgot them as soon as they moved to the next screen.  Words failed.  After many attempts and revisions, we settled on the current version: we introduce tilting by putting the player on an empty screen with nothing but a bouncing star and targets to land on. Some adults get confused, but 95% of kids immediately explore the device and figure it out.  Only after the game interaction has been mastered do we introduce the idea of a star holding a fraction.  We find some validation for this approach from education pioneer Maria Montessori, whose “auto-educating apparatuses” were designed to allow children to individually explore concepts deeply by leveraging their senses. “This makes it possible for the child to work by himself, and to accomplish a genuine sensory auto-education, in the visual perception of form,” she wrote. Our users are now able to understand the application controls without verbal clarifications.

4. MULTIPLY THE MOTIVATION

One second grader struggled to make it past the first level, and we expected her to stop playing. Instead, she bragged to her friend: “I got 6 stars!” and immediately played again. Minor bits of motivation can maintain engagement, so we kept adding different forms of feedback:  progress stars for each correctly solved problem, a cumulative score, new level indicators, bonus effects, and a game-over indication of overall progress – presented as a fraction, naturally. Different levels of positive feedback motivate learners at different skill levels.  At the same time, we didn’t want to create a game that overly congratulated our players or made victory too easy: that’s why we’ve made the last few of our 24 levels quite difficult.

5. DOUBT YOUR OWN AWESOMENESS
After working for many months, our natural bias was to love our game, and it was easy to go into user testing inclined to hear affirmation.  When we go into classrooms and the teacher announces that we are App developers testing a new game, the kids get very excited and put themselves on best behavior. (The Hawthorne effect is the psychology term that describes this phenomena.)  In general, we must remain skeptical about our customers’ reactions to the game despite their praise and enthusiasm.  More telling are the subtle indications of their liking, such as picking to play our game again rather than other iPhone games on the same screen. When a kid presses “Play Again” without asking for permission, they are showing real engagement.  They have found the game easy enough to begin, but they also believe that they could improve over their last performance, and when they play again over and over they are in a state of flow: continuous engagement.  This is an ideal state for maximal learning and enjoyment.

Just hanging out with children reminds us of our important role in maintaining their excitement about learning. With focused observation and questioning, and letting go of our own biases, user testing can provide the feedback and inspiration to create great learning products.

Gabriel Adauto and Jacob Klein are 2010 graduates from the Learning, Design, and Technology program at the Stanford School of Education and Co-Founders of Motion Math. Their first game, a bouncing star fractions game for the iPhone, iPod Touch, and iPad, was a finalist for the Sesame Street Workshop Joan Ganz Cooney Center’s Innovations in Mobile Learning Prize. Motion Math can be purchased here for an introductory price of $.99. It was featured in the Wall Street Journal’s Tech blog.

One Response

  1. Rick Carlile
    Rick Carlile March 26, 2011 at 6:41 am |

    Awesome post. I wish there was some way to force published/developers of *other* games, or any software for that matter, to read your post.

    Using kids to QA pretty much any software title would probably change the landscape of development in les than a year, and perhaps save millions of users the heaches associated with the myriad of horrendous titles released every other day.

Comments are closed.