A guide to user testing for mobile apps

Every mobile team has experienced it at some point. The analytics show a problem. The development team has theories. But sprint planning is hard, because nobody actually knows what the cause is.

That's where user testing fits. Let’s look at an example.

The designer of a survival shooter game knows a shotgun is supposed to be a close-range weapon. They never get frustrated when it does minimal damage at long distance. They built it that way. But new users don't know that.

This is the central problem in building any mobile experience: the person with the most knowledge of how your app works can’t experience it the way a new user will. 

You can't unsee what you built. You can't not-know what you know. Every blind spot in your design lives in the gap between your understanding of the app and a first-time user's first touchpoint with it.

User testing is how you close that gap.

The thing that surprises publishers that are new to user testing isn't what users struggle with. It's what users struggle with silently. When things are broken, users generally let you know. When things are just… not excellent, you need to do more work to find out.

That’s because when onboarding is unclear, users almost never say so. They assume they've missed something. They blame themselves. They close the app without filing a report or leaving a review. 

User testing is, at its simplest, watching real users use your product before you've decided you understand them. Moderated sessions let a researcher observe in real time and probe what they see—asking what a user expected to happen, following unexpected behaviour as it unfolds. 

Unmoderated remote testing has participants complete tasks independently while recording their screen, which allows for larger samples. Some researchers also incorporate biometric signals like facial expressions or eye movements, which can surface moments where what a user does and what they report feeling are pointing in different directions.

The right method depends on the question. All of them are asking the same thing: what does a user actually experience the first time they download your app?

There was a playtest of a well known game where a player said, after finishing a number of sessions: “There’s nothing to do”. He had been playing for three hundred hours! 

What that player reported and what his behaviour showed were completely different signals—and both were true. He clearly, in the moment, had been engaged enough to keep coming back for three hundred hours. His survey response had nothing to do with that. The gap between those two things wasn't a contradiction to resolve. It was an opportunity to investigate. The core loop clearly was engaging enough to encourage repeat play, but users who finish a session without a clear ‘to do’ for their next session are much less likely to return, and much less likely to report motivation to return. 

That's where user testing fits alongside the tools most mobile teams already use. Analytics show you what happened, such as where players left, how a product change moved a curve, or what a retained user is worth. User testing shows you what players were experiencing when they made those decisions. The two sit inside the same optimization loop: analytics identify the friction point, user testing investigates the experience behind it, the team makes a change, analytics measure whether it moved.

You don't need to choose between them. They're answering different questions.

Here's something that gets designed by nobody and noticed by nobody until a user sits down to experience it for the first time: the passive chain. A user finishes a lesson in a language app. Victory screen. Reward animation. Loading screen. Interstitial ad. The next series of lessons load. 

Each of those elements was designed individually. Each was tested individually. They’re all the best possible version of that element that the studio can produce. But not many studios test them as a combination.

Nobody sat down and said "let's create a forty-five second passive experience." It happened because each piece made sense on its own. Users experience it as the immersion stopping. A portion of those users will close the app during a passive moment the design team never considered as an exit. It shows up in session data as drop-off with no obvious cause.

This is what user testing consistently surfaces: the mechanism behind a retention problem is almost never what the team assumed. Users ignore tutorial text placed immediately after a reward, the reward has their full attention, and anything communicated in that window disappears. 

Players become emotionally disengaged well before they behaviourally churn—they keep tapping, but the quality of attention has already left. From our studies, when three or more prompts appear in sequence, the first receives around five to six seconds of genuine attention; by the third, that's down to roughly one second. Users aren't evaluating the offer any more. 

Instead of giving users more chances to convert, you’ve trained your users to dismiss your prompts, and it shows up in conversion data as poor performance with no explanation.

These money moments are tricky to spot in a dashboard. But they jump out at you when you watch.

One thing that’s common when you show a team what users actually experienced after a playtest: the response is almost never "we disagree." It's almost always "we had no idea."

The same principle applies inside the game. Peaksel, a mobile puzzle studio, had measurable drop-off before level 3 of their game. Their analytics showed exactly where players were leaving. A user test found the mechanism: the tutorial had taught a linear sequence, but the live game immediately required non-linear exploration. Players weren't failing the puzzles. They were failing to understand the rules of the world. Two levels were redesigned. Level 3 completion rose 9.6%. Lifetime value rose 14.5%.

Six participants. Two redesigned levels. The commercial validation happened in the live data afterward, but the mechanism that made the fix possible came from watching.

John Hopson, VP of Research at Emhance, learned something during user testing on Halo that is illustrative of how to think about teaching players about your app.

The right approach isn't to run through everything once in an onboarding and move on. It's to teach it, confirm the user has acted on it, wait, and resurface a reminder if they haven't used it after a reasonable interval. 

Not because users are forgetful, but because learning only happens in context. You need to know what something does before you have a reason to understand it. Teaching too early is the same as not teaching.

The same logic applies to user testing itself. A comprehensive study before launch is useful. More useful is to run focused tests frequently. Watching five to eight participants navigate a specific flow will surface more than most teams expect. The goal isn't exhaustive coverage in one test. It's building the infrastructure to answer a specific question your measurement has raised.

Across mobile, around 97% of users churn within the first thirty days. Of those who stay, roughly 5% will never make a purchase.

The math is unforgiving. But more users surviving means larger cohorts and means more users reaching the moments—a first purchase trigger, a meaningful engagement hook—where revenue is actually generated. 

A 10-point improvement in day 30 retention turns an underperforming app into a unicorn, without touching the monetization mechanics at all. It’s the same user profile, spending the same amount, just more of them surviving long enough to do it.

User testing doesn't move those numbers on its own. What it does is find what's limiting them, specifically, in the moments that matter most.

The job of a user researcher is to make experiences more enjoyable. Not to have opinions about app design or onboarding flow decisions, that's the developer's job, and they have fantastic experience and analytics to do that excellently. The job of user research is to bring evidence into the room that the team couldn't have gathered themselves, and then get out of the way while they decide what to do with it.

For more information about user testing, see Emhance. To see first-hand how Adjust can grow your app business, request a demo today.