Everything you need to know about A/B testing for mobile apps
Senior Content Manager
Dec 17, 2019
With a more competitive app market than ever before, learning how to optimize your app – and your marketing campaigns – is crucial. Even a small change in your app’s user experience can have a significant impact on conversion rates, so it’s important to test what works. For example, e-commerce company WallMonkeys increased its conversion rate by 550% using A/B testing tools. This type of testing is an essential practice for all app marketers as it offers clarity on how your app can be optimized. In this guide, we’ll show you everything you need to know about A/B testing, including best practices for optimal results.
What is A/B testing?
A/B testing for mobile apps works by segmenting an audience into two (or more) groups and seeing how a variable affects user behavior. It is used to identify the best possible user experience and deliver the best possible results. For example, let’s say you want to drive installs for your mobile game. As part of your user acquisition strategy, you decide to target young males based in the U.S. with video ads. Instead of throwing money at ads that haven’t been proven to work, it’s smarter to expose your ads to a small group of that audience – and even smarter to A/B test your video ad. In this scenario, you can learn which video ad delivers the best results. If video A had smaller text than video B, and the latter had a 20% higher conversion rate, it makes sense to expose a larger audience to the video with larger text.
When A/B testing, it’s critical to develop a hypothesis before implementing any tests. This will help you improve over time. This common practice gives companies actionable insights that can help them achieve their goals. For example, Sony tested different calls to action on their banner ads. They tested “Make it personal” against “Customize your VAIO,” and found that the latter increased CTRs by 6% and increased shopping cart adds by 21.3%.
What are the benefits of A/B testing?
A/B testing for mobile apps is an industry-wide practice because of the method’s numerous benefits and high confidence levels marketers can have in their analysis. The example above shows that you can discover how to boost conversions in a way that isn’t risking a large portion of your ad spend. However, there are many other benefits to A/B testing. For example, you can:
- Optimize in-app engagements
- Learn what works for different audience groups
- Observe the impact of a new feature
- Gain a better understanding of user behavior
The overall benefit to each of these examples is that A/B testing eliminates guesswork, instead allowing app marketers to rely on data-driven conclusions. This is something you can’t afford to avoid, and the earlier you can begin A/B testing and developing your ongoing hypothesis, the sooner you can ensure your app (and your ads) are in the best possible state.
What A/B testing tools can I use?
Because A/B testing for mobile apps is so important to the development of any app, there are many tools available to app marketers. However, this also makes it harder to choose which tools will help you deliver the best results. Adjust’s Audience Builder is a segmentation tool that is proven to drive growth by A/B testing and retargeting. Using your Adjust data, this tool allows you to immediately define audiences – saving you and your team considerable time and energy.
With our Audience Builder, you can build detailed audience segments that can be sent to your partners in an instant. Once you have created your audience groups for A/B testing, you can send your partners a dynamic URL that contains all the information needed to reach those users. For more information on how Adjust’s Audience Builder can save you time and enable you to A/B test with ease, take a look at the official product page. This will also outline how our Audience Builder allows you to set up retargeting campaigns.
Different types of A/B testing for mobile apps
There are two types of A/B testing that are relevant to app marketers and developers. These both work with the same principle (using comparable audience groups to find a positive variable) but have different functions.
In-app A/B testing
This is how developers can see how changes to your app’s UX and UI impact metrics such as session time, engagements, retention rate, stickiness, and LTV. There will also be specific metrics that will depend on the specific function of your app.
A/B testing for marketing campaigns
For app marketers, A/B testing is a way of optimizing conversion rates, drive installs and successfully retarget users. For example, discovering which ad creative works best for new user acquisition campaigns, or learning which creative makes churned users most likely return.
How to do A/B testing right
A/B testing is a cyclical process that you can use to continually optimize your app and your campaigns. With this in mind, here’s how to do A/B testing right:
Develop a hypothesis
First, you need to research and analyze the information available and develop your hypothesis. Without this, you won’t be able to define which variable to test. For example, your hypothesis could be that having fewer products on show upon opening your e-commerce app will increase session time. This hypothesis, which should be informed by prior research, can then be used to define your variable (the number of products on your homepage.)
A/B testing checklist before implementation:
What do you want to test?
Who is your target audience?
How will you proceed if your hypothesis is proven/disproven?
If you are struggling to define what you’d like to test, start by outlining a problem you’d like to solve. This will give you a good starting point whereby you can define what should be monitored to solve that issue.
Segment your audience
With your hypothesis and variable in place, you’re ready to test these variants on audience samples. Remember that having multiple variables will give you a lower confidence level during your analysis. Put simply, it will be much harder to identify what has influenced your campaign’s performance.
Using an A/B testing tool such as our Audience Builder, you should now segment your audience groups and expose them to versions A and B. You will need an audience size big enough to give you reliable data to analyze. If your audience is too small, you risk misidentifying optimizations for your app that will not have the desired influence on larger audience groups.
You can now determine which variant delivers the best results. Remember to look at every important metric that may have been influenced, because this allows you to learn much more from a single test. For example, even though you’re looking to increase conversions, there may have been an unexpected impact on engagement or session time.
If you have found a positive result, you can confidently expose a larger audience to the successful changes. If your test was inconclusive, this is still useful data that should be used when updating your hypothesis.
Adapt your hypothesis, and repeat
A/B testing enables you to continually develop your hypothesis over time. You should always be testing to learn new ways to boost conversions because there will always be ways to improve. Continue to build your hypothesis on fresh data, and implement new tests to stay ahead of the competition.
5 best practices for A/B testing
Define what you want to test
In the early stages, it’s critical that you know the reason why you are testing a certain variable. Do not start testing before you have a clear hypothesis and know how you will proceed based on different outcomes. This may seem like a simple step, but knowing why you are implementing these tests ensures you aren’t wasting time and money on a test that won’t deliver actionable insights.
Be open to surprises in your analysis
User behavior will always be complex, and that means sometimes your A/B tests will sometimes reveal surprising results. In this scenario, it’s important to be open-minded and follow up on these learnings. Otherwise, you risk leaving money on the table by failing to learn from your own data.
Don’t cut your tests short – even if you aren’t seeing results
A/B tests are valuable even when your hypothesis turns out to be false, or when the result appears to be conclusive very early into the testing period. It’s essential to stick with your tests long enough that you have a high confidence level in the result.
Don’t interrupt your tests with additional changes
Because A/B testing for mobile apps is all about identifying which variables will improve performance, it’s crucial not to make any mid-test changes. This diminishes the confidence you can have in your findings because you will no longer know which changes have produced the desired result. Remember, you are trying to find a cause and effect based on conclusive results.
Regardless of vertical, your results will be subject to the time period in which you’ve tested. You can, therefore, test the same variables in different seasons and find different results. For example, it could be that a particular creative that didn’t perform well in Summer would see impressive results in Winter. This is especially important for verticals such as e-commerce, where users will have clear incentives to behave differently depending on the season.
Learn from your own tests, not just case studies
In his article on A/B testing case studies, Yaniv Navot, Vice President of Marketing at omnichannel personalization platform Designer Yield, claims that “Generalizing any A/B testing result based on just one single case would be considered a false assumption. By doing so, you would ignore your specific vertical space, target audience and brand attributes.” He adds, “Some ideas may work for your site and audience, but most will not replicate that easily.” With so many A/B tests available for marketers to read and learn from, remember that their findings won’t necessarily work for your audience. Instead, the development and testing of your own hypothesis should indicate what gets results.
A/B testing is an essential tool that helps marketers continually improve their campaigns. If you’d like to learn more ways to improve your campaigns, take a look at our guide to social media marketing. We also have resources for new user acquisition and mobile marketing automation.