Blog Gain user trust - and their opt-in: A/B testing best practices and analysis

Gain user trust - and their opt-in: A/B testing best practices and analysis

Change is in the air as Apple’s iOS 14 announcement has the industry working overtime to ensure it’s ready for what’s to come. In preparation for early 2021, when apps will need to request permission to track users, Adjust has been busy researching strategies for successful opt-in prompts and helping clients gear up for the changes.

In case you need a refresher about the privacy policy changes Apple introduced in iOS 14, you can read a summary in our Help Center guide. We also recommend reading our first blog post in this series, which looks at how UX design can help secure user opt-in.

In this article, we’ve put together information on best practices for A/B testing, with suggestions for what to assess in each round and what to look out for in your analysis. To start off, we’ll be looking at user views on data privacy, and how you can use this to inform your opt-in strategy.

User attitudes towards data privacy

Encouragingly, research suggests that many consumers are more comfortable with personalized advertising than initially thought. One study performed by Oxford Economics in 2018 suggests that around 70% of consumers are open to opting-in for a tailored ad experience. The data shows that only a small minority are against a more personalized, targeted experience:

  • Just 17% feel uncomfortable with personalized offers
  • Only 15% feel uncomfortable with personalized products and services

Trust is a huge factor that plays into how comfortable users are with opting-in. Another study, conducted by Salesforce, asked over 8,000 consumers what trust means. 75% responded with “privacy”, and 70% with “transparency” - showing the extent to which trust is interlinked with privacy and the clarity with which privacy policies are communicated.

Best practices for privacy notices

Privacy notices are typically used by organizations to explain how they process personal data and how it applies to different data protection policies, such as the GDPR.

To better understand how privacy notices are presented on mobile, Adjust ran an audit of common practices. We saw three trends in how information is usually presented:

  1. Some apps allow users full data control, with the most granular opt-in options
  2. Some apps limit control by not listing all partners they work with
  3. A third group of apps use an “all in” or “all-out” approach, where users are allowed to either accept sharing data with all enlisted parties or none of them.

We will always recommend clients go with the first approach: be as transparent as possible, and give users full control over their data. To support this, you can explain the reason behind data collection through compelling copy and illustrations as well as outlining the positive benefits of opting-in.

Grouping

Many brands are wondering whether Apple's request can be grouped alongside other privacy notices. When privacy notices are grouped, you can potentially improve the opt-in rate by positioning the notice at the bottom of the screen and presenting two options: “Accept” and “Decline”. However, under GDPR, users cannot be directed into a given reply by pre-selecting opt-in checkboxes or making the opt-in CTA a primary button.

One study has shown that proper framing of the consent message positively impacts opt-in rates: if there are two options to give consent and the message is framed positively, users are more likely to opt-in - such as in the example below.

Although some of our clients have reported opt-in rates of 30-60 % for displaying the Apple pop-up on its own, we recommend you use the remaining time to rigorously test different opt-in strategies for your own users, including bundling privacy notices.

Evaluating your ideas - A/B Testing

A/B testing is a great way to evaluate your solution by comparing two opt-in strategies and assessing their success.

To start, we recommend you A/B test both bundling your opt-in message with GDPR privacy notices, and presenting it as a standalone message. If a user accepts your opt-in message, don’t forget to then also simulate Apple’s ATT pop-up.

Below, we’ve outlined test rounds with different aspects that can help you define a research plan.

First round of testing:

You can then build on these results by introducing further variables. For example, if a bundled privacy notice that includes the Apple request is more successful, you can explore the effects of different copy or design on opt-in rates. Alternatively, if you find that displaying a standalone request (e.g. a pre-permission prompt or the Apple pop-up) was more successful, you can assess the timing of when it’s served.

Second round of testing:

If you have a large user base and enough resources, you can also consider evaluating the effects of more than one variable on opt-in rates using log linear analysis. We also recommend evaluating the frequency of displaying your opt-in approach again for users who didn’t initially opt-in.

You can explore whether there are statistically significant effects for different user segments. You might find that the opt-in rates for new users are higher than for existing users, or that users from one region opt-out more than users from another. With this knowledge, you’re one step closer to dynamically adapting your strategy to further improve opt-in rates.

After any A/B testing, you should calculate a confidence interval to interpret the data. This helps determine what range the true opt-in rate would fall within if the test was conducted with every one of your app users.

Predict the opt-in: Predictive Modeling

Predictive modeling uses statistical techniques to predict certain user behaviors. There are two types that can be helpful for analyzing your A/B tests:

  • Regression analysis investigates the relationship between variables. It can be used to predict the value of an outcome variable based on predictor variables.
  • Decision tree analysis is used to predict a target variable's outcome based on the observations of input variables.

With these analysis methods, you can investigate which variables are most influential in predicting a user’s response. Using certain contextual information, this lets you predict which of the two categories (opt-in vs. opt-out) a user likely belongs to. Here are some examples of predictor variables that you could use as a training data set:

Both logistic regression and decision tree analysis are good methods for solving classification problems. Logistic regression is generally the better approach if you believe that your data set divides linearly into two parts, one part associated with the decision to opt-in and the other with the decision to opt-out. You should also use regression analysis if the values of your predictor variables are continuous.

But if you’re unsure about the data separation, a decision tree is a better fit. And if your dataset contains a lot of outliers, missing values, or is skewed, a decision tree is also often the better choice.

We recommend that you start by applying both methods and then decide which model gives the best result. As a next step, you can assess the individual contribution of the predictor variables to see which variables (e.g. install type, region, demographics etc.) have the biggest influence on the user decision.

Uncovering consumer motivations: speak to your users

A/B testing and regression analysis will show which factors are likely to increase the user opt-in rate - but these methods won’t tell you why the approach works and why specific variables are more important than others. Ultimately, this comes through speaking to your users, and conducting in-depth interviews that turn quantitative findings into defined decision paths. Insights from these interviews, both with users who are likely to opt-in and -out, will let you improve your dynamic opt-in strategy even further.

Summary

Data privacy is the most important topic in the digital sphere right now. At Adjust, we strongly believe this is something every company should embrace, not ignore. Ultimately, taking a clear and transparent approach will help build your app users’ trust, and make them more open to opt-in to sharing their IDFA.

With just a few months to go until the new privacy guidelines are introduced, now is the time for extensive testing and defining how to optimize the consent flow. The more you do to prepare, the better chance you have of continuing to build respectful relationships with users and securing high opt-in rates. We’re working closely with several clients to implement data-driven approaches to optimizing opt-in, and we’ll continue to share our learnings throughout the process. Keep your eyes peeled on the Adjust blog for more insights in the coming months.

Want to get the latest from Adjust?