The ins and outs of incrementality testing for app marketers

The divide between organic traffic and paid conversions is not as wide as some marketers may think. In fact, marketers are right to worry if they are paying to acquire new users that would have converted anyway. To gain clarity about how your campaign ads are influencing user behavior, it’s essential to utilize incrementality testing. Let’s look at what incrementality testing is, explore how it works, and dive into how this method can improve your mobile marketing strategy.

Continuing with the example above, let’s say Group A (a control group without ads) had 100 installs, and Group B (the exposed group with ads) had 120 installs. Here’s how you would find the uplift and the percentage of incremental installs:

• Lift is the increase from Group A to Group B (20 installs, 20% increase)
• Incrementality is the percentage of test Group B that converted due to marketing spend (20 installs, 16.7% of Group B total)

You can now accurately calculate whether your total spend is worth an additional 20% of installs. If it’s too costly, you could theoretically pause ad spend and expect to keep 83.3% of installs.

However, you will not always see a positive incremental lift. You may also find that a campaign’s incremental lift remains neutral. In this case, a campaign may be generating sales but does not have incremental value. Consider changing the creative or updating your target market to gain a positive incremental lift.

Finally, you might see a negative incremental lift. While it’s not common, a marketing campaign may have a negative impact. For example, an overly aggressive retargeting campaign could turn off potential users. Clearly, in this case, you should halt the campaign and rethink its concept.

1. Hit the breaks

An effective way to understand your organic, unpaid traffic is to temporarily press pause on all marketing. Doing so will give you a baseline for organic installs, which is the first step to knowing how much you're paying for new users.

Then you can utilize A/B testing to measure the incremental lift between organic (baseline) and paid results. Knowing this incrementality percentage will help you calculate whether the ad spend was worth the additional installs and if you should dial ad spend up or down.

2. Think regional

Many factors create regional differences — from seasonal trends to cultural nuances to brand awareness. To get a more accurate read on incrementality, marketers must stay vigilant and look at their baseline often and on a region-by-region basis. It’s important to understand ads that cannibalize your organics in one region may not do so in another. Therefore, you need to build a strategy for each region you are testing.

3. Switch it up

Once you've performed incrementality testing and determined which ads are cannibalizing your organics, it's time to act on this newfound knowledge. Continually refining your paid strategy can lead to ads that provide substantial growth. Whether this means going back to the drawing board on creative or rethinking your campaigns from a strategic standpoint, incrementality testing arms you with actionable insights that allow you to refine your campaign in a data-driven way.

With results in hand, estimate how certain you are that this test group represents what you’ll see on a larger scale. Note that user behavior will change over time, so it's crucial to continue your incrementality testing periodically. Additionally, as you scale up your campaigns, there’s always a chance that your findings won’t reflect future results, so your strategy needs to be agile and adaptable.

Before setting up an incrementality test, there are a few key things to keep in mind.

• Control your variables

  First, your groups must be statistically equivalent and have identical conditions, such as the same testing period.

  Without the right segmentation, you might incorrectly attribute uplift due to variables that weren’t controlled. As with any scientific study, having more variables leaves greater room for doubt, potentially making your results unreliable.

• Outline your primary outcome

  It’s vital to decide on your primary outcome before implementing a test. The impact on your primary outcome will inevitably shape your future marketing spend, so it’s important to know what you’d like to measure and why.

  Let’s say you want to measure an ad’s influence on in-app purchases instead of using installs as the conversion event like in our earlier example. For instance, if your app sells shoes, each shoe purchase is a conversion event, so your primary outcome would be the uplift in shoe sales.

  If Group A bought 10 pairs of shoes and Group B bought 20, the primary outcome now shows 100% lift and 50% incrementality. Based on this, you can determine that your incremental users are highly likely to make a purchase. So, as you can see, by changing the primary outcome in this use case, that ad spend appears far more valuable.

• Determine your test model

  Incrementality testing is all about finding a baseline figure that measures the impact of your campaigns, then developing a hypothesis based on your results. Consider these two models for implementing an incrementality test. Each has its own risks and rewards.

  Test model 1: Pause spending

  While it seems counterintuitive, pausing all your marketing spend will help you identify a baseline for organic installs. Then you can slowly introduce each marketing channel and measure the uplift of the value of KPIs you’re trying to obtain.

  Test model 2: Blast spending

  Alternatively, you can “blast” your marketing spend to find a ceiling — essentially, the limit to your success. Although it can be a risk, incrementality testing using the blast model can help you quickly progress toward your long-term goals.

  Michelle Huynh, Director of Growth at Poshmark, provided an example of this method and how it helped them move forward with growth targets:

  “TV is a relatively new channel for us. We took one week’s worth of budget and decided to spend it all in one day. It was a bit risky, but we were looking at the blended cost and an hourly chart, looking at the week-over-week just to see if there was a spike. We then proved that TV users are incremental and a little bit expensive, but this helped us move forward a lot! We were able to scale TV spend efficiently, and it’s now one of our biggest channels.”

• Measure marginal cost

  Once you’ve successfully implemented an incrementality test and established your baseline, it’s evaluation time. When looking at the results of a lift test, make sure to assess your marginal cost and not a blended average. If the marginal cost is highly expensive, it might not be the channel your business should try to scale up.

• Combat cannibalization with an MMP

  The last thing marketers want to do is mistakenly spend the marketing budget acquiring organic traffic, i.e., cannibalization. To eliminate cannibalization, partner with a mobile measurement partner and set parameters that detect existing users. With the right audience segmentation tool, marketers can perform exclusion targeting to this effect. For example, Adjust has a vast set of dynamic macros (placeholders) that deliver data to clients via real-time callbacks.

The benefits of incrementality testing are manifold. Not only is it the most accurate way to deduce the actual cost of acquired users, but it can also prevent you from cannibalizing your organic traffic. In addition, the process provides valuable insight into your marketing spend and allows you to achieve an essential goal for all UA managers: acquiring the right set of users at the right price.

As we wrap up, let’s review the incrementality tips and tricks we’ve covered above.

• Find your baseline with A/B testing, measuring incremental lift
• Have a clearly outlined primary outcome
• Look at the marginal cost, not just the blended average
• Use your results to scale-up appropriate channels and stop cannibalizing traffic
• Adapt your hypothesis based on the results
• Utilize your mobile measurement partner’s macros

It’s hard to argue against the benefits of incrementality testing. Not only is it the most accurate way to deduce the actual cost of acquired users, but it can also prevent you from cannibalizing your organic traffic. The process provides valuable insight into your marketing spend and allows you to achieve an important goal for all UA managers: acquiring the right set of users at the right price.

And once you’ve determined which users to target, how do you utilize this knowledge to grow your app? Check out Scaling your app to 1 million users: The ultimate guide, Part 1 for tips on user acquisition and ASO.