Bot or human? Our panel at Gamesbeat Summit 2019
In-app bots are a menace — no-one knows that better than Adjust and Unbotify.
Adjust’s Global Product Director, Katie Hutcherson-Madding, and Yaron Oliker, CEO of Unbotify (Adjust’s bot protection offering), took to the stage at Gamesbeat Summit 2019 to sound the alarm about bots to app game developers of all kinds, reminding listeners that if your game features monetization of any kind, it’s a juicy target for fraudsters.
Below are a collection of key takeaways from the talk. You can also find the recording and transcript below to catch up on the whole panel.
Flying under the radar: The scale of ad fraud is immense, but many marketers aren’t aware of how much it affects them. “From our clients, we see $2 million in savings from fraud rejection a day,'' said Katie. Recently, Juniper estimated that fraud will take $42 billion in campaign spend this year alone, though Adjust estimates around $29 billion. This is a vast sum, and if you don’t think you’re in danger of ad fraud, you’re almost certainly wrong.
Two sides of the same coin: App marketers need to be aware that there are two types of fraud at work. They’ll either go after your ad budgets, or harm the in-app experience. Serial criminal organizations work on the former, while the latter is driven by individual players looking to boost their experience in-game, or farm items unfairly. However, both develop at a quick pace — in fact, as Katie pointed out, fraudsters, “are exponentially growing in terms of their intelligence, and they’re trying hard to outmaneuver you.”
Beating back the bots: “It’s about making it more expensive to attack than to defend,” says Yaron. Unbotify’s solution is built on behavioral biometric models which emulate how real users play with their games. It would take a significant investment for fraudsters to combat this: “Your ability to create an adversarial model to understand what authentic interaction looks like would be ridiculously expensive.”
Keeping an eye on fraud: However, there are tactics marketers can employ to fight fraud on their own. Katie told the assembled audience, “start to really look at your traffic, start to look at some of those key indicators, and realize that you know if you don’t have something, your reference points and all of the traffic you’re looking at, all of your typical KPIs, are bogus.” Katie detailed her three “BS metrics” that helped marketers to spot ad fraud.
While fraud is attacking budgets round the clock, keeping vigilant, doing your homework and activating Adjust’s and Unbotify’s fraud solutions can be the difference between bots being a nuisance, or a significant problem. Take a look at the full video of the panel below, or scroll on to find the transcript.
The full transcript
Moderator: Today we’re going to talk about typical gaming behavior bot or human, because, you know that person in the game with you may not be human! I’m Steve Peterson, I’m the CEO of Story Force Entertainment and I’ll be moderating this panel. With us we have Katie Hutcherson-Madding — she’s Global Product Director for Adjust and Yaron Oliker, he’s the co-founder and CEO of Unbotify. And I’ll let you each talk about your roles and what you do in this space of bots and humans and mobile games.
Katie: I work at a company called Adjust and we are a mobile marketing solution. So basically whenever you’re running all of your marketing campaigns, advertising, we’re able to show you your performance, track that, optimize it, build audiences and then my work here today is to talk about fraud prevention. And we recently acquired Unbotify, and I’ll let Yaron speak to that.
Yaron: Unbotify is a company that was started about two and a half years so, in the space of cyber security and fraud detection and we basically help our customers understand if a user in an app is a bot or a human.
We do that by collecting our sensor data from the device they can tap or swipe and how you hold the device on a mobile phone, sensors like gyroscope and sensometer, and then what we do is we build the model with machine learning of how authentic interaction looks like, and we use that to detect anomalies. For these bots and fraud tools it’s very hard to fake this kind of interaction. So, in the context of gaming, we’re working with some of the largest gaming studios in the world, helping them clean out their community from people who are cheating, using bot services to cheat or get an unfair advantage in the game: that’s Unbotify.
Moderator: So Katie can you frame for us just how big a problem is this? Is this a significant amount of money that we should be concerned about and a significant amount of cheating that’s going on?
Katie: When we began offering our Fraud Prevention Suite the majority of our clients were oh we’re not affected by that, we never see fraud kind of in our spend, in our KPIs they’re all solid, right? And in 2019 digital ad spend is supposed to be around $80 billion. Based on what Adjust has measured (which is a small portion of this ecosystem) we expect around $29 billion in fraud, which is pretty crazy. On top of that, just from our clients, we have seen on average $2 million in savings from fraud rejection a day.
There’s really two types of fraud out there, that gaming publishers need to be aware of, so first is basically the type where you are spoofing engagements or attribution, so companies or fraudsters that are going out there and actually trying to get the payout saying that we drove this user to install this game. Right? The other type of fraud that I think is much more systemic to gaming is actually spoofing users and user activity and user behavior which is what we want to talk about today. I think that key piece about this though is that a lot of companies will go out there and say okay, we’re going to get a fraud detection tool. I think that this is where we want to start that way we can kind of learn and see how much fraud we actually have. And I like to compare this to a credit card, so I just got back from Barcelona, and let’s say my credit card company called me up and said we found fraud, you know somebody stole your credit card, you’ve got $500 in fraudulent purchases, here you go. Then they say, how about you take care of that fraud. You go after it, we’ll let you see that it’s fraud, we’ve told you that it’s fraud, but now you get to watch all of these additional purchases and take care of it.
That’s detection and none of us would actually want that from our credit card provider right, like I’m not trying to say oh yes go take my credit card, have fun with it, I’ll go find the person that stole it and deal with all the consequences. So it’s actually critical to get a fraud prevention solution. Basically, a company that will detect the fraud in real time, and reject it. That way you’re not having to go back retrospectively and have these conversations with the company saying hey I want my money back. You can give them the laundry list as to how you were able to detect these different types of fraud, Click Injection, Click Spam, Distribution Modeling, all the stuff that’s out there, but at the end of the day you’re going to want a fraud solution that doesn’t make you have to have that conversation.
Moderator: Because it can take me a lot of time to explain to a credit card company what I’m talking about and why I should get my money back and this is esoteric to them, it’s not something they understand.
Katie: Exactly and it’s not something that when you’re spending all of your time kind of going in developing these really cool games, developing these amazing user experiences, you don’t want to have to at the end of every month, go back and settle up with these different companies and say listen I know this was fraud, then try to get your money back.
Moderator: What are fraudsters doing inside of games to cause problems?
Yaron: So part of it is users who are using bot services to gain an unfair advantage in the game. This problem is about as old as multi-player gaming, if you go back to Runescape or World of Warcraft, back in 2007, they both had big bot problems that they actually never solved, so this never went away. Take away kind of popular strategy MMO, and you try Googling, “cheat games app” or “Clash of Clans bot” you’ll get you know a million results, people build the bots and then actual players can rent those bots. So there are services where you can pay money that to attach your account to that bot. The bot will play the game for you, and will farm in-game resources to gain an unfair advantage. There’s also like other cases of selling either like maxed out accounts, or end-game currencies, especially in games that have elements of training right? Like World of Warcraft for example. You can really, if you can automate that stuff, you can kind of really hurt the games community, and annoy the real players. We’ve seen like on strategy games that we’re working on sometimes these bots will usually crush humans too. Like they pretty good at applying strategy and perfect execution.
Moderator: We’ve even seen that in professional e-sports matches right, because some of the competitors have been revealed as using aimbots and other things to help them in professional matches.
Yaron: Yes absolutely, I mean especially if you’re talking about a game that is making a push for e-sports then fairness becomes a really big issue right? Otherwise it’s not, what’s the point of it. But also like if you open Reddit for literally any successful MMO you’ll find people complaining about bots ruining the game.
Moderator: It is kind of corrosive if you start wondering are the other players playing fairly, am I competing on a level playing field. But what, you know, where are we at with fraud now and going forward? Do you think the problem is getting worse as we get larger audiences and you know diversity of platforms or is it – are we starting to make it better, you know where are we at? What is the future look like?
Katie: If we look at Game of Thrones, obviously a big fan here: regarding their app, their mobile game app, one of their affiliate networks drove more clicks for advertising for them than there are people in the world. In a month.
Moderator: So they suspected fraud?
Katie: So you know there’s little hints like that, especially, the phrase that everyone uses: fraudsters follow the money, right? But it’s also for any of these popular games as well at least on the user acquisition side there, it’s pretty clear if you have more clicks than you do people in the world I think you might be a little suspicious, you might be able to call BS on that. And then we were just talking about it from the actual in-app user experience, and you were able to find in two seconds.
Yaron: So you know I just googled Game of Thrones conquest bot there are plenty of like, commercially available services that will play the game for you and basically help you progress in the game right? All these games you get to that point where you have to pay to advance otherwise, or invest a huge amount of time, but you can cheat, and it’s cheaper.
Moderator: So when you’re developing a game, you’re making a walking target for fraudsters. What can you do to make yourself less of a target? You know is there types of games or things they can do with the game, or is it just I need to hire security, I need to get fraud prevention, is that what can you do as a developer?
Yaron: So the games that are more susceptible to this kind of bot attacks are games that have either real money gaming like gambling that’s always a big target, because if you’re playing online poker and you know two hands on the table are you and your bot, that’s going to work out probably better for you. And basically any game that has trading features that you can trade stuff in accounts, is going to be probably more attractive to fraudsters, as they can sell off stuff that they farm in the game. And like basically games that are more mid-core and above are almost always susceptible to this kind of bad behavior.
Moderator: So on the reverse side the hyper casual games are not really of interest.
Yaron: Yes if you’re playing bubble shooter you’re, you know, you’re probably good.
Moderator: Okay, what, you know Katie is there a problem with the ad fraud you think is that growing at the same pace?
Katie: Yes so, internally we call this kind of the nuclear arms mission crisis. And I know that that sounds a little dramatic, but back when we started this initiative to get rid of fraud, basically SDK spoofing which was simulating traffic, or simulating in-app behavior started on Android with your simulators, your competing cloud services. On iOS it was device farms, and that was kind of the old school right?
Moderator: I remember pictures of cell phones on racks-
Katie: The exact same, you know, here we go, this is the tilt and we know the exact same behavior and what’s funny is that every time we actually reject that behavior the fraudster becomes smarter, that’s more information that they now have to trick their own system into outmaneuvering whatever fraud filter you have created. So, a ton of clients that go out there and say we’ve got our fraud detection or prevention tool, we’re good to go, we’re set.
I think the key piece is that any company out there that’s offering a fraud suite has to constantly think about it from a fraudsters perspective. They are exponentially growing in terms of their intelligence, and they’re trying hard to outmaneuver you. What we have seen most recently is what we call SDK spoofing, where hackers will break open SSL encryption, hash signatures and send us what looks like perfect human traffic. Literally has all of the metadata that we would expect from a human. But it’s not. And I think that that’s the craziest part about this is that fraudsters are constantly coming up with new ways to throw up any of the filters that you have just built. So you’ve got to be innovative, and you have to think from their perspective and anyone out there who thinks okay, I turned on the fraud detection for a month, I’m good to go, I’m set, it’s never going to happen for me again, that is laughable. Because you are literally just throwing yourself out there for a world of hurt and it is so much money, it’s so much time and another thing I would say to that is a lot of our clients initially will say well we, we can’t handle that because it’s our CPIs are going to go through the roof, we’re going to have to pay so much more now to acquire users, to play our games, because there’s so few out there. So we don’t actually want to prevent this. It actually works for us right? And I think that that’s another key piece, the majority of our clients who’ve actually used our fraud prevention suite have had a 12% increase in their return on ad spend — because they’re no longer focusing on quantity but quality.
Moderator: Yes it’s not about the number of users, it’s about good users.
Katie: That quality yes. Human users, even maybe a few of them.
Moderator: But what you’re talking about reminds me of antibiotics versus bacteria, you know, you only have, we have several different antibiotics but you use them more and more and the bacteria get resistant.
Moderator: Then your antibiotics don’t work anymore. What do I do now?
Yaron: It’s a cat and mouse game, but it’s there is actually a way to beat this, and it’s about making it more expensive to attack than to defend. So if you’re talking about the context of using bots for ad fraud, what we do at Unbotify and at Adjust now is we actually build a behavioral biometric model for every app, how that app-specific real authentic users behave, as you go through the games, it flows. How they go through the tutorial, how they go through a battle screen or whatever that specific games context is. And if you’re doing ad fraud, if you’re trying to run an install fraud operation, your ability to create an adversarial machinery model to understand what authentic interaction looks like on every app you want to defraud, that’s-
Moderator: Ridiculously expensive.
Yaron: Yes it doesn’t scale nicely which is the whole point of using bots for fraud right? So, you know while in security there is never a 100% bullet-proof solution, it’s not about making it impossible to break, talking about defending against bots like political bots on Google and Twitter, you know so you’re up against State-sponsored actors that have a lot of incentive, a lot of resources to try to break your systems. But if you’re talking about the fraudsters they’re in it for the money, well it’s about making it not cost effective anymore, so you know, you might as well try some other kind of fraud.
Moderator: Is the problem similar is scale on PC, or console? How does it compare versus mobile?
Yaron: I can, you know if you’re talking about in-game bots. Yes, so console is relatively much more secure than web and mobile. Web is the most open platform, it’s really easy to build a bot on web, to abuse any kind of game, any kind of website, and on mobile specifically like using technologies like leveraging the sense of data, using behavioral biometrics, you can build something that’s really difficult to break. So, when you’re talking about consoles you’ll get mostly the signature type SDK spoofing type attacks.
Moderator: Is this something that law enforcement is interested in pursuing at all or I mean is it just there’s this cyber war, against fraudsters going on and law enforcement like we don’t even understand that and we don’t care, or what?
Yaron: So generally in ad fraud there were maybe I can think of five cases of people who were prosecuted for running ad fraud. It’s relatively very safe and very, very lucrative. The world federation of advertisers has 20 billion dollars a years is lost to advertising fraud right?
Moderator: If 20 billion dollars was lost to bank robberies people would be getting more security on banks and they’d be prosecuting the bank robbers more heavily.
Yaron: Yes advertisement fraud is the largest kind of cybercrime right now, in the world, and probably the second largest organized crime operation, after drug trafficking. So it’s pretty big.
Moderator: Well especially if you, as a criminal you look at it and say what’s my potential downside, how many people are going to be prosecuted. You know they may shut you down I’ll just re-open something somewhere else.
Yaron: Yes it’s a much safer.
Moderator: Do you see us reaching a point where I don’t have to worry continuously about it, or is that you think that technically we can get there, or is it just every year, every month, every day, I’m going to have to spend some time thinking about this as a game publisher?
Katie: Yes and it’s something actually that internally we’ve been asked by a lot of our clients okay, well if you get machine learning behind it, if you guys have set up all these different filters then you should be good to go, it can kind of just you know keep iterating on itself and I think that the key there is that, while machine learning is great once you actually have the set algorithm and the set play, it’s not great for identifying new methods, and there are constantly new methods. Like I was saying before the first time the ad fraudster doesn’t actually get the payout, just rejected, they’re going to look into what was the anomaly and what they did that was found, that made their traffic not seem like a real device or a real human and so that’s where I think that when it comes to a fraud suite that constantly gives you that piece of mind, where you can say okay, I know that my traffic’s solid, I know that my reference points are not diluted based on a ton of fraudulent traffic, that’s where you really need a company that’s constantly looking into okay here’s the type of fraud that we’re looking into. What’s the next method that these fraudsters are going to use to outsmart us. And I think that that’s kind of the key piece, because there’s always going to be something new. And literally in the past two years I think we’ve had seven different filters, that we have released to try to prevent basically just the spoofed attribution and that’s one type.
Moderator: So what happens is that, the fraudsters in general get used to a particular thing, and then they all shift to a different method which there isn’t enough protection again and it’s kind of like you know Facebook ads versus, well everybody is jumping into Facebook ads because those are working great, but after a while once everybody jumps in they won’t work quite as well, so people will be looking for another new thing and this is essentially what the fraudsters are doing, hey this is working great for us, oh no it’s not working so well we’ll come up with a different way.
Katie: Exactly and going back to the expensive point right? So in a lot of these different methods that they’ve had over time, they become expensive and cost prohibitive because of what we’ve used to actually reject them. So they’re constantly looking for that margin right? They are looking for the method that’s going to have the least cost, and the most payout from all of you guys and I think that that’s kind of the key piece. Fraud will always be prevalent, as long as there is a margin within this space, a margin to be made for them.
Moderator: Is that, are there, are they, looking for ways within the game itself to help prevent this or make it more difficult, raise the cost of the attacks do you know publishers that are doing that?
Yaron: Yes, so changing the game economy, or changing something in the game, I mean yes that’s one way to tackle the problem you know, I believe that there’s technology solution for this, which is stuff like what we’re doing. And when you’re talking about building a technology barrier you should also think about you know it doesn’t have to be the you know 100% bulletproof, it has to be better you know. If I have bars on my window and you don’t it doesn’t matter if they’re from titanium you know or not.
Yaron: That’s always the case in security, you have to be a step above your peers. I believe that technology can solve this and you know we’ve, but we’ve also seen you know, we’ve seen companies move to mobile as it is a more secure platform and I think you will see a lot of the, like biggest bot targets out there, think of apps like, I don’t know, Nike for example. Nike have a bot scalping problem, that’s been an ongoing war for years, and I think you can expect companies like that to move only to do like promotions on mobile rather than web. Because it is easier to secure that.
Moderator: Because with mobile you have those sensors that you can read data from and figure out a protection scheme.
Katie: You have a lot more metadata to pull from.
Moderator: So, well I think it’s a, you know it is a huge problem, the scope of it is kind of appalling and are there any countries or jurisdictions that you think are taking some measures other than you know just-
Yaron: Well, we’ve seen the EU shut down bot operations, but that happens after you sue and you issue like, you send them a cease and desist letter and eventually you know after years of legal dispute you can shut down a bot operation. And then they will spin up a new one with a new domain and new company. So that’s, you know legal action is not the right way to solve this. It’s too slow and it’s really easy to spin up a new anti-
Moderator: Yes when your operation is mostly virtual and you can open a new office in some third world country that doesn’t care and go there, well you know are there when you advise companies you want to advise developers what to do about this what you know, what should my take away be as I prepare to launch an application, or as I have an existing application that’s out there. What should I do if, you know I hadn’t been concerned about it before, hadn’t realized what the scope of it was, and I didn’t realize that I could be losing millions of dollars, I didn’t know. Where do I start? What do I do?
Katie: So from the advertising perspective, especially from gaming I have what I call my three BS metrics. Basically the first one is if your click-through rate is less than 1%, fraud, call BS, that’s just not, that’s not what you should expect. The next is you all have very different LTVs, but if your LTV is super, super low, especially at that sub-publisher level that’s also a BS. That’s just, that’s not real and then the final one which I mentioned a little bit earlier, but if you were receiving more clicks than there are devices in the world, if you’ve got over 2.5 billion clicks happening within a certain time frame, also call BS. Right? So those are kind of the easy three to just quickly realize. The unfortunate part is once you’ve been able to call BS that’s a little too late right? That’s retrospective and then you still have to have the conversation. So the next piece of that is actually getting a fraud prevention suite. So basically there’s all these third parties out there that’ll offer you kind of different fraud solutions, where you’ll be able to go away and look and say okay, this percentage of my traffic or this percentage of my users look to have fraudulent tendencies, but at the same time you still then have to go out and have the conversation, right the wrong, at the end of the month. So you want a solution that will actually reject this fraudulent behavior in real time and the solution has to have filters, that are based on a certain confidence – right you don’t want solutions that are just going to reject everything, have tonnes of false positives and then kind of you know obviously really affect your marketing budget. You are going to want a solution that probably has around the 95% confidence interval, and does that real- time rejection.
Moderator: And what about for in-game fraud, what is, what’s your recommendations to the publishers as to how they can deal with that?
Yaron: Yes you’re a small studio and you’re launching a game that’s probably not your biggest concern, but if you have like an existing game and you want to build a community, these things can be really, really harmful in terms of creating resentment in your community, when you’re playing like counter strike and somebody kills you with an aimbot you’re really pissed. Once your game is successful, you’re going to start encountering those things more and more, and especially strategy games, as soon as they get traction, they get the bots.
Also I could use acquisition, if you’re just starting out it will probably put your money on your Facebook and Google, but once you start you know running larger UA budgets across multiple channels, or you’re going to like APAC where there is no Google and Facebook, you’re going to be much more susceptible to this kind of problem.
Moderator: So, as a game company, as I’m reaching a good market now, I’ve got a million users, for my strategy game, it’s getting some good traction, you know how do I even detect fraud, especially if it’s a web game, you know how do I know what kind of end game fraud I’m getting, obviously for ads there’s good tools to detect that, what kind of detection is doable and you know are there something like Katie’s BS clues for in-game fraud?
Yaron: Yes well when you’re just starting out you can pick out players that play 24/7 and never sleep, that’s a clear. And then once you start blocking the obvious stuff, these tools evolve to mimic human behavior more closely. But you will know because your players will be complaining because your players will be churning and it will affect your bottom line, like that’s a good answer.
Moderator: Yes because your player base is your first line of detection because they’ll let you know pretty quickly if they, this seems pretty fishy.
Yaron: Yes and like most of these games have flagged this account as a potential bot. That’s not a great way to tackle that problem.