Predicting is the Future

22 May 2013

“Big data” is one of those great buzz words (or buzz phrase I guess) I get asked about it a lot. No surprise, we were pretty good at it at my last job. Generally my explanation goes something like this: instead of using sampling like the Nielsen ratings to extrapolate what people might be watching on TV based on a small sample of viewers, we can get the actual viewing data from set top boxes and know with better certainty what most people are watching.

Lately, though, I’ve been thinking about what else big data can do for us. I think the next major step is predictive analysis. Instead of “just” looking backwards at what has happened, like the case of TV viewing habits, we can make pretty accurate predictions of what will happen. Then I realized this is already going on in several places, in very cool ways.

House of Cards

House of Cards
Photo by Netflix

Many people questioned whether or not Netflix could be successful in publishing original content. Netflix never questioned House of Cards. By analyzing their massive data collection, they saw the perfect storm in the mashing up of Kevin Spacey, David Fincher and the original UK series. Their $100 million investment really paid off. Literally, in just three months, Netflix earned back most of that money in new subscribers. This model totally turns Hollywood on its head. Historically in Hollywood, new projects are picked by a handful of people that control vast sums of money. It’s all based on their gut (and past performance to some extent). In the long run, I think the Netflix model will make a lot more money. Looks like Hollywood is beginning to agree.

Google Now

Google Now
Image by Google

Recently released on iOS and long the must-have feature of Android, Google Now is predictive search tailored per person. While House of Cards was a move to figure out what a large group of people wanted to watch, Now aims to figure out what a lot of individuals want to know. I travel a lot and can say that Now has become a constant go-to for me. If I do a map search on my laptop, those directions, along with traffic info, show up as the top card. When it’s meal time, Now shows me nearby restaurants I might like. If I’m away from home (and it figured out where home is all on it’s own), I see a list of nearby attractions and photo spots I might want to check out. More than once I’ve checked out a new restaurant or gone out of my way to see a cool view that I would never have actively searched for. If autocomplete is Google’s attempt to finish our thought while we search, Now is Google’s (pretty damned successful) attempt to read our minds in advance.

Obama for America

Facebook Notifications from Obama for America

At the campaign, we had some of the best data scientists in the world. Their work enabled a multitude of data-backed projects. One that I am pretty familiar with used Facebook social graph data to predict which friends of our Facebook app’s users would be most likely to be persuaded to vote for the president and which of our users had the best chance to be the persuader for us. Each of the friends were assigned a “persuasion score” based on a number of factors from publicly and commercially available data. Then the connection between the app user and their friends with high persuasion scores were ranked by “social proximity”, a ranking of how likely it was that the two people were close enough friends that our user could actually impact the end voter. We then asked our users through a variety of methods to encourage their friends to vote. But not just “friends” in the generic sense; the individual people we’d identified through this data. On Election Day alone we sent 7 million Facebook notifications. When people ask me what 2016 is going to look like, I think it’s a lot of engagements like this where individuals reach out to individuals encouraged by a lot of analysis to figure out who both of those people should be.

Where we could go

The brilliant Anil Dash makes the argument that all dashboards should be feeds. The point he is making is that it isn’t enough to show big charts that are an overview of vast amounts of data, that people need digestible, concrete chunks of information out of that data. Both he and I would say that not all charts are bad, but they should be mixed in with highlights of specific points. Where prediction comes in is when data can be used to suggest a next action. Consider a few scenarios:

Large companies have had access to this kind of software for a while. It’s called business intelligence. I see a real growth market for services that provide this same kind of benefit to small business. This becomes especially powerful when the data collected for the whole service is leveraged and not just the data collected for a specific customer. A mass emailer could suggest best time of day to send an email not just from your record of email opens, but from the average across the whole system.

With the rise of open data, particularly from governments, small businesses can incorporate much larger datasets than they could ever generate on their own. For example, the United States made weather data openly available. Compared to the similarly sized economy of Europe, which mostly does not make weather data publicly available, the US commercial weather industry is roughly 13 times larger. This is a case where big data enabled an existing market to expand, but there are multiple ways in which open data can be used in indirect ways. Small businesses can use traffic flow data to determine where a storefront would get more drive by visibility. Instead of buying one expensive billboard with high traffic levels, buy five cheaper ones that actually get more aggregate eyeballs. Or use census data to determine which areas have income levels that make the residents likely clients for direct marketing campaigns. There are real opportunities here both for businesses that consume big data and especially for those that use that data to advise others.

On a recent episode of In Beta, Gina Trapani and Kevin Purdy discussed how apps that give users useful notifications just at the time the user needs them gain a real sense of value from their users. Implementing effective big data prediction into your apps is a sure way to increase user retention. Leveraging what you know to suggest compelling actions will give your service a much better chance of a long, profitable future.