Written By: Steve Zagieboylo, Senior Architect at Calavista
While it may seem I’m just trying to work in as many buzzwords as I can, in fact, there really is an important intersection of these three elements. I’ve been interested in both big data and fast data for several years, and my newest tech interest is machine learning. As I have learned about the latter, I see that there are problems that require all three to be truly effective. One application for which I’m looking at bringing together these technologies is in Recommender Systems for brick and mortar shops.
Big Data + Machine Learning -> Recommender Systems
Probably the first big win for machine learning was Recommender Systems. You’re probably familiar with these in your online shopping, movie watching, or music selection activities, where the website suggests (usually pretty accurately) additional products, movies, or music that you would enjoy. There are a few algorithms for generating these recommendations, but one aspect of them all is that they need a lot of data to “train” the system.
In a shopping recommender, data is fed into the system as “features.” These are measurable pieces of data that the developer thought would be relevant to the buyer biting on additional lures and adding more items to their cart. The developer doesn’t have to know how those features are relevant; as long as he is feeding into the machine learning system a reasonable set of features, it will figure out which are significant and to what degree. The developer “trains” the system by feeding in known data with known results. The machine learning program tweaks all the parameters – how much it cares about each feature – until the known data tends to create the known results.
The synergy for big data and machine learning, therefore, is straightforward. There is a ton of data that is potentially relevant, and the developer will need a lot of known examples with all that data available before he can train the system. Since he isn’t sure what data will make good features, he has to collect everything, even though some of it will turn out to be inconsequential. It can be surprising which pieces of information actually matter, but he won’t find out until actually training the system with all that data in place.
Fast Data
Fast data systems also deal with a lot of data, but do it in real time. Typically, there is less variety of data types that what you would see in big data, but the quantity can be staggering, and the decisions it makes need to be timely.
Fast Data in Recommender Systems?
Returning to our example of Recommender Systems, there doesn’t seem to be much need for fast data. Collecting the feature data for a single user to process through the trained system is not demanding enough to need fast data.
But now, let’s think about brick and mortar shops. Say you’d like to improve the up-sell opportunities to those customers, but to do it in a way that is not as annoying as the ubiquitous “would you like fries with that?” At first, it seems that you know very little about the customer. You don’t have a history like an online store maintains for its logged-in customers, and, unless your salespeople are extremely observant, you don’t have any information about what the customer looked at and considered when making a purchase.
Cue the ominous, “George Orwell’s 1984” music, here.
Or do you?
Mobile phones today broadcast their GPS locations to all who would like to listen (or, more accurately, to all who would like to pay for the “anonymous” data). This data is accurate enough to know not just that the person is in your store, but where in the store he is. If you’re tracking this information, you can know how long the phone spent in each department, even how long was spent right in front of a particular product display. If you could feed this sort of information into a recommender system of the same sort your online store has, very likely you could prompt your salesperson to point out some specific items in the clearance section, or to offer a specific coupon.
However, the data does not come into the system in such easily digestible form, with locations and time spent, the sort of data you could feed to your recommender system. If the customer has connected to your free wi-fi, he is very likely sending a simple location 30 times a minute. When you multiply by all the customers in all your stores, it is a flood of data that is difficult to use for any sort of purchase recommendations. Your big data system could store all this, and could probably crunch it to be useful, maybe tonight, but that’s much too late to do any more than send the customer an email which will probably be ignored. If you can’t act while the customer is standing there, you’ve lost the opportunity.
The piece that is missing is a fast data. A fast data system can process that stream of GPS locations and build useful information that becomes features in your machine-learning-based recommender system. I don’t know how each datum translates to sales, but that’s the beauty of machine learning – we don’t have to know. If you can generate data that is somehow – probably – relevant, and some known results with which to train the system, then it will do the rest. As you get more data and more results, both positive and negative, then with each retraining your recommender system will get better and better.
All three pieces are necessary for this complete system: You need the fast data processor to generate the data while it is still relevant. You need a big data system to hold all the inputs over time, what was recommended, and whether or not it succeeded in generating additional sales. And you need a recommender system based on machine learning to be able to improve your recommendations in the future.
