Retailers spend about $2 billion every year to build and run loyalty card programs in the hopes of creating lifelong, devoted customers. However, those loyalty programs often fail to deliver as advertised. But now, advanced analytic techniques running on big data platforms like Hadoop promise to help retailers get closer than ever to realizing their “one-to-one” marketing dreams.
Part of the problem with traditional loyalty programs is the lack of good, clean data. When people sign up for programs, they often refuse to answer questions in the questionnaire, or they intentionally lie about their age, marital status, or whether they have kids. In many cases, all they care about is getting the 5 percent “Club Price” on broccoli at Von’s or getting the 12th tall coffee for free at Starbucks. They could care less about whether the retail has accurate data.
All that bad information meant companies often scale back on their plans of running highly targeted marketing campaigns, says Andrew Robbins, the CEO and founder of Paytronix, which helps companies execute customer loyalty programs.
“There are things every marketer in the world knows: You should segment your guest base, you should think of targeted rewards, and you should run targeted campaigns,” Robbins says. “We were finding customers weren’t doing that. They were just blasting everyone.”
Marketers had grown sour on loyalty programs due to a “relevancy gap,” Robbins says. Instead of sending offers to specific customer segments, the lack of trust in the makeup of those segments was leading to a shotgun approach. While a single man in his mid-30s may appreciate an offer for $5 off a kid’s meal composed of a grilled cheese sandwich and an 8-ounce apple juice, a more effective offer, research has shown, may involve a half-pound cheeseburger and a 16-ounce beer.
Age is a critical factor in marketing, but it turns out that people lie about how old they are. “About 10 percent of people lie, and another 20 to 25 percent won’t answer,” Robbins says. Getting information about children in the household is also tricky. “There are lots of moms who don’t want to tell you they have kids because they’re afraid for their kids’ safety. We ask these questions 50 different ways and all of them generate pretty bad data.”
That’s where big data comes in. Instead of taking the direct approach and asking people todescribe themselves, the modern marketer can use external sources of data and advanced analytics to infer things about her customers. Instead of asking customers to describe themselves, one can accurately ascertain facts just by observing their behavior. For example, if somebody buys a cheese pizza and a milk, “it’s much more likely to be a substitute for a kids meal than it is for an adult,” Robbins says. Similarly, mining for likes on Facebook and Twitter can reveal very detailed preference data for individuals.
Using this approach, a marketer can segment their customer base with 95 percent accuracy, Robbins says. The downside of this approach is that it requires more data. In fact, it requires about 1,000 times more data than the old approach, according to Robbins. That’s why Paytronix decided to abandon SQL Server as a data warehousing platform and invest in Hadoop.
Big Data Validation
Today, Paytronix runs Cloudera‘s Distribution of Hadoop (CDH) on Amazon’s cloud service. SQL Server still has a role in serving insights directly to Paytronix’s customers, which includes companies like Panera Breads and Outback Steakhouse. But for advanced analytics, the relational data store is no more.
Before Hadoop, Paytronix only stored the demographic and loyalty data. But with CDH, a big data application from Platfora, the power of R, and BI tools from Pentaho, a relatively small groups of data engineers at Paytronix has the tools to dive inside the fine-grained data and pull out relevant patterns.
The bulk of the additional data is contained in the “checks,” or the customer receipts generated by each restaurant transaction pulled from the point of sale (POS) system. That’s the gold that Paytronix was after. But keeping track of all that data is no easy task, and requires powerful tools for validating and mixing the data.
“You want to make sure that each field within a check makes sense: How they paid the cashier, the table they sat at, the memo information that’s just typed into the check that says ‘Salad dressing on side–peanut allergy,’” Robbins says. “A lot of this information might be just typed into check in free-form fields.”
Being “close enough” is not good enough in this line of work, so Paytronix takes steps to ensure the data is accurate before a customer acts upon it. “When you have thousands and thousands of these stores all throwing data in, their data could look good. It could be 90 percent correct, but portions of it could be horrible,” says Robbins, a veteran in this field who has degrees from Princeton, MIT, and Harvard. “That’s a data validation problem, and if you don’t try to fix that before you mix it with something else,” you’re asking for trouble.
In the old days, Paytronix would have used ETL tools to build multi- dimensional cubes to validate the data before acting upon it. But that was a slow and time-consuming process. Instead, the company now speeds up the process with Platfora. “They have this really elegant tool that lets you point at raw data in Hadoop, define a cube in an abstract language they call it Lens, and then visualize it. That can be done by a business user, not a software engineer,” Robbins says.
Data in the Mix
“For most retailers, to get to one to one, you’re probably talking about 100 to 1,000 segments, overlaid with personalized communication,” Robbins says. “Maybe the strategy would be, for this segment, I’m going to give them a discount on the last item they bought. In the end, it’s a one to one strategy.”
This is where getting the small things right–like the peanut allergy, the preference for soy milk in coffee, or the preference for hand-tossed pizza–counts a lot. No one person or team of people can be expected to track all this data manually. But thanks to big data tools and technologies, companies can act on this data, and do so with confidence. For marketers looking to build a customer loyalty program, that’s a potential game-winner that can’t be ignored.