Marketers are increasingly driven by data. But as I parodied a couple weeks ago, we have to be careful with what data we choose to listen to.
Data is malleable. We can cherry pick data to support just about any argument we care to make. That’s one of the skills we learn in business school.
But as Kraft’s Julie Fleischer said last month at the Ad Age Data Conference, “90% of data is crap” and “they don’t teach data literacy in business school.”
Big data doesn’t inherently lead to greater insight. Marketers have greater access to data than ever before. But finding the signal in all of that noise is the hard part.
Confirmation bias — the tendency to search for or interpret information in a way that confirms pre-existing beliefs — is one of the many pitfalls for marketers learning to work work with data.
Data doesn’t have biases. It’s people who collect and select the data who bring bias to it.
I’d love to hear your thoughts on some of the pitfalls of working with data as marketers.
(Marketoonist Monday: I’m giving away a signed cartoon print. Just share an insightful comment to this week’s post by 5:00 PST on Monday. Thanks!)
Jeff Stoltman says
The apparent rush to embrace ‘huge data’ needs to be tempered by a simple truth. Paraphrasing here: “Numbers don’t know where they come from.” Lord, F. (1953) On the statistical treatment of football numbers. American Psychologist, 8, 750-751 was required reading in graduate school (decades ago). It offered a contemplative alternative to the other required reading:”How To Lie With Statistics.” Implicitly, we were being schooled in judgment,the principles of scientific discovery, the identification and avoidance of bias, and the proper use of analytic tools. It seems nowadays the mere fact that numbers exist compels action. Best to ask: Who knows where these numbers come from?
Michael Zimmerman says
For years, I worked with businesses in the printing industry – an industry plagued with overinvestment in rapidly aging technologies and characterized by continually declining production volumes. Year after year, industry experts reassured print-shop owners with surveys that showed “the slump was over” and print buyers would spend more in coming years. Invariably, these survey results were based on the owners’ collective opinion (read: “hope”) that the market was about to turn – that happy days would soon be here again. And many printers believed the forecasts… because they needed to. Despite obvious trends to the contrary. (Some of which were megatrends.)
What’s the root cause of our addiction to unsubstantiated good news? I’m no psychologist. But I can’t help but be reminded of a Monty Python skit (The Meaning of Life, 1983) in which a British army doctor offers a comforting prognosis after his patient’s leg has been “bitten sort of… off.”
“Yes, well, this is nothing to worry about… there’s a lot of it about… probably a virus. Uh, keep warm, plenty of rest, and if you’re playing football or anything, try and favor the other leg… Be right as rain in a couple of days… Any other problems I can REASSURE you about?”
We seem to have an undeniable need to believe in something. And it might as well be something pleasant. Maybe we find comfort in survey results that REASSURE us. Even when our legs have been bitten sort of… off.
Vlad Dogaru says
Very valid point when it comes to the challanges of using data. In terms of the solution, it’s all about using the right methodology in the end? Big data still relies on all the basics we learned in market research and statistics. However, how many organizations are trully data-driven? I would imagine that the emotional and political factors still play a major role… so data is not the answer to everything!
BC says
For the record, I like data and the analytical process of working with it – if you can approach it with an open mind, it creates opportunities to think in new ways, see connections you might not have otherwise considered, etc.
But I did some work with one of the big guys in data analysis and two things stick with me: 1) the mathematical correlation between various data points is only meaningful if it can lead to *practical* action (noting that “practical” is something which depends on factors unique to each situation); and 2) there were many times when a group of people involved in a project were looking at the exact same data and coming up with very different “insights”, sometimes including ideas which were nearly opposite in terms of interpretation.
I chalked up those different interpretations to a sort of “bias” though not meant in any negative way but a factor regardless. Interpretations of information is of course influenced by each of our individual experiences, skills, etc. A senior-level marketer with top-10 CPG brand experience and a major budget will process, consider and interpret information differently than a mid-level marketer who has worked with smaller brands and all that implies. Marketers in the high-tech vertical will likely process information differently than those in low-cost consumables.
Regardless of my examples, the point is that there’s never only one “correct” way to interpret information. So I have come to believe that the trick is to pick an interpretation which “feels” right, which “fits”, and apply it *with passion* — stick to it, commit to it over a reasonable period of time.
In general, we don’t get to do (big) things two different ways simultaneously, measure them, and take credit for the better approach. That said, it’s my feeling that every practical approach will deliver results and the variation in outcomes, if they could all be done and measured, would likely *not* be substantial. Please assume I mean practical ideas and note that I’m not dismissing the benefits of better results (if you can predict them…) but rather making the point that over-thinking and over-analyzing data is less important to the outcome than making the commitment to an approach and sticking with it long enough to make a difference.
Chris Neumann says
Why not just agree on the relevant metrics ahead of time? That’s part of my standard checklist prior to running every A/B test.
I think the problem you’re making fun of here is the difficulty in finding someone to run the data that understands business – that’s a *very* hard position to hire for. This is why I’m constantly advocating for marketers to understand a little bit of SQL – that way they can speak the data person’s language. One of the big challenges for marketers is that everyone thinks they know marketing. I also think that a much worse problem is marketers doing things on gut rather than trying things they think will work and then finding out if they do based on metrics agreed to ahead of time.
Allen Roberts says
You need look no further than the swill delivered up by our elected so called “leaders” in the political sphere for examples of selective use of data that “supports” a pre-determined point of view.
I am sure it is the same in the US as it is here (Australia), these political ghouls break new ground on selective data manipulation on a regular basis.
Greg Hopper says
Approaching data with a testable hypothesis is a way to find the needle in the haystack (the “signal in the noise”). Too many data mining projects are undertaken to see IF there is a needle in the haystack (which is a predictably open-ended, expensive, and unpredictable process), rather than proving or disproving a hypothesis.
Marjorie Valin says
It’s been my experience that organizations make decisions based on minimal or incomplete data because their platforms are not fully integrated. For example, the ability to track email clickthroughs but inability to relate them to web activity. These sets of unconnected data show 2 completely different results that provide no insight, although they are used as the basis for big monetary commitments.
Jim Flynn says
“Big data” has been the big-buzzword in recent years. Don’t get me wrong, it still has a valid role for many marketers. What is becoming increasingly valid for more-and-more organizations is interpreting data gleaned from their own inbound marketing efforts. Their customers. Their prospects. Real beating hearts, raising their (real) hands–in real-time and telling us what they think, like and want. Analytical psycho-babble, I think not. Intuitive marketing, backed by solid data based insights…you bet. The beauty part of the software today is that you don’t need an army of IT people to run queries to interact and figure it out. Oh, and when the guy on the other side of the table is quoting his “unbiased data” to call your idea stupid…well your idea might just make more sense…and you can prove it with revenue and real-time customer interactions. Good stuff!
John Miglautsch says
GM Credit card built a database with 250 variables on each person. They wanted to answer the question, “Who would have bought a car anyway, without the $5,000 rebate?”
“I love those kinds of questions.”I replied.
“Why?” They asked.
“Because I can answer any way you like.”
“But we thought you were a data person.”
“But I suspect that you have no data for that
question, so I can answer it any way that you like.” They hadn’t set up appropriate testing samples. With all that data, it wasn’t a data issue.
from Spinning Straw Into Gold – http://bit.ly/Straw2Au
Sean Regan says
The problem with this critique is that data doesn’t tell a story. Only human beings can see an insight, a story coming together, and then turn it into action. Structuring criteria ahead of time (as Chris N suggests) is a useful discipline to help more objectively evaluate whether or not your story (hypothesis) is likely to be true. But data will never provide “objective” insight, because only human beings can connect pieces into a whole, and we are subjective by nature. The good news is, so are our consumers.
Jon Plummer says
“I can’t find it right now. But you are stupid anyway.”