Marketing is increasingly data-driven. But the data we choose matters. It’s tempting to cherry-pick the data that makes us look or feel good. Or the easiest data to persuade an audience to buy into whatever we want them to believe.
This is particularly the case in digital marketing, where over-simplified numbers like Likes or YouTube views are frequently used as proxies for success. There are many “vanity metrics” commonly reported in marketing that are easily gamed and have nothing to do with whether a campaign actually made a real impact.
Last year, an ad agency in Minneapolis called Solve ran an experiment where they uploaded a 4-minute completely blank video to YouTube (no sound, movement, or anything but a blank screen). They then managed to get over 100,000 YouTube views by simply buying viewership for 1.4 cents a view. In addition to the view count, they were able to cherry-pick a few engagement metrics (1-in-5 saw it all the way through, higher than average click-throughts, etc.) to make it seem compelling. But all along it was just a blank video.
Solve’s Neil James concluded:
“You could swap in any video, any brand, any objective, and we’d be able to spin a dazzling PowerPoint fable with elegant charts, eye-popping numbers, and poetic proclamations. Oh, the engagement! Oh, the dwell! Oh, the buzz we achieved! But despite the narrative, despite the numbers, we accomplished nothing. Absolutely nothing.”
Metrics only matter if they’re actionable. I found an interesting post from Lean Startup author Eric Ries in 2009 on the distinction between “actionable metrics” and “vanity metrics” in startup life. Eric shares three takeaways that I think are relevant for any form of marketing:
“1. Measure what matters. It’s tempting to think that, because some metrics is good, more metrics is better. That’s why vendors routinely list the thousands of reports they are capable of generating as a feature. The truth is, the key to actionable metrics is having as few as possible. Detailed reports are useful when we’ve diagnosed a problem and are looking for clues as to what’s gone wrong. But where does that diagnosis come from in the first place? Actionable metrics help us realize we have a problem and point us in the right direction to start solving it.
“2. Metrics are people, too. Great metrics tools allow us to audit their accuracy by tracing reports back to the individual people who generated their data. This improves accuracy, but its more important effect is that it lets us use the same customers for in-depth qualitative research. Not sure what the numbers mean? Get the customers on the phone and ask them.
“3. Measure the Macro. Lastly, even when we’re split testing the impact of a minor change, like a wording or a new button, it’s important not to get distracted by intermediate metrics like the click-through rate of the button itself. We don’t care about click-through rates, we only care about the customer behaviors that lead to something useful, whether purchase, retention for advertising CPM, or some other measurable “success” particular to your business model.”
I would love to hear your thoughts on how to navigate actionable metrics versus vanity metrics.
Richard Warland says
February 28, 2016 at 2:45 pmGreat article Tom. Call me old fashioned, but the only metrics that matter to me are orders, sales and profit…
Rodolphe R. says
February 29, 2016 at 3:07 amagree but not always short term.
Ignacio says
February 28, 2016 at 8:46 pm@Richard That’s about the size of it. Social media has become an act of selling smoke unless you can relate the likes, shares and such measurements in an improvement of sales. The problem is that marketing people are being distracted by trying to get good numbers on the wrong metrics
Richard Warland says
February 28, 2016 at 9:46 pmWhat bugs me is – why do their bosses let them get away with it? Is it because they are digi-ignorant and believe the crap their marketers and agencies spout? Surely someone has to eventually ask about sales and bottom line!
Debra Askanase says
February 29, 2016 at 7:16 amWhat a fabulous example, that of the blank and silent 4-minute video! I’m shocked that people watched it all the way through, but not surprised that deeper engagement metrics (shares, comments) did not surface.
I have thought for a long time that social media Metrics That Matter is looking at 3 things, very similar to what Mr. Ries wrote: 1) Does it move you toward your SMART goals? 2) Does it tell you what people want to talk about and care about? 3) Does it help you identify who cares the most?
All said, it is mostly about actionable items, but also about knowing whether or not you’re sharing content that is relevant and meaningful enough that fans/stakeholders share, comment and then take action.
Jonathan Goodacre says
February 29, 2016 at 7:39 amBrilliant. Actually the frightening thing is how easily people believe what is put in front of them. There are several reasons – one is that you expect experts to be experts because that’s why they are presumably standing in front of you. Also, many people don’t understand basic statistical or data ideas – what are the measures actually measuring (a percentage of what exactly?)? Then, many people only see what they want to see, and interpret it within their own frameworks – ladder of inference is the key diagram required here I believe!
It’s incumbent on us as researchers, consultants, marketers etc to be conscientious and use data for our own expert interpretation but with honesty and integrity.
Increasingly, I reckon that if something looks wrong it usually is; it’s not that we’ve suddenly discovered some remarkable new phenomenon (sadly).
Peter Wilson says
February 29, 2016 at 7:47 amTrack the metrics to sales – its the ONLY real ‘metric’ so any lead generated needs to be tracked through to a sale/repeat order. Yes, I know long sales cycles can make that difficult but long sales cycles also give you time to make sure leads are passsing through the pipeline.
On a second point I work a lot with smaller companies – and vanity advertising / promotion makes the owners feel good because they think folk need to see, or want folk to see, them in a specific journal etc. Costs can often be well reduced or made more effective by taking the vanity out of the equation.
Ethel Pole says
February 29, 2016 at 10:37 amSo ‘metrics’ is the ‘new’ word for ‘statistics’, which as we know can be used to prove anything. 9 in 10 agree.
ajinkya says
February 29, 2016 at 9:40 pmThe prime driver of marketing vanity is the powerpoint. It allows enough space-time for suspension of disbelief to take effect. That is when con of many kinds get perpetrated.. the one you discussed with that lovely example included. Here’s a thought I shared briefly about. essentially, answering the question ‘if the medium is the message, then what is the message of a powerpoint?’
http://thejinxedone.blogspot.in/2016/02/powerpoint-is-con.html
Anahit says
March 7, 2016 at 7:47 amThe question really is, WHY did these people (1-in-5!) watch it?
Or, to be more precise, how do we know that they watched it at all?
If I were a person who is paid by the click and I have nothing else to do, I would just click on video and let it run on some background window without watching it and while doing something else on my computer.
Either way, there could be many reasons for this.
The experiment does however bring to the light the issue of the vanity metrics, which are becoming more and more vane. I guess, they are just too pretty to ignore – just like on this cartoon. Who would go after SMART ones, then the vane ones look soo darn good!
Abhay Dubey says
March 9, 2016 at 1:25 amCouldn’t agree more, especially when social media numbers are the new yardstick for marketing success. I just wrote about this on LinkedIn. Some cliffnotes from my thoughts.
– Understand that data helps build the story but isn’t the story itself
– Be aware that different metrics matter to different businesses
– Instead of shunning vanity metrics altogether, use them to track unexpected surges and dips to understand the bigger picture
– Accept that success takes time and not all numbers give ego boosts. Instead of relenting to the temptation of spoofy vanity metrics, keep focusing on the numbers that matter
Here’s the long-form version. https://www.linkedin.com/pulse/vanity-versus-actionable-metrics-what-you-measuring-abhay-dubey?trk=prof-post