Welcome to Shift Happens – the art, science, and chaos of modern marketing. Each episode unpacks the forces reshaping marketing from AI and data to privacy, creative and performance, and asks experts how they transform disruption into advantage.
Nasser:
In this first episode, I’ll be joined by Dan Temby, the Senior Vice President of Technology and Analytics, here at DAC to talk about how marketers can shift from vanity to value metrics, to actually proving business impact.
So recently, I heard someone ask us, “we have dashboards full of metrics. But how do we know if any of this actually drives sales? Or if we’re reporting vanity numbers?” And that’s exactly what we’re going to explore today. How marketers can shift from vanity metrics to value metrics that actually prove business impact. And who better to talk to about this than somebody who knows a thing or two about vanity? My good friend, Dan.
Dan:
I wasn’t expecting that, but I don’t mind, I’ll allow it.
Nasser:
There, you’ll allow it. So let’s start here. Why do marketers still get stuck on vanity metrics like impressions for example?
Dan:
Right. Well, I mean, they tell a story, but they’re…vanity metrics are easy and they kind of feel like progress, but they’re not necessarily the same as real business impact. There’s a correlation often, but there’s not always a causal relationship. I think there’s inertia at play. A lot of businesses are used to recording those metrics. They’ve been around a long time now. Those metrics have been a staple at the boardroom table for many years. So that organizational inertia, forces us to sort of keep reporting on these same metrics.
And again, they can be an indicator, of what’s going on, but they’re not really connected to, to value the same way.
Nasser:
Okay. So, if you think of it that way, like, give me an idea of how this can be applied elsewhere.
Dan:
I told a story recently, my wife is a personal trainer, as you know, and, she yells at her clients a lot of times for weighing themselves every day. They check their weight every day. And that way to your weight is not a good indication of your fitness level. And that’s what people want. They want to be fit and they want to be healthy. And your weight is just an indicator of that. You can lose muscle and think that you’re losing weight, but you’re actually getting less fit, less strong.
And conversely, you can gain weight. And that’s actually because you’re putting on quality muscle tissue. So, I’ve heard her actually yell at her clients about, you need to eat more food to get healthy and lose weight. And that’s counterintuitive. And that’s sometimes what we find, in a marketing program – we need to focus on metrics that may not be satisfying that immediate need, that immediate behavior, that recurring, behavior that we’ve had, looking at those metrics and focus on different things.
Nasser:
I feel personally attacked with that story.
Dan:
Yeah. I was trying to not make it about you, but let’s make it about you.
Nasser:
All right. Thank you. So if I were to summarize a vanity metric tells you something, but not necessarily something that really matters, right?
Dan:
Not all the time. That’s right.
Nasser:
Okay, so as we think about what to measure instead, if clicks don’t equal customers, what should marketers be measuring?
Dan:
Well it’s really about value not volume. That’s hard. E-commerce is a tough business. You know, we’ve got a strong drive to make sales today. You’re only as good as what happened yesterday. As the holidays approach, all eyes turn to the numbers on an hourly basis, and I totally understand that. Right? But it’s sometimes in fits and starts.
That’s very important. But when you zoom out and look at the overall program strategy, we really try to look at how we can move away from that type of thinking. We’ve recently worked with a luxury menswear retailer where we’ve nudged the program, thinking away from how many sales have we gotten in this given period and are we getting the right people in to buy the right stuff at the right time?
Because when we consider lifetime value, it can be far more beneficial long term as part of a strategy to look at saying, did we get the right people who are inclined to buy more frequently, to buy the right kind of products that statistically will bridge them into buying higher ticket items? And if we can do that, it’ll sometimes contradict what we might see in a traditional how many sales yesterday report. But push us to a year from now or maybe even shorter time horizon, having a much better story to tell in terms of total value.
Nasser:
So how do you go about doing that? How do you shift that way of looking at the business and looking at business impact?
Dan:
It’s tough. I mean, you have to have a team that’s ready to think that way. And, and maybe move away from what’s traditional, specifically here we looked at, product pathing. So what products led to the high value stuff that we looked for? We looked at post code analysis to understand geography. And if there was a specific target, area that we should refactor the campaign around. And then ultimately that changed our approach to category level bidding, media targets, measurement overall. And really got us to help bind the connection between the activity digitally to those in-store sales, which were really high value. And again, it takes a bit of a team effort and then some education and some togetherness that’s required with everyone involved. It was pretty powerful when it comes together.
Nasser:
So you talk about being powerful. What was the impact of that kind of shift in mindset?
Dan:
I mean, numbers like 15% increase in overall ROAS over a given period of time. Again, if you narrow the window down on a given day, maybe not as good, but when you zoom out across a financial quarter or a period, we start to see the numbers moving and we wound up 15% ahead, by taking that lifetime value approach as opposed to the sort of what have you done for me lately approach.
Nasser:
So while clicks make noise and conversions carry impact, lifetime value drive strategy.
Dan:
Absolutely. Yeah. Absolutely it does.
Nasser:
And strategy has a lasting impact for sure. So how do we help clients move from dashboards to decisions?
Dan:
I think dashboards are important. Everybody wants to know, you know, what happened and where we’re going. And if the colors of the right colors and the lines are moving in the right direction. But I think if we can turn the data behind a dashboard into direction, there’s this constant struggle with analytics between are we telling a story retroactively, or are we telling a prospective story about what can happen next? And what we’re talking about here when we talk about data science and decision science and there’s science involved, it doesn’t mean it’s perfect.
It doesn’t mean it’s guaranteed. It means that there’s some rigor and some structure around it where we’ve identified patterns.
Nasser:
Sorry. Explain that when you say there’s science involved, it doesn’t mean it’s perfect. Like, I wrote two things at odds with each other?
Dan:
I mean, this is you can open up a can of worms and a broader debate about science generally here. But the very definition, of science as a vocation, as a field, as an approach, it’s about iteration and experimentation and eliminating what’s wrong just as much as it is about identifying what’s right. So it’s not about being 100% certain. It’s about establishing a level of confidence that gives you the courage to make a different bet.
Right. So if we can look at it that way, when we spend marketing dollars, we’re kind of gambling. We don’t know what’s going to happen tomorrow. Something major could happen tomorrow. we just don’t know, right? But there’s enough history and enough signals in our past to give us confidence to say, if I put money here, I’m probably going to get this outcome.
The more of that analysis we can do, we can get a higher level of confidence on that bet. No guarantees. But again, if everyone’s willing to say, hey, let’s make this bet because we might find a way to optimize or get outside some local maximum and find a new maximum. But that again comes with this scientific approach to experimentation.
Nasser:
Well, hopefully the bet that we’re making isn’t roulette, but something like blackjack, right?
Dan:
Yeah. Yeah, exactly. You can make a that’s a really good thing. If you’re making a…look at you go, you can look at the data that’s in front of you and help that to evaluate the decision based on math that’s going to decide on your next, your next move.
And that’s basically what we’re trying to do. And again, dashboards have the source fuel to do that. But they don’t help you make that next decision. That’s where the decision science comes into it.
Nasser:
All right. So let’s talk a little bit about that next step in that decision science. As we think about the future of analytics how is the analytics space evolving and what should marketers prepare for?
Dan:
Well, I mean, this is the longest conversation I’ve had in some time, but we haven’t said AI at all. So we’re saying for the first time now. And obviously that’s a big part of it. There’s no hiding the fact that AI and the availability of, like, amazing compute gives us the ability to do much more of this at scale quickly.
I think it’s the reason that this topic is so heightened right now. Everyone’s talking about this. They want to know. It’s not… you can’t just have data and AI and jam them together and get magic out of it. That’s not going to work. And we’ve suffered through that a little bit. We’ve tried and failed thinking that, hey, if I’ve got an unbelievable language model and I’ve got a pretty good data set, if I can connect the two together, I’m going to get great insights. And the truth is, you wind up chasing your tail for a long time.
Nasser:
Why is that?
Dan:
I think to get AI to work properly, you know, you need curated data, which is, you know, validated and correct. You need it to be connected so it can’t be disparate and isolated. You need things to be in one spot. But more crucially, as we look to AI and its influence on data science and analytics generally, is this idea of semantic layers.
How do we describe the data so that the AI model understands it? When our analysts go and look at a data set – they’re coming in with this implicit understanding of exactly why they’re looking at what they’re looking at, what it means, what the realities of the business might be, internal taxonomy and other language, complexities. So, to equip your data to work in the AI world, that creation of these semantic layers is very important.
And though semantically, you think of them as like, a handbook or a map that describes the business context of the data itself. And that’s very, very important for AI, so that you can use that in a really productive way.
Nasser:
So can you talk a little bit about how you’re applying that, kind of the semantic, description of data in, you know, I know you’ve been building for for a while now our operating system or agency operating system, with AI at its base and from the ground up, which we call IRIS. Can you talk a little bit about that? And how you’ve been working with the data for it to actually work?
Dan:
Sure. Remember when I said, if you just get data and AI put it together doesn’t work? That’s how I know. Because we’ve tried that a few times. We got there in the end, we ask an AI agent to build a scatterplot that brings together, you know, 4 or 5 metrics around a set of dimensions, and it took a million tokens and 37 tool calls to get this very pedestrian outcome. And then you ask a follow up question and the whole thing broke down.
So we’ve worked very closely with our partners, at Snowflake, and our internal data engineering team to understand the nuances involved in creating these playbooks and these semantic layers. And they give us a really powerful ability to really turn on what we’re calling sort of conversational intelligence. It’s like having someone in the room that can perform these analyses, execute these models, give you a valid opinion. With all the confidence scores that you would expect, in a sort of real time nature, which makes it a very powerful tool when you’re especially when you’re in rooms. Earlier I talked about being in the right rooms with the right people and getting everyone aligned around, a certain vision. It can make it easier for them to understand when you’re working in a conversational context and it’s not, okay, here’s the way we want to do the data scientist run off and do that in isolation.
Being able to do that together with the people that are involved in the business is very powerful. So, IRIS is a really interesting project. And we’re leveraging a lot of really cutting edge technologies, but absolutely, you know, carefully curated AI models with the right instructions down to semantic definitions of our data and then our data put in the right spot, connected the right way. And when those three things come together, you can get really powerful outcomes.
Nasser:
So, in other words, AI amplifies strategy, but only if the data speaking the same language.
Dan:
Yeah, you know, we need to interpret and it’s and it’s powerful and incredible as this technology and I urge anybody listening to this, if they haven’t really considered that union of the data and the AI, especially if they want to start talking about advanced analysis beyond trivial metrics, like we started this conversation about vanity metrics, right?
That’s easy. We’re talking about making real analyses for big decisions that are going to make big bets on large budgets. You’ve got to get these pieces in line. And exactly that’ll that will really be an there’s not going to be a marketing or media analytics strategy that doesn’t have this in the middle of it. You know, as we go into 26 and beyond, that’s just not going to happen.
Nasser:
Okay. So here’s the shift. Vanity metrics might make dashboards look busy, but value metrics prove impact. Smarter marketing means measuring what truly drives the business forward. Now make it happen. Follow Shift Happens where you get your podcasts. Leave us a review and share it with your team. Thank you, Dan for your time today.
Dan:
Thanks, Nasser.
