Advertising Statistics Suck

This post is a continuation of my previous post How to Value Advertising.

Specifically, it’s a reply to Andrew Eifler who posted the blog post I responded to. He raised this point:

On the subject of variables and, as you point out, there can be quite a few – i think one of the biggest issues is how we quantify presence on each media channel. Universally the units that are used are “GRPs” or Gross Rating Points which are the product of “Reach” and “Frequency” against your target audience. For advertising measurement to really progress we really need a new unit of measurement. The system of GRPs worked great when the only media options were TV, Print, and Radio – but in today’s world, with such a fragmented media landscape, there really needs to be a more fitting measure. Maybe something like “Persuasion units?” Interested to hear what you think about this.

- Andrew Eifler

In general, I doubt I could come up with a decent replacement statistic, simply because the data is so poor. I agree, however, that the current statistics you use – GRPs, and also TRPs – are woefully bad.

Why GRPs Are Bad

The statistic Gross Rating Points (GRPs) is calculated by multiplying percentage reach by frequency (is that average frequency?). Now, this is all well and good if all you’re interested in is "impressions" as in "banner ad impressions." But the experience of NON-obnoxious (overlay) and NON-personalized banner ads should lead people to be VERY skeptical of the worth of impressions as something useful.

Click-through rates on banner ads are what, 0.2-0.3%? If "actions" (like clicking through is an action) on TV ads are similar – I wouldn’t be surprised – that’s very low. And what’s the conversion rate after that? 5%? 10%? Depressing, but I suppose it’s beside the point.

A more important observation is that (i) people will have variable marginal returns to seeing an ad repeatedly, and (ii) the distribution of view frequencies are highly unlikely to be normally distributed through the population or segment. In the case of (i), I think that the marginal returns are likely to resemble a S-curve as well; of course if your ad is particularly irritating, it may tend negative after some additional inflection point.

I hope the current methodology takes (ii) into the account; I would expect some members of the population to be more likely to see an ad more times. I suppose you can mitigate this by segmenting the population in the right way (e.g. segment by number of times they have seen/are likely to see and/or respond to the advertising). Otherwise, you’re asking to be mislead.

The Problem of Advertising Statistics

However, there’s a deeper problem with such “industry standard” statistics that do not measure the result they measure the deliverable. Or: they do not measure what the customer cares about (increased sales); they measure what the advertising did. “We showed most of the people you care about this ad 3 times – and surveys (?) indicate that consumers remember the ad!”

Sure, I get that’s the best you do. You’re not selling increased sales; you’re selling something very specific. If the company gets increased sales, good for them; if not, well they decided to do it in the first place. Hell, figuring out how to grow sales is (allegedly) why company executives are paid so much.

But, it would seem to me, part of empower companies to make their own choices about how much to spend is giving the information they care about – “How much did the advertising campaign increase my revenue and profit?” – not the deliverables (which are only really of interest internally to the advertising company). Making the process transparent – letting the company buying advertising know what advertising was delivering to their market – may (i) seem like a good thing to do, and (ii) help justify fees to the customer, but it’s really not a very bright idea.

An Unfounded Extrapolation

The reason is simple: it reduces the profit margins of marketing companies. Oh, not in the short term – but in the long term. The selection pressures for marketing shift from making the most effective marketing to making the… well, less effective marketing. If you can make the process less effective, you get paid more. At best, efficiency will stop increasing.

The approach should be reversed. Marketing companies shouldn’t sell different product lines – “Yes, you can spend $3m on TV, $1.5m on radio, $1m on billboards, and $2.5m on online advertising, we can do that for you” – they should be selling increases in sales. Hell, if you really wanted to motivate an advertising company you’d get them some percentage share of the increased revenue attributed to advertising (though with provisions to prevent gaming).

Sure, yes, I know I’m both reaching a bit and have no empirical evidence to make such an allegation and information about how (digital) advertising is changing things implies the opposite. The amount of innovation occurring in digital advertising (albeit, sometimes creepy innovation) is staggering. The allocation algorithms behind Google’s AdWords and AdSense programs are mathematically guaranteed to lead to the most optimal outcome for all involved; the personalization possible with customer tracking (e.g. DoubleClick) is getting there; the shift to Actions and not Impressions is coming fast, etc. I just don’t like statistics like the GRP (unless I’ve completely misunderstood it…).

Screw the Literature

Of course, it’s not like the (academic) literature is any better. I dipped into a couple of (mathematical) marketing journals earlier today when I was researching the (earlier) response; I lost most of my references, though, and as this is not an academic paper I’ll refrain from re-locating them. The upshot, though, is that modern mathematical and economic accounts of advertising assume that (i) you can segment your population well, and (ii) that all segments are homogenous (note that (ii) implies (i)). Is this accurate? – I would be terribly surprised if the segmenting was that accurate.

Given these constraints, most models assume that (i) you represent advertising as contributing to some sense of “goodwill” among people, and (ii) in the absence of advertising the amount of goodwill a customer feels towards you or your product declines (which is a great way to justify advertising! –> apparently it’s a well-established empirical pattern; I’d like to check that).

This is known as “inventing a variable which doesn’t really exist.” The scientific justification for it is that “goodwill hypothetically relates to several variables – known and “latent” (which means “unobservable”) – all of which are correlated, such as goodwill in an “index” for those other variables.” Actually, what they would probably tell you is that goodwill is a latent variable that can be distilled via structural equation modeling from some empirically observable variables – but it’s a distinction without a difference. Mostly.

Now, I suppose goodwill is a better thing to try and measure than GRP. However, it’s still (i) artificial, and (ii) has nothing to do with revenue or profit. Thus, the literature is little better.

However, one good point (which I had not considered) is that given these assumptions, then the approach you take is influenced by whether or not you are trying to maximize goodwill at some point t, or if you are trying to maximize the integral of goodwill over the advertising campaign. The obvious example of the former is in selling tickets to some event.

Closing Thoughts

Measuring advertising is hard. So the tools you have are limited.

However, I think the statistics cited should have more to do with what the company buying advertising needs. That may be revenue and profit, of course – but it may be something else. I mentioned aspirational advertising in the last post; I have no idea how to measure that (except really crudely, with surveys and interviews).

And I think the statistics used should have less to do with impressions, unless you’re trying to improve effectiveness (“Our GRP is 250, but sales only increase 0.4%! Something needs to change!”). It’s certainly never something you should show the customer.

Statistics related to inputs are useless. GRP, and similar statistics, look pretty much at what you put into the campaign – just like college rankings look at what goes into the colleges (SAT scores, money, etc). They don’t measure outputs, e.g. how successful each college student is, or how much advertising increased sales. Why? Because it’s hard to measure.

But abandoning something just because it’s hard is no way to live; and adopting an inputs-based measurement process will do nothing but increase cost (like it’s done for the college industry).

Unless, of course, that’s what you want.

---

Also, I have to confess that the reply to this I wrote earlier was lost when my computer crashed… teach me to use something lacking autosave.

Lying in Job Applications

Knowledge@Wharton has some very nice advice on the problem posed by exaggeration and embellishment of former accomplishments.

The core points the author makes are:

1. Everyone embellishes a little (and if you don’t, you’re at a disadvantage of those who do – the tyranny of asymmetric information!).
2. People don’t necessarily do it deliberately. Memory is unreliable, and there are different norms in different locations/industries/companies.
3. Employers really don’t like it.
4. It’s becoming a lot easier to both record and find (Google) transgressions.

The overall effect of the article is to discourage the embellishment of statements that are easy to check, and nudge people toward those which are “minor.” By minor, the author seems to mean “not binary;” that is, you either did or didn’t get a degree; you either did or didn’t hold such a position, etc.

The rest of the post is divided into two sections: they have nothing to do with each other. I just had two reactions to the piece.

Personal Reaction

My take is that asking people to accurately represent their experiences is like asking money to rain from the sky. Memory just doesn’t work that way, and cognitive errors (such as cognitive dissonance) ensure that people’s memories will always be self-serving.

Now, I don’t like to embellish, personally, because I fell like I’m lying. If anything, I tend to understate what I’ve done and expect the other person to know the unstated difficulties that needed to be overcome (an OK strategy when talking to experts in an area – I blame my years of academia, where people actually knew what I was leaving out).

But in general, and when looking for a job in particular, either understating one’s accomplishments or telling the absolute truth is more deceptive that embellishing a bit. If a hiring manager expects the interviewee to embellish, then they’ll discount all statements, since they have no idea which are embellished and which are not. (Also because a false negative is better than a false positive – better to drive away someone qualified than to hire someone incompetent).

As a result, considerate advice to the job-seeker must be to embellish a bit, but not too much… just around as much as everyone else is.

The Bigger Question

Now, there’s a bigger question the article raises. To what extent do these embellishments matter? Or, worse, qualifications at all?

Take Marilee Jones. The article says this:

…a former dean of admissions at the Massachusetts Institute of Technology and the author of a popular guide to the college admissions process. Although she encouraged college applicants not to overstate their accomplishments, Jones resigned from her position in 2007 after it was discovered that she had fabricated two academic degrees on her initial job application in 1979 and added a third later on.

Wikipedia reveals that (i) her book received terribly critical acclaim, being featured on the NYT, WSJ, etc, and (ii) in 2001 she received MIT’s Excellence Award for Leading Change.

In other words, Marilee Jones was a highly capable and highly effective employee.

Not having – and even lying about – her degrees did not detract from that.

Another example that springs to mind is former RadioShack CEO David Edmondson. Mr. Edmondson lied about his college degree, as well, but was also highly capable and added a tremendous amount of value to RadioShack (hired in 1994, won multiple industry awards, promoted internally to CEO). Apparently the trade rag Advertising Age recognized him as “one of 100 top marketers.”

These two examples have something in common:

They lied about their college degrees. Perhaps they lied about other things, but…

… that raises the question of whether or not college degrees made any different to one’s competence at work? Are those four years wasted?

It’s a scary question, as someone who’s just graduated (and currently unemployed!); and a serious issue for an industry whose pricetag has skyrocketed for decades.

(My answer deserves its own post, but: Sort-of. Employers use college as a way of filtering applications so they don’t have to determine the real qualifications… it’s a proxy for competence).

Evolution

This post is very confusing because it has nearly no structure; it is the jotting down of a few thoughts. Please be warned.

In my opinion, evolution is the most powerful framework for understanding the world – and what works. Evolution is the way the universe works.

I will go so far as to say: Anything that does not adhere to the precepts of evolution will either fail or stagnate.

Now, I don’t mean (just) biological evolution, the kind Charles Darwin wrote about. In fact, I know relatively little about the modern synthesis and molecular biology. Evolution, though, is not restricted to biology.

Evolution is simple. The entire theory can be explained with three words: Variation, Selection, Heredity. Make a change, see how well it works, and then do it again.

The change does not have to be random (and some modern evidence suggests that genetic change is not random), the selection criteria do not have to be immutable, and inheritance just means “repeat, because it worked.”

There is a tendency for people, such as (some) evolutionary psychologists, to attempt to reduce behavior to biology. This is not an evolutionary explanation; it is a reduction from one level of explanation to another level of explanation. They are very different things.

An increasingly popular approach in business is called “iteration.” While most noticeable in web startups (e.g. Basecamp), businesses everywhere are adopting it. The approach comes down to: Make something fast, show it to the customer, see what they think; fix problems, make more changes, see what they think.

Iteration is evolution by another name. Vary your product, measure it against the selection criteria (customers), then change what doesn’t work and improve what does.

On a side note, the concept of “iteration” is incomplete. It largely ignores the selection criteria – it takes that on faith. A more rigorous examination of the selection criteria makes obvious the similarities (and differences) between real customers, focus groups, surveys, etc. People should pay more attention to selection criteria.

The consequence of accepting evolution is that “designed” becomes a dirty word. Centrally managed, independently created, committee-verified… are all models bound to fail, compared with an evolutionary approach. Certainly, they may succeed; in fact, it’s inevitable that some will succeed. But success is far from inevitable.

If you want a system or a process to succeed, design it in an evolutionary fashion.

Economists like to claim that the market system is the most efficient allocation system. That may be true, given the highly idealized assumptions one must grant them before they will say that.

However, a market system is a two-dimensional reflection of an evolutionary system. Sellers in a market subject an array of similar but different goods to the selection criteria of buyers. Successful products are purchased in larger numbers; the adaptations which made them successful are then copied.

A traditional market analysis lacks the notion of hereditary, or of change between products over time. Once you add the notion of time in, the concept of a market is functionally identical to (but less flexible than) an evolutionary explanation of economics.

I’m obsessed with evolution, right now. I’m sure you can tell.

The biggest reason is that evolution is arational. It is not irrational, or against reason; it simply has nothing to do with reason. It functions independently of both truth and God. It is a system which progresses – changes, at the very least – without any component of the system (or the system itself) having (i) access to the truth, (ii) direction or intent, or (iii) awareness of the system. It changes endogenously, and requires very few assumptions to work.

It’s beautiful.