FOE2 – Metrics and Measurement

Metrics and Measurement

Panelists: Bruce Leichtman, Leichtman Research Group; Stacey Lynn Schulman, Turner Broadcasting; Maury Giles, GSD&M Idea City

As media companies have come to recognize the value of participatory audiences, they have searched for matrixes by which to measure engagement with their properties. A model based on impressions is giving way to new models which seek to account for the range of different ways consumers engage with entertainment content. But nobody is quite clear how you can "count" engaged consumers or how you can account for various forms and qualities of engagement. Over the past several years, a range of different companies have proposed alternative systems for measuring engagement. What are the strengths and limits of these competing models? What aspects of audience activity do they account for? What value do they place on different forms of engagement?

• Jim Nail – Cymphony.  do tracking.  My passion for this is because internet marketing never really hanged things.  Word of mouth is a change to things and I got really excited.
• Maury Giles: was a newspaper journalist, got into consumer opinion, became fascinated about tapping into these things.  then running political campaigns, market research.  You have to be accountable with politics, a great lab of learning, how to be accountable. Passionate about taking a holistic viewpoint about connecting with audience, how can you do specific metrics to understand what you are doing and how do you optimise.
• Stacey Lynn Schulman, runs the ad services for the network. Just joined in last month.  Before them on agency side, Interpublic, at Initiative a media agency.  Consumer experience.  about understanding tv audiences. This has been measured the same way for years.  Has tweaks only.  My reputations was built in hit predicting…what was going to be big.   Now fragmentation is ridiculous, it was harder to predict hits.   Clients were getting angry that we could not identify what was going to be a hit.  It made us think…about how to do this going on.  How audiences are moving across spaces and how we measure them
• Bruce Leichtman, Leichtman Research Group.  We specialise in future of entertainment measurement.  We have to start with the present – combining research form multiple sides.   All time in market to understand what consumers are doing.  You can never use the sample of one…it is not ‘me’.  You are not the mass.   My passion is finding out what is America.  I really enjoy entertainment and that is what I focus on.
• (Sam Ford is moderating)
• SF: the writers strike brings up a question, about new media, about valuing them,  you can’t apply tv metrics to these spaces.
• BL: a lot of people applauded at the end of the writers strike video.  It is based on hockey stick curve of revenue.  what if that is wrong.  that money is not going to be made in the near term.  Maybe it would be better to strike in 3 years?  We have to be careful when making a premise based on hype – how is the market going to evolve.  Is it really 4.6billion?
• SS: It’s interesting; how many times have we seen research, with these growth curves and they do not happen.  they are predictions, not fact.  We don’t know, but I will say content is king, so they do deserve a stake.  What kind and what degree is a different question..can we hang hats on projected numbers.
• BL: Does the projections really mean money?
• SS: we can’t quantify how successful this stuff.  it is a new science.  without those metrics, don’t know.  do we fuse data or not, can we track across devices? 
• MG: short term issue on predicting, not sure  but more to a definition of what is success…this strike can push to forefront the debate of defining on traditional terms about eyeballs and have that frame it or are we going to shift the debate to the consumer opps..how big is that audience..they can pull down and use the content in so many different ways.  the old paradigm will fall down in trying to determine value. Passalong and viral etc…how do you account for niche.   how do you monetise this?
• JN: I have to stand up for the analysts….never believe those hockey sticks.   those kind of predictions are opinions in numerical form.  We try to focus them on our assumptions and if clients agree with them. you have to watch the consumer.  I lost sight of the consumer when I was doing my big projections.  The tech changes far faster than the consumer or the industry.   What is good about this strike..they have to get away from ‘audience’ thinking and get to individual thinking.  They (the writers) got screwed on VCRs and they will not do that again. It will drive to individual.  those numbers are out there.  it will drive other changes in metrics as well.
• SF: so keeping in mind the general audiences; I found I would be talking to people on coasts and there is a disconnect in what people have..I was on dialup in Kentucky.   On the revenue stream, is it someone’s fault that there is a good one that can be quantified.
• JN: assigning blame is an extreme.  these are the structures and you will not change them overnight.  But the networks have the choke on distribution, it is a threat to their business model.  If everything can be distributed in many ways the networks have no role.  it will take time to work out the transition
• BL: perspective is important.  Looking at today, with Comscore data on legal online video.  6bill mins/month to 24bill mins/month in 2 years/  That’s 6 minutes/users per month – so not that much. It is growing..but not taking away from tv…it is very segmented.   Twice as much time is spent on Don’t forget the lyrics then all the time on YT on a given day (that’s comcast on demand).  you have to keep things in perspective.  it’s about evolution, not revolution.  The typical consumer is not there yet.
• SS:  if you go to any marketer today, the perspective, about all the changes, new techs, audience fragmentation, etc..we would consistently hear from clients about the next shiney object…the ‘get me one of those’ strategy..  The industry gets the sense that everyone is in these spaces..80% of their mindshare is on the bright shiney objects.  but 80% of dollars is in traditional media.  They want the shiney stuff..with GRPS and frequency of 18-34.  They want the shiney stuff expressed in mechanisms they understand.  and then it breaks down. they can’t say the money is working as the metrics are not there.
• JN: look at online ads..timeonline vs spend.  now we are up to maybe 8% of budget.  a lot is related to GRPs, buying eyeballs instead of audiences.  If they do not have a way to understand these things then tey shy away from the decisions.
• SS: you also have the marketer that has the opposite…the lean back arms crossed people.  Who use the TV success as this works…it works for them, you are trying to get something new.  Those are also the clients that fire you the next year as you never got them a new idea.
• MG: all these come from a discussion about making money from a distribution channel – about reach.  But now the channel is individual centric, not mass.  The dialogue has to change, away from distribution to relevance, to theindividual consumer.  For the last 50 years we have been making an apple pie, we put what was best there and get the pie to many people.  Now it is an orchard, not a pie..the audience want to make their own things, pulling down apples and mkaing their own stuff
• BL: is that today?  The biggest audience ever on cable was Disney High school Musical 2.   There is still mass.  Some people want what I want when ..others just want stuff.
• MG: appoinment tv is not dead but I make the appointment.     There are different ways.  To truly revolutionise the way of monetising, you cannot have the same discussion as there was in the past – you have to be about relevant ways of connecting with consumer
• JN: you can’t fall into the trap of the world is all this way or all that way
• MG: the measurement debate is all one way.
• JN: alive sport event is appointment.  Thursday night entertainment is not.  HSM is appointment for the kids.
• MG: you can still timeshift between the event and when you will next interact with people.
• SF: back to the writers strike….we are talking about making a deal about where we are today or for longterm..  Take a look at Jericho, the fans saying they were watching online, on DVR, why did their eyes not matter as much.  Can there be a metric across these platforms or are the platforms so different
• SS: the argument about engagement is one attempt.  Looking at buzz about a show.  We launched Prophesy 5 years ago,   to see what people were looking for.  We were able to predict a number of hits that was different to the industry stuff.  Most people in the industry thought Lost was not going to work…we did the analysis and it was 2/3 in the rankings of discussions.  Then it premiered with 20 something shared.  there are lots of shows that had small engaged audiences…Veronica Mars, Friday Night Lights., Jericho, The Office..big in buzz.  You can’t always tell by size of audience the longer effect.  we should be looking at small, engaged fan audiences.  We forget that consumers are channels as well – they distribute.  They do different things at other times.
• SF: I keep hearing the word engagement? Should we take more quality of viewes?  So what is the engagement?  What do you mean by the term?
• SS: we have been through lots of ways of looking at marketplace, it is fragmenting, they can skip the commercials.  Industry is looking at ways to reavaluate the content to make it valuable to advertisers.  It is not the size (that is getting smaller).  we have 2 things – integration and how valuable it is, how engaged it is., ie they like it so much that they watch with the commercials.  But it is not a currency, all it tells you is a percentage of the audience are invested.  But if you do not know WHY they are engaged, then may do things that could insult the audience and disenfanchise them
• JN: the advertising measure group have siad the way we have been thinking about the audience is wrong.   We now think people have an emotional reaction and then get info to rationalise – so it is not about getting info out, it is getting emotion out.  Engagement can’t be a currency, it is too hard to measure, can it be meaningful?  You need soem sort of proxy, the size, time spent etc.
• SS: we looked at that – is it points of contact…
• MG: it is about connection with content. We tried to look at it from involvement, at the channel level.   What are people doing in the channel?  We have multiple indexes..you have to understand how engagement plays in the mindset of the consumer.  You have to map the consumer decision cycle, you then can build a communications architecture along that decision making path and know when you can touch them, the jobs you are trying to do at different points in the decision cycle.
• JN: it was hard enough when only a few..now it is almost unwieldy with multiple channels
• MG: when does a specific channel pay a role? That’s what you have to think about engagment.  it;s not jsut the content, it’s also the impact of the channel
• SS: in 99 we did a test of interactive advertising.  We found that the level of interaction was lowered in primetime as they are looking for content, not ads.
• JN: they wanted to watch
• MG: this is when you need to understand mindset of consumers..tey are less likely to be engaged at primetime.
• BL: the goal is always to sell..if it does not help sell the product then the advertising time is not worth buying.
• JN: the branding , the tomorrow stuff is hard to track.  building the brand..how can you relate that to measurement.  Branding does work  The hard thing is quantifying that emotional level.
• SF: Buzz is not ness about audience measurement but is about discussion and loyalty. and criticism is a positive metric.  (what Jim had said before)
• JN: I do get asked if I can use it to measure impact of a viral..I say no, they don’t tend to talk a lot about advertising.  But what they do talk about can be important.  Crit is good, as they are engaged..’we like you product but we don’t like this’. If they did not care then they would not want you to get better by telling you what is wrong.  Research I have done sows that people are so fed up with advertising and they do not even engage it, they trust people more.  Interested people,, even with negative, want to get involved and fix it.  It is no longer about interruption…I can enlist consumers in development, I can get feedback.   I can get feedback form the stuff in the market.  Marketers have to let go of the total control.
• SS: it’s not a one way conversation anymore.  it’s 2 way
• JN: it always was that way…it’s the co-creation of messages.  We did not know it was happening could not find it, but we can now.  By understanding that we can build stronger relationships.
• MG: it;s no longer saturation as well.  Now the channels that we can connect with is as much as part of the connection as the message is.  It;s a cultural shift, a blending of advertising and marketing.  We have to be smarter in engaging with the channels that facilitates the interactive dialog.  The masses are not fully there but it will hit us faster than we think
• BL: but what if you plan too fast?  What if you get prepared for something that does not happen.  We should not be nebulous with the future…we have to go 3-5 years…We have to understand the nect 3-5 years.  So lets take DVRs/  Not all have DVRs.  over 75% do NOT have a DVR.  1.6million have a standalone tivo these day, with direct tivo there is 4million… There was not a consumer pull for DVRS.  It took off when putting in a satelluite box.  And then when put it in a cable box..then when added to HD boxes.  It was push.  About 5% of viewing is ondemand.    Interest levels in DVR has not increased in research..it will grow to 50% by pushing,  I think 15% by 2012 will be ondemadn viewing, even when  world with half households have DVR.
• JN: I’m going to violently agree!  The real point is want the consumer is really doing.  Forgot the numbers, 70mill blogs etc, look at minutes of times etc.
• BL: the 15% differs by timing..primetime is being hit.  Original stuff hot harder.  Tivo has a sample of 20k..but the goal of the sample needs to reflect the populations,.  So tivo users do not represent dvr users.
• AudQ: when you are making your projections on adoption, how do you figure in the aging of the population?
• BL: that is where it differs..you have to set the parameters on time.  How they affect the world in 2012 is different to 2025.     Factors of age do not really effect in 3-5 ages.
• SS: I hate seeing research about teenagers..it may be true that they do things now…when they are crushed by life with a dead end job and cannot pay the rent and all they want to do is watch Tivo and do not have the energy to multitask!.  Because they do this today does not means they will do in an adult mode.
• AudQ: there’s discussion of word pf mouth but not a lot.  With net we can track consumption and dispersal.  New forms of advertising and distribution comes across.   WOM..people you trust.  how do you see WOm affecting how you do your jobs.
• SS: at this moment of time not everyone is out there making a trail.   I see us getting the set top box, the cable stuff, the broadband, you can connect those dots about what they are doing and what they look for.  I want to get my hands on these databases….look at the connections.  it’s finding the commonalities and the connecting tissue, that creates those groups.
• JN: part of WOMMA, we look at things about what you should measure and how to.  80% is still offline.  You are still getting a proxy.  Look at the Never Ending friending report…the shocking revelation is that you can stick banners etc the real thing is to put widgets and logos and brand assets that your users can take and put on their profile which reaches far more people.  70% on total impact of Purchase intent was from the widgets on friends pages, not the advertising.
• MG: as a marketer you have to think about what jobs the channels do.  Facilitate does not mean create..you cannot be doing the stuff, you have to facilitate the conversation, Think about how to make it relevant, given them the stuff to create their own material, let people take stuff. Nike and Apple with the running shoe.  You need to know the job of each point and then you can track it.
• SS: people are concerned that the big reach things are going away.  The world in general is pretty fragmented, not just in media.    We have to be able to tell people who we are and what we do, the things we use has to come from pop culture, a common reference frame.
• AudQ: what are other measures of engagement?
• MG: what do you want to do with the channel. Metrics are based around what you want them to do.  The limit is about how you define what you are doing.
• JN: there are things in marketing that lead to future sales, not ness tied into direct ROI.  You have to build awareness first. 
• MG: there are some metrics that will change how things will be done.  Marketing mix model etc.  a field that fascinates me, is using complexity science techniques to take non-linear models, to create environments that you can test in and try new things.  You create virtual agencies, you can introduce new things into it to see what happens.  the knowledge you gain about the decision cycle, you can create scenario planning tool to explore the new tactics. 
• AudQ: the measurements from the TV are closed, people do not have access to them.  Closed blackbox proprietary systems are good at somethings but not others/.  What do you think about an open source metrics system and what may it give you?
• MG: it may be more useful..but there is so much tied into it.
• SS: we have a history and a currency and it is difficult to move between them.  The currency is changing right now, to C3.  There’s so much processing that the data is not there for 3-4 weeks.   There are issues.  The way the currency is set up, there is a lot of dependence on knowing we are going based on where we have been.  There is a lot of money on it, it is difficult to predict if the currency changes.
• AudQ: you say consumers are another channel.  What about bebo giving a channel for their users, with content.
• SS: not aware of the bebo deal..in terms of the strike it is very layered there.  there is costs, we do not have the evidence that it is profitable…It’s too all over the place.  From a consumer experience POV it’s great but it is not as functional than what consumers are doing.  It is easier to go to torrent and do what you want to  Industries are trying to control content and not yet understood they cannot.
• BL: look at the office -you get the revenue from ads online  It could be a zero sum game, the revenue online could come from the tv revenue. it is not all incremental
• AudQ: any metrics that advertisers find convincing beyond eyeballs?
• SS: IAG have a trivia game, ask people on the content.  They sell the data to advertisers as a measure of commercial recall. about delivering environment that gets people to recall.  The methodology not liked -it incentivises people to watch the shows.  It’s accepted as there is no other metric that sees if people recall points
• AudQ: Monster have built a great brand..(person from Monster) we have tended to think that the offline media is better to build brands and online is better for direct.  
• MG: it clearly has a direct response, but can be an amazing source of facilitating an experience with your brand.  That experience you are creating is just as much as a brand builder as a TV add, but can be longer lasting.  We look at loyalty and brand affinity scores in people who have engaged in these activities.
• JN: online advertising ads a small incremental to brand building.  but online is reaching the right TV viewer.  online extends reach. 
• AudQ: are there any case studies that surprise you?
• BL: High School Musical – mass levaraged in many forms.
• JN: YouTube, MySpace, Facebook.  2 years ago it was all about blogs…now video…now MSp, then FB.  we were going to test a weeks worth of MySpace data – 50GB.  we could not scale.  We have been analysing the Superbowl ads, and when companies said they were going to be involved.  Look about them being kept secret vs announcing it.   The ones that did it before got more share before AND after the game – that was surprising.  The connectors see it and then they get people to watch it on the show and then people talk about it more.
• BL: AppleTv..a great case study.  analysts said 2million units…now they never talk about it!
• AudQ: any other measures, emotional context, that may have an impact.
• SS: Neurofocus using brain scans to see what people are responding too. they do it for adverts and shows etc.
• MG: Swarm and complexity science..marketing is like an organic system
• SF: so what are you doing that is innovative to answer these questions?
• BL: my job is to understand the consumer and where they are going.  Tech is way ahead of the consumer.  That is what people need..what is the time frame  How fast are these techs going to be adopted.
• SS: how to figure out how to keep commercials on levels similar to programme minutes.  there are lots of new methods in getting advertisers involved with the content, eg dinner and the movie, integration.  we have to find ways to keep you engaged.
• MG: we need to understand how they engage, the dynamics, triggers and stages of decision making, as we study and get it mapped out, then getting channel power scores and indices against the channels at our disposal so we can plan within the context of the comms architecture.
• JN:  we are thinking about aggregating.  It’s not about what is the audience for the TV, but about getting that amount of people of that time in that time period – can be across channels.  People get more engaged with niche content and there will be a halo effect that will go to the brand.  You may rather aggregate from lots of audiences instead of one big one.

Tags: FOE2, metrics, engagement