Monday, October 6, 2014
More on Listen First's DAR (Digital Audience Ratings) courtesy of Erin
Greetings,
As luck would have it, Erin-- who works at Showtime-- had the chance to see Listen First pitch their services last week-- including providing data.
I've linked a folder with the relevant documents.
I must admit that after closely examining the material, I'm somewhat skeptical of the validity of the data.
A few questions/concerns that clients might have would include:
1) Why is a Facebook post = Google+ post? (In what world are those two social media equal?)
2) Why is Wikipedia used as a measure of engagement at any level?
3) Why are the YouTube numbers consistently off the charts-- especially relative to the Facebook numbers? It seems to me that exposure is not necessarily equal to engagement. For example, if I watch a YouTube clip, is that the same as producing a parody of a scene from a TV show? It seems to me that the latter offers a level of engagement significantly higher than the former.
4) What about 'mixed' engagement? What if someone tweets an Instagram photo and I like it via Twitter? Does that count as engagement via Twitter, Instagram, or both? Or what if I watch a YouTube clip posted via Facebook and comment?
5) Do the digital audience ratings differentiate between organic (user-based) content and corporate (studio) content? If not, why not? It would seem that user-based content shows a higher level of engagement than do likes/shares/re-tweets of studio-based content.
6) In the 7-day DAR (provided in the folder linked below), how is it possible that 'Dancing with the Stars' has more than 2 million YouTube units, ZERO Facebook units, and more than 500,000 Google+ units?! This doesn't pass the face validity test. This seems to suggest that the rank order of those three social media is YouTube-- Google+-- Facebook. Everyone reading this can tell you that Facebook is #1, YouTube is #2, and Google+ is a distant, distant #3 in this list.
7) Erin did some investigative work for 'The Mysteries of Laura' and would like to know how it's possible that YouTube is the primary driver of activity when, collectively, the YouTube videos posted have only garnered 50k views three weeks into the season?
8) Is it really that likely that Dancing with the Stars, with close to 4 million overall units of engagement, offers more than 4X the engagement of The Mysteries of Laura-- the next closest show?
I have several more questions (like why is it that Listen First's DAR ratings aren't anywhere near RENTRAK's "stickiness/engagement" ratings when they both purport to measure the same social media?), but I'd like for you to take a look at the ratings and post any comments/questions you may have here in the comments' section.
Thanks again to Erin for this info.
Here's the link:
https://drive.google.com/folderview?id=0B2Dv4MQMdbN4ei0yWVctbmdHQ1E&usp=sharing
Subscribe to:
Post Comments (Atom)
Alright, well my first question will piggyback off of Jack's complaint/question about measuring Wikipedia at all, which I definitely don't understand. Does Listen First only offer their packaged DAR ratings, or can they deliver a rating based on only the social media you choose - say only Facebook and Twitter, for instance. It seems like there is an issue with the Youtube numbers, and definitely agree watching all or part of a clip is not engagement, and Ithink all of the other sites measured barely touch the impact and reach Facebook and Twitter have.
ReplyDeleteReading all of this also reminded me of a thought I had while watching NFL football on Sunday - about the interactions between content producers and social media.
I was watching NFL Redzone, the channel that shows all the NFL games live, cutting from one to another based on factors like how close is a team to scoring, how close is the game, etc. There were only three games at 4 PM (Arizona Cardinals vs Denver Broncos, Kansas City Chiefs vs San Francisco 49ers, and New York Jets vs San Diego Chargers) and as halftime approached for the games, the host on NFL Redzone asked viewers to tweet with any of three hashtags for which game they wanted to see after the half. Cardinals/Broncos and Chiefs/49ers were both close games and the Jets/Chargers was already a blowout, but the viewers by a wide margin chose the Jets/Chargers game. The host announced this, but it seemed to have little impact on their production, and they still barely showed this game.
While I think it was the right decision not to show it, I feel like this strategy is wrought with flaws. If you show the bad game, you're doing a disservice to all your viewers, if you don't, you're going against the social media participation you asked for.
So as I said, I found it to be the right decision, but found their initial "hashtag the game you want to see" request to be a cheap and transparent ploy, and makes me think overall that content producers, while they need to elicit digital engagement, need to be very careful about how they react to it, and what they do with it.
Sorry to go off on a tangent, but this really bugged me - curious to hear thoughts from others.
I agree completely with the fantasy football analogy. What one is really doing here is creating a scale-- in this case a weighted scale-- to get a numeric value that represents a general assessment. As for the NFL Redzone, I watch it every Sunday and it's silly to leave it up to viewers for the simple reason that the biggest market teams will always win. They have producers who make the decision, so they should just show the best games (and cut to the Jets/Chargers game when someone is in scoring range).
DeleteHey Scott - interesting that Redzone would pose a question like that because as a consumer watching and opting in to their call-to-action, you'd expect them to follow through. It does bring up an interesting concept of an audience dictating the programming though and whether that will every be plausible in a non-competition setting.
ReplyDeleteAlso to answer your question you can only buy the DAR package as they offer it with the 'rating' being an aggregate of all platforms.
Thanks for sharing this info Erin!
ReplyDeleteI agree with all of your points Jack. The main broader issue is really the lack of a way to measure the quality of the engagement (if any at all). While it's quite hard to be able to pinpoint the amount exactly, I'm surprised they at least don't try some rough point equivalents (almost like fantasy football). So if a viewer watches a YouTube video, it's 1 pt, if they comment, it's 3 pts, if they create their own video, it's 15 pts. Or a Facebook Like is 3 pts and a Google+ +1 is only 0.5 pts.
For me, the Wikipedia ratings make sense a bit. It's the viewer looking to more deeply engage with the content matter and backstory, which is important to know, especially if a new character or a new reference is introduced into the show. Also, if the viewers are creating and expanding wiki entries, it serves as an information source and a way to see how viewers interpret the storyworlds. Some of the more complex storyworld producers and writers have actually used fan-based wikis as references to keep track of some elements of the storyworld.
Hi, Zack--
DeleteI completely agree. (See response to Scott's post).
Zack -- I'll have to agree with you on the wiki piece, I think the engagement around fanboys who want to create content/ backstory (via wiki or other platform) are hugely valuable fans and we need to identify and capitalize on them! :)
DeleteI've posed a few of these questions already to Jack after my meeting with them but to respond formally:
ReplyDelete#1 - anyone with even a cursory knowledge of social media will tell you that Google+ is far less superior/impactful/useful than Facebook. I envision these being weighted differently as they continue to tweak the model
#2- Including Wikipedia in this data shocked me. Since Wikipedia is not a social network I don't think it should be tracked.
#3- I agree that Youtube viewing is definitely the most passive form of engagement, particularly if a user is just using it in a utilitarian way - watching the lookahead to next week. If ListenFirst tracked user-generated content I think we'd see a lot of cult classic shows that aren't even on air (Game of Thrones, Twin Peaks, etc) making their way on this list.
#5 - I confirmed with ListenFirst that DAR only tracks content that the network releases (not fan content) which doesn't seem to be a fair barometer. Additionally, they only track hashtags of the official show name so even if there was a slew of activity around a series but it didn't use the #NameOfShow it would not be counted.
#7 - this really bothered me (I am admittedly showing bias because I think the show is terrible) and the only justification I can come up with is that a high performing video was removed from the channel (perhaps due to music licensing or something?) or is in a different channel?
The only other question I will add to the list above is concerning paid media. Currently there is no way for Listen First to parse out paid media impressions with organic ones so currently, a network could be "buying" these ratings.
I can't imagine that it would be that difficult for Listen First to hire someone-- literally one person-- whose job it would be to identify paid media vs organic...
DeleteJack, Erin, Scott, I agreed with all of you regarding Wikipedia until I read Zack's statement. I can see a deeply engaged fan updating and contributing to show entries. Beyond that, my real gripe is that they're tracking very surface level data, much of it that I can glean myself by looking at 'likes' and shares. If we don't see how things are inter-related based on a person's IP address or log-in, does this data really feel relevant? As Erin pointed out, a lot of the numbers we are seeing could be paid for impressions. And as Jack noted, how does mixed engagement 'tweeting to Instagram' get handled? To me, real measurement would come from knowing bobby@ updated "OITNB" page on Wikipedia, made a fun video of Crazy Eyes and posted it on YouTube, shared a post from the Orange FB page, etc. I would start paying attention to their data if I could see how individuals engaged across multiple platforms.
ReplyDeleteWell, after reading your posts, Susan and Zack, I wanted to consider this Wikipedia aspect a bit more.
DeleteI suppose I can agree that Wikipedia can be included when somebody alters the page, but that number has to be miniscule in comparison to the other social networks, and would there be a penalty measurement for people that are adding false information (maybe similar to an interception (-2) in Zack's fantasy football model).
I would agree that a person checking a Wikipedia page is engaged in that they're looking for more information, but by the same token, that means that a person reading an article on any website is equally engaged. So I feel like either you need to use cookies to track even more data about which articles people are reading, or need to throw out the Wikipedia metric.
Also - agree with Erin that the possibility of companies essentially buying their DAR ratings really jeopardizes the whole thing.
In response to Susan's comments-- there are companies out there keeping track of engagement at the individual level and that database is being built (unbeknownst to most consumers). It is indeed valuable and it's only a matter of time before it reaches the mass market and becomes commonplace. I suspect that 5 years from now we'll think it was quaint that we used to look at engagement in general rather than at the individual level. If we operate under the assumption that people engage most with their phones/phablets/tablets (and that's clearly the direction we're heading), then just think of all that data already being logged by apps such as Instagram, Twitter, Facebook, Snapchat, etc. It's all there... Ethically? Well, that's a completely different matter...
DeleteYou're so right, Jack. Do we know how the Comcasts and the AT&Ts of the world are saving and/or using our data with other sources?
DeleteErin -- Thanks for sharing!
ReplyDeleteI'll have to disagree with a few of you and agree with Zack by mentioning that while social media has been a traditional playground for us to measure 'engagement' I still stand by the definition that it is OK to also measure other areas on the web where fans can 'participate' and engage with the content of a show or movie (real or created by the fan). Fanboys are hugely valuable in the ecosystem of viewership. So on that note, I don't have as much of an issue with the inclusion and measurement of this piece.
WITH THAT SAID -- I do think it is all about weighing things appropriately and the wiki data should not receive as much attention as something like the official page’s Twitter or Facebook.
The biggest thing I’d like to challenge in DAR’s strategy is the way they examine and value the Wiki page views as the only piece of wiki data valuable. I think that while views are valuable, additions, changes and creations in that ecosystem might be a far more valuable benchmark for engagement.
The Google+ v. Facebook piece we have all gotten tripped up on I'll have to jump aboard and say that is CRAZY! I think we all agree that collecting the aggregate data from all sources is certainly not dangerous, but weighing them similarly?! This could truly be the item that turns me off most to this data.
I wanted to focus on Last Week Tonight with John Oliver for a second because I think its success in the DAR rankings tells a lot about the methodology and its possible weaknesses.
ReplyDeleteFirst of all, I’m a huge fan of John Oliver. I watch the show as much as I can, but almost never without a prompt from one of my social networks. Unlike the Daily Show, for which I regularly tune in on my own, I rely on social feeds to tell me what Last Week clips are worth watching (remember the famous FCC clip, that exploded through social media and blogs a few months back?). For me, it’s as if the entire show lives on social media—and I wonder to what extent that is due to strategic decisions by HBO or others to make this a more social show (i.e. making clips freely available on youtube). Is the high engagement deliberate marketing or merely good luck? I’d love to hear any insight on the show’s success you guys might have.
I also wonder how John Oliver is able to eclipse other shows like the Daily Show in “engagement” considering the Daily Show’s unrivaled leadership position in the comic news segment (as we’ve discussed before, it’s essentially a credible and necessary news institution for a large segment of society). Is it possible that measuring “engagement” shows bias towards newly created shows that generate excitement and hype in its audience, while penalizing more established programs that may not get people to pick up their phones, but are far more influential?
As for the rankings themselves, I have no idea how one can equate “likes” across platforms ,and how one goes about trying to quantify engagement through instagram/tumblr/wikipedia. I suspect the weighting formula will have to be adjusted constantly over the coming years to improve the data’s validity.
Hey Tim,
ReplyDeleteThought I would respond to the stuff about Last Week Tonight - I watch it, The Daily Show, and The Colbert Report, and we have our DVR set up to record all three, so for me I try to watch them all as much as I can, and I take none of my cues from social media.
I happen to think that John Oliver is not as funny/talented as the other two guys, but think his success really relies on the infrequency of his show. I think it comes through pretty clearly that the production value is higher for last week tonight, mostly because they have a week to prepare instead of a day. I also think having a week in between shows allows all of the show segments to build in online popularity. Every so often a Daily Show or Colbert Report clip has the same kind of popularity, but perhaps it gets buried by another show coming right away.
All three shows seem to request social engagement in one way or another, but perhaps Last Week Tonight is the most consistent with examples like the FCC segment you mentioned. I can remember similar social engagement from The Daily Show, though, like #mcconelling.
And to respond to your question about the hype, it probably does skew towards a show launch, if the show has good marketing, and I suppose the shows that are the best are able to sustain it through either social engagement requests or quality content.