Wired runs an interesting article today on a new Nielsen television ratings system that eliminates the decades-old practice of using diaries to calculate viewership. The computer-based "people meter" attaches to the television, VCR, and game box to give a 24/7 report on what Nielsen families watch. This new system has brought a change that has called into question its accuracy:

Just this month, the company came under attack from television networks, minority groups and even lawmakers when a test of its electronic "people meters," newly installed in select New York homes, began reporting a sharp decrease in viewership for television shows that feature minorities. Because the current system -- a decades-old technique involving week-long diaries that are mailed to the homes -- had never yielded such a drastic swing, the critics contended that the new technique must somehow be unreliable.

For those of you with a TiVo, the system connects in much the same way to entertainment systems and dials the information back to Nielsen on a regular basis. In contrast, the journaling system data could only be collected four times a year and took long man-hours to analyze, a bone of contention for decades:

With the diaries, Nielsen is only able to do local-level metering four times a year. Not only is the system slow, it has resulted in a phenomenon known as sweeps week, a period during which television producers pursue sensationalistic news stories or introduce zany plots or celebrity walk-ons in their shows in hopes of raising their ratings.

Worse yet, with the diary system, viewers have been known to wait until the end of the week to jot down what they watched, making the diary entries less than perfectly accurate -- a point that even Nielsen concedes.

"There may be a tendency in a diary for people to forget to write down what they were watching until a couple days later," said Elliot. "With people meters you don't have to go back and remember what you watched a week ago."

Minority representatives and lawmakers have expressed outrage over the new system and have already decided that it's flawed and should be withdrawn. However, just on the face of it, it's difficult to see how a journaling system could possibly be superior to an electronic system that has no capacity for the surveyed to modify. I deal with timesheets from employees that are supposed to be filled out daily but mostly get completed at the end of the pay period, with numerous mistakes about arrival and departure times when compared to the computer-system login/logout times. It's not a question of dishonesty, it's a question of reliability.

Human nature and diary bias is the more likely explanation. Most people want to see television shows that feature minorities to succeed, especially when they're well done, like George Lopez as an example. Nielsen families must feel a lot of that pressure, along with a natural desire to demonstrate an open mind. Couple that with a system that is entirely self-reporting with no controls or quality checks for accuracy, and one in which the reporting can be done far from contemporaneously, and you have a recipe for well-meaning but significant error.

I can understand the frustration that these results have produced, but the answer isn't to bury the people meters; it's to roll them out to a much larger sample and see if the same results occur. New York is not indicative of the entire nation, after all. If the results stay the same, install people meters in homes using the current diary system. If results continue to deviate from expectations after that, we likely will have found the culprit, and it won't be the computers.

Having said that, if we find out that the most popular television show winds up being My Big Fat Obnoxious Fiancé, I will be the first to lead the torch procession to Nielsen headquarters.

Trackback Pings

TrackBack URL for this entry is