to close to call

I think I may have done this post before. It feels like I’ve had this debate.

The NHL reupped it’s deal with Versus thru 2011 today, which is dumb enough. This line, written in the New York Times is actually significantly dumber:

Through 29 games, Versus’s 0.3 rating is modest, but it is 50 percent higher than the 0.2 average at the same time last season.

I’d like to introduce the NYTimes writer to the phrases “statistically insignificant” and “margin of error”. Which is funny, because just last year there was apparently a big issue about the same thing in the same paper.

I saw this again reading Figure Four Weekly, though it should be noted that all the newsletters will be talking about this as a ‘win’ for TNA.

ECW did a 1.2 rating this week and 1,488,000

[TNA did] a 1.22 rating …. 1,623,000 viewers.

How do we know the Neilsen ratings are accurate to the .01 rating point? Or to a 1,000 viewers? I do know they poll a sample size of random people for their television viewing habits and projected those numbers for the full (US) TV viewing universe, adjusting for things like race and age. If you want to know how big that sample size is or anything else exact about accuracy, you’ll waste your time going to their website. Everything there is in general terms.

I don’t have exact numbers either, but I do have enough of my college statistical course still locked in my brain to remember these numbers are just the mid point of a range of equally likely possibilities. Of all the possible numbers in the world, the TNA is most likely exactly 1.22, but that doesn’t necessarily mean the rating is at least 50% likely to be 1.22 – it may be 33% likely to be 1.22 or less. There’s a 95% chance* the rating is within the margin of error, with the best chance a 1.22, there’s a 2.5% chance it’s way less than 1.22 and there’s a 2.5% chance it’s way higher than 1.22. If the margin of error is 4% or less, TNA probably had more viewers. Once the margin of error gets to 5%, there are situations where ECW actually got more viewers.

* – probably, but I’m assuming here to cut down on the numbers and maybes

It all depends on what the margin of error is, and I believe it’s a lot higher than anyone would figure for channels like SciFi or Versus. Channels that are available less would get counted by a small portion of the Neilsen sample, increasing the margin of error. It’s probably smaller for network shows, but it’s going to be an even bigger swing; just a .04 margin of error for a show that gets a 7.5 means they’re actually saying it’s probably between 7.2 and 7.8.

For TNA and ECW and the like, .1 up/down weekly changes (and differences) are probably meaningless. .2 are likely as well. If there’s a trend of a show going down .1 every week for many weeks, it’d mean something then, but declaring winners and losers based on anything less than .2 difference is a waste of time. Those are the most inconclusive victories.

Of course, the real stories here is, even if the numbers were perfect, TNA shouldn’t be celebrating beating ECW – this is much more a story of ECW falling to TNA levels, TNA’s barely moved themselves – and the NHL is just as self deluded if they truly believe getting 0.3 ratings is a sign of something going right.