How good is the Coaches’ Poll?

The 2008-09 WCHA Coaches’ Poll is out, and if you’re interested in that you can go view it at the Herald or talk about it on the message board.

The release got me wondering how useful the poll is. Do the coaches make better picks than last year’s final standings? Do the coaches even make better picks than random draw? This post uses some light stats, but I’ll point out the important numbers.

To measure how well the coaches did, I compared their predictions to the actual outcomes (the precise measurement is the sum of the squares of the differences between outcomes and predictions). As a point of comparison, I also compared the previous years’ standings to the actual outcomes, to see if the coaches’ predictions were an improvement over those.

Coaches’ polls’ differences from outcomes vs. previous years’ differences from outcomes (lower numbers are better).

Year Sum
Sq. Diff
Coaches
Poll
Sum
Sq. Diff
Previous
Year
2007 65 84
2006 52 43
2005 21 48
2004 86 89
2003 62 146
Avg 57.2 82

Looks good for the coaches. In only one of the five years was the coaches’ poll a worse predictor than the previous year’s final standings.

Already with some faith in the coaches’ picks, I also compared them to the average performance of random picks. To do that, I performed 200,000 simulations of random picks to get the shape of their performance. Using the same measure as above, throwing darts would achieve the following error:

Mean Std. Dev
175 55.1

Note that the 175 mean squared error for random picks is much bigger than either the 57 the coaches achieved over 5 years or the 82 the previous years’ outcomes achieved. (The coaches’ poll is 2.1 standard deviations better than random picks, the previous years’ outcomes 1.7 standard deviations better). From that, we can conclude that both the previous years’ final standings and the coaches’ poll contain useful information that provides a more accurate prediction than a random draw.

Here’s the raw data:
http://spreadsheets.google.com/pub?key=pDN6PFrUpi3c_P6VEQ353Kw

One thought on “How good is the Coaches’ Poll?”

  1. Someone on the forum criticized INCH’s prediction of UND finishing 7th, so thought I’d run the INCH WCHA preview through the same treatment:

    Year Coaches INCH Previous
    2007 65 51 84
    2006 52 37 43
    2005 21 36 48
    2004 86 78 89
    2003 62 48 146
    Avg 57.2 50 82

    This is how far off the prediction was, so lower numbers are better.

    We can’t even blame INCH’s anti-UND bias ;), because the only time in those five years they’ve picked UND to land lower than #2 was 2004-05 when they predicted a #4 finish. The Sioux finished 5th.

Leave a Reply