The 2008-09 WCHA Coaches’ Poll is out, and if you’re interested in that you can go view it at the Herald or talk about it on the message board.
The release got me wondering how useful the poll is. Do the coaches make better picks than last year’s final standings? Do the coaches even make better picks than random draw? This post uses some light stats, but I’ll point out the important numbers.
To measure how well the coaches did, I compared their predictions to the actual outcomes (the precise measurement is the sum of the squares of the differences between outcomes and predictions). As a point of comparison, I also compared the previous years’ standings to the actual outcomes, to see if the coaches’ predictions were an improvement over those.
Coaches’ polls’ differences from outcomes vs. previous years’ differences from outcomes (lower numbers are better).
| Year | Sum Sq. Diff Coaches Poll |
Sum Sq. Diff Previous Year |
|---|---|---|
| 2007 | 65 | 84 |
| 2006 | 52 | 43 |
| 2005 | 21 | 48 |
| 2004 | 86 | 89 |
| 2003 | 62 | 146 |
| Avg | 57.2 | 82 |
Looks good for the coaches. In only one of the five years was the coaches’ poll a worse predictor than the previous year’s final standings.
Already with some faith in the coaches’ picks, I also compared them to the average performance of random picks. To do that, I performed 200,000 simulations of random picks to get the shape of their performance. Using the same measure as above, throwing darts would achieve the following error:
| Mean | Std. Dev |
|---|---|
| 175 | 55.1 |
Note that the 175 mean squared error for random picks is much bigger than either the 57 the coaches achieved over 5 years or the 82 the previous years’ outcomes achieved. (The coaches’ poll is 2.1 standard deviations better than random picks, the previous years’ outcomes 1.7 standard deviations better). From that, we can conclude that both the previous years’ final standings and the coaches’ poll contain useful information that provides a more accurate prediction than a random draw.
Here’s the raw data:
http://spreadsheets.google.com/pub?key=pDN6PFrUpi3c_P6VEQ353Kw
Someone on the forum criticized INCH’s prediction of UND finishing 7th, so thought I’d run the INCH WCHA preview through the same treatment:
Year Coaches INCH Previous
2007 65 51 84
2006 52 37 43
2005 21 36 48
2004 86 78 89
2003 62 48 146
Avg 57.2 50 82
This is how far off the prediction was, so lower numbers are better.
We can’t even blame INCH’s anti-UND bias ;), because the only time in those five years they’ve picked UND to land lower than #2 was 2004-05 when they predicted a #4 finish. The Sioux finished 5th.