The release got me wondering how useful the poll is. Do the coaches make better picks than last year’s final standings? Do the coaches even make better picks than random draw? This post uses some light stats, but I’ll point out the important numbers.
To measure how well the coaches did, I compared their predictions to the actual outcomes (the precise measurement is the sum of the squares of the differences between outcomes and predictions). As a point of comparison, I also compared the previous years’ standings to the actual outcomes, to see if the coaches’ predictions were an improvement over those.
Coaches’ polls’ differences from outcomes vs. previous years’ differences from outcomes (lower numbers are better).
Looks good for the coaches. In only one of the five years was the coaches’ poll a worse predictor than the previous year’s final standings.
Already with some faith in the coaches’ picks, I also compared them to the average performance of random picks. To do that, I performed 200,000 simulations of random picks to get the shape of their performance. Using the same measure as above, throwing darts would achieve the following error:
Note that the 175 mean squared error for random picks is much bigger than either the 57 the coaches achieved over 5 years or the 82 the previous years’ outcomes achieved. (The coaches’ poll is 2.1 standard deviations better than random picks, the previous years’ outcomes 1.7 standard deviations better). From that, we can conclude that both the previous years’ final standings and the coaches’ poll contain useful information that provides a more accurate prediction than a random draw.
Here’s the raw data: