Earlier this year, USCHO raised eyebrows among college hockey geeks by beginning to publish bracketology articles on Nov.10. USCHO justified expending the effort to predict what the tournament would look like had the season ended after just three weeks by claiming it was a natural response to other sites (no names named 😉 ) making PairWise Rankings available before USCHO’s typical January date. While some may quibble that seeding a bracket is a different activity from letting an automatically calculated rankings table be published, I became most interested in the question, “when is the appropriate time to begin looking at PWR?”
The real answer is, obviously, whenever you’re interested in it.
My primary interest in looking at rankings is to see how each team is doing — who is good, who isn’t, who is likely to make the tournament? The value of the rankings for that purpose is dependent on their stability. A team ranked #1 in an accurate ranking should probably be able to lose a game and still be ranked in the top few teams. So, my first tactic for determining the quality of the PWR rankings was to determine how much ranked teams move within the rankings from week-to-week.
Before December the PWR rankings have wild swings, though settle into an average move of 2-3 slots in January and under 2 slots by March (the low movement in late December is the holiday break in which teams play fewer games).
That’s interesting, but the actual pairwise comparisons are only meaningfully performed once, after the conference tournaments. So, the reason we really look at the PWR earlier is to get an idea of who might make the tournament. Therefore, to determine the quality of an individual week’s PWR we’re actually most interested in how well it predicts the final PWR.
Though PWR in January seemed somewhat stable, in that teams only move a couple of slots each week, as those movements add up it actually proves a pretty poor predictor of the final PWR. Each team has been ranked an average of 3-8 slots differently on April 1 than in January over the past six years. Even the March 1 PWR rankings have deviated from the April 1 rankings by an average of 2-4 slots.
So, enjoy looking at the weekly PWR tables, but remember that they’re all for fun.
A couple notes:
- April 1 isn’t quite the final PWR because some NCAA tournament games have been played by that date.
- The average movement in each of the above charts is mean movement of those teams that had PWR rankings in: 1) both weeks, and 2) the week in question and on April 1. No attempt was made to determine variance, or whether that mean was overly influenced by a few large moves.
References/links of interest:
- USCHO’s Bracketology blog: http://www.uscho.com/blogs/uscho/bracketology
- SiouxSports.com PWR: http://siouxsports.com/rankings/pwr/
- CollegeHockyNews explanation of PWR: http://www.collegehockeynews.com/info/?d=pwcrpi