Are The World Rugby Rankings Actually Doing Their Job?

Are The World Rugby Rankings Actually Doing Their Job?

World Rugby's Rankings are important references for seeding teams and setting up games, but are they doing the job?

Dec 28, 2017 by Alex Goff
Are The World Rugby Rankings Actually Doing Their Job?

The New Zealand All Blacks will end 2017 ranked No. 1 in World Rugby's Men's World Rankings, and while second-place England has inched a little closer, the Kiwis retain a commanding lead.

That's some serious stability atop rankings that have seen changes elsewhere in the top 10. Ireland moved from No. 4 in January to No. 3 by the end of the year, replacing Australia. Wales dropped from No. 5 to No. 7, while Scotland moved up from No. 7 to No. 5. France dropped from No. 8 to ninth, behind Argentina.

All very exciting stuff. But lower down, it seems like it's becoming harder and harder for lower-ranked nations to get any traction. The fact that Tier 2 versus Tier 1 matchups are still very rare means that upsets don't move the needle that much. Tier 1 nations playing each other might make one nation move up or down a spot, but the differences in rankings points mean a major loss (New Zealand's 57-0 crushing of South Africa, for example) actually has very little effect.

We decided to look into this further. What if we stacked the World Rugby rankings deck? We took three Tier 2 nations and gave them each a demanding schedule — playing the top six teams in the world on the road. What would happen if Georgia, the USA, and Canada each played New Zealand, England, Ireland, Australia, Scotland, and South Africa and won?

(Note: We had each team play England, Australia, Ireland, South Africa, New Zealand, and Scotland in that order. The order of play does affect what happens to those nations.)

(Additional note: We are working with the men's rankings, but our math and our conclusions translate to the women's rankings, as well.)

Here's what we found out:

Scenario 1

Georgia (currently ranked No. 12), USA (ranked No. 17), and Canada (No. 21) all play the top six teams in a row, on the road, and win. For purposes of this experiment, no one other than these nine teams changes rankings.

Rankings After Georgia, USA, and Canada Sweep Top 6
Rank
Team
Pts
Rank Dec 2017
1
New Zealand
87.99
1
2
England
84.87
2
3
Georgia
84.59
12
4
Wales
82.08
7
5
Ireland
80.39
3
6
Australia
79.49
4
7
USA
78.75
17
8
Scotland
78.83
5
9
Argentina
78.22
8
10
France
78.09
9
11
South Africa
78.08
6
12
Fiji
77.93
10
13
Japan
75.66
11
14
Canada
73.13
21


Wow. So even after sweeping the top six teams in the world, Canada isn't ranked above any of them and hasn't even caught Japan. The USA gets past Scotland and South Africa, but that's it. Georgia does move up a lot. The biggest down-mover is South Africa, dropping five spots. 

Scenario 2

Georgia, USA, and Canada win all of their games by more than 15 points, kicking in a rankings boost for a blowout. Here is how the rankings come out then:

Rankings After Georgia, USA, and Canada Blow Out Top 6
Rank
Team
Pts
Rank Dec 2017
1
Georgia
89.03
12
2
New Zealand
84.99
1
3
USA
83.79
17
4
Wales
82.08
7
5
England
81.87
2
6
Canada
78.75
21
7
Argentina
78.22
8
8
France
78.09
9
9
Scotland
77.97
5
10
Fiji
77.93
10
11
Ireland
77.40
3
12
Australia
76.49
4
13
South Africa
75.84
6
14
Japan
75.66
11


Now we're getting somewhere. With a six-game sweep of blowouts, Georgia becomes the No. 1 team in the world, and USA moves to No. 3. Canada still lags behind England and New Zealand.

(Scotland played each of the Tier 2 nations last, which means the rankings difference between the two teams was smaller than at the beginning of our experiment, thus Scotland gets dinged less for a loss than do Ireland and Australia, which is why Ireland and Australia drop below Scotland.)

Scenario 3

A bit closer to realism. Here we have the teams involved go .500, beating England, Scotland, and Australia and losing to New Zealand, Ireland, and South Africa.

Rankings After Georgia, USA, and Canada Go 3-3 vs Top 6
Rank
Team
Pts
Rank Dec 2017
1
New Zealand
93.99
1
2
Ireland
86.39
3
3
England
84.87
2
4
South Africa
83.88
6
5
Wales
82.08
7
6
Australia
79.49
4
7
Georgia
79.37
12
8
Argentina
78.22
8
9
Scotland
78.14
5
10
France
78.09
9
11
Fiji
77.93
10
12
Japan
75.66
11
13
USA
72.87
17
14
Tonga
71.87
13
15
Italy
71.25
14
16
Romania
69.58
15
17
Samoa
69.03
16
18
Canada
67.13
21


So this is perhaps the most damning result. Consider: a Tier 2 team plays the six best teams in the world and goes .500 and yet can't move higher than No. 18? USA is at No. 13 and trails by significant points to Japan. Only Georgia gets a significant bump.

What This Tells Us

There are many other ways to look at this, including 719 other combinations of games involving the top six. But this simple approach does show something. 

It is enormously difficult to gain entrance into the rarified club of the top tier, even if you beat all of the best teams. 

The historic results are still embedded in the points total for each nation, and so, an up-and-coming country has to get several years' worth of results to erase a bad year or two. 

In addition, this experiment shows that if you don't play very often your rankings points don't change. A highly ranked team can hang on by not playing, but a lower-ranked team (USA fans will understand this) has a really tough time moving up if a) it can't find games, and b) the opponents it faces are not ranked highly.

And finally, World Rugby puts a limit on how many points you can get from one result. That is capped at two points (three in a Rugby World Cup game). This rule stops one result from making a drastic change in the rankings, but it also entrenches the top teams.

For example, let's say the USA plays New Zealand in Auckland and somehow ekes out a 25-24 victory. According to World Rugby's formula, the USA would gain a maximum of two points, not even good enough to move the Eagles out of 17th. New Zealand would lose two points and stay No. 1.

But if we used World Rugby's formula without the two-point limit, the USA would gain 4.01 points, good enough to move from 17th to 15th, and New Zealand would lose 4.01 and drop out of first place.

The World Rugby rankings are a great thing and have a lot of thought put behind them. But they are showing their age and sometimes serve to entrench groups of teams. 

Given that World Rugby uses the World Rankings to make decisions about fixtures and sometimes even seedings, maybe the system needs to be a little more dynamic.

(Additional technical assistance, research, and math provided by Owen Goff.)