For some history of the International Rankings Royale, take... a look... at this.
The Combatants (OR Can Robots Fight?)
Yes, 2013 has ended some time ago so it's far time that we take a lot at how each international rankings system did. Just a reminder of the contestants:
- FIFA Rankings - Important for any major competition to sort teams into correct pots. These rankings are used to seed for the FIFA World Cup Qualifying but also (I think) every confederation's continental cup. So these rankings are clearly the most important. But are they the best?
- ELO Rankings - A ranking system expanded from the chess rankings. I don't really know how it works but it obviously has a lot of math. But is math good?
- SPI Rankings - Nate Silver's brainchild is used to give odds and neat graphics like these. But can we trust robots?
The old link at the top gives the methodology of it all (and you can look a the results from all 921 games from this past year here) but basically it came down to goal differential. I put each ranking system on the same playing field of saying that every 20 spots in their rankings meant the two teams were separated by one goal differential. I froze the rankings at the start of every month to make it fair (ELO, SPI update daily) and capped the goal difference at four. So a 6-0 win still only counted as a 4-0 win in these rankings. I know there a lot of variables like teams sending reserves or home field advantage but every ranking system is at an equal disadvantage in those situations.
Results Backed by "Math"
So which rankings system was the most accurate over the 921 games? Well it was close. Each number below is the number of goals they were off the actual goal differential, or the goal difference difference (GDD), from the combined 921 games.
- ELO - 1448 (1.572 per game)
- FIFA - 1506 (1.635 per game)
- SPI - 1519 (1.649 per game)
Over 921 games, the best and worst were only separated by 71 goals. I tried to find some discrepancies between the three rankings. Surely one is better, say, at predicting UEFA success while another overrates CONMEBOL. Right?
Below is a table (it's just a picture, don't get too excited) of how far off each system was from the correct ranking in each match. ["Else" includes any team not distinctly in FIFA but still plays matches. CONCACAF has a number of these with the Caribbean teams.]
(I'm going to explain the math in this paragraph so if you hate math just skip this one.) We'll use the first game of 2013 as an example. Niger defeated Togo 3-1 despite being an underdog in each system. Specifically ELO said Togo was 103rd in the world going into the game while Niger was 122nd. The difference in rankings should have, by ELO's prediction, given Togo a +.9 GD. However since Niger won by two goals (giving us a 2.9 GDD, 2 + .9) ELO's rankings were significantly off. The two goal difference translates to a would-be 40 spot gap in the table. I combined the 40 would-be gap and the 19 actual gap to average a correct ranking difference but maintaining the same middle ground. Halfway between 103 and 122 is 112.5. (You don't need to check that on a calculator, I'm pretty sure that's right.) ELO should have said that Togo should have been ranked 132.5 while Niger should have been 92.5 because it's a 40 spot gap and halfway between 132.5 and 92.5 is still 112.5. (I told you it was a lot of math!)
The top row is the number of games each confederation appeared in for a total of 921 doubled, because there are two teams to each game. The next section is how far off each system was from was for each confederation. So for each Asian team (AFC), ELO was 18.91 spots off in their rankings every game, or almost a full goal. This is less than SPI and FIFA's misrank of around 23 and 25 respectively.
The next section tells us if they under or overrated those teams. ELO was fairly consistent with only a -0.95, rarely underrating teams while SPI consistently underrated Asians squads. For CONCACAF teams, however, FIFA and SPI typically overrated them.
Overall, they're all pretty similiar. ELO excelled at some but not by much. And everyone was a lot closer with CONMEBOL teams than the rest of the confederations. I thought that might be because of the minnows outside of the top 50. To test this, I sorted out any game where at least one system didn't have one of the teams in their top 50.
Not a huge change. FIFA and SPI was actually less accurate with the better AFC and CONCACAF teams. CONMEBOL didn't change at all except for a hundredths. I think that's an error on my end but I couldn't find out why the small change because the number of games stays the same. (Sorry.) And Martinique is the lone representative for the "Else" category, when they played Mexico and Panama in the Gold Cup this summer. It says FIFA was underrating them by 130.5 rankings but that is because Martinique isn't in theirs so I just listed them last, 210th.
I didn't differential between friendlies and more important games because I forgot to label them as I started and it'd be a bit of work to do that. But I didn't feel too bad about this as all these teams still uses friendlies to find out their rankings. If someone wants to go back and figure it out by labeling all 921 games, be my guest.
Conclusion
There's not that much of a difference between the three. I think we get tripped up on one system being flawed because they ranked X team ten spots lower than we would. We rarely consider the strength of a ranking system by surveying the bottom 150 teams. But if you had to say one was the best, I suppose you could lean towards ELO.
The FUTURE
For 2014, we have a new face: We Global Football. Like SPI and ELO, they update daily and even go as far to predict every game's goal difference. We had a number of matches in 2013 but it won't hold up to last year's near millennium mark with qualifying and confederation cups on their off year. Already through early March, 2013 has doubled the number of matches that 2014 has. Here are the standings through 84 matches:
- SPI - 123 (1.468 per game)
- ELO - 127 (1.507 per game)
- FIFA - 128 (1.520 per game)
- WGF - 139 (1.652 per game)
(View the complete game-by-game results here)
But since I was paying attention, I marked all the non-friendlies for 2014. So far there have only been fourteen and they've all been in the AFC with the 2015 WAFF Championship and qualification for the 2015 Asian Cup. We need more games but I did think the results were interesting.
- ELO - 20.2 (1.443 per game)
- WGF - 20.5 (1.464 per game)
- SPI - 23.1 (1.650 per game)
- FIFA - 25.1 (1.789 per game)
Again, it's very early, but SPI's GDD per game jumped from 1.468 to 1.650 when we only counted non-friendlies. FIFA makes a similar jump and WGF and ELO are actually more accurate.
Lastly, here are the three biggest misses by each system at this point in 2013. I judged them on how far off the other three systems were from guessing.
date - (rank) nation at (rank) nation [score:score] (GDD, how far off the other three's average)
ELO
1. March 5th - (77) Uganda at (89) Zambia [1:2] (1.6 GDD, +1.28 off average)
2. March 5th - (154) South Sudan at (130) Botswana [0:3] (1.8 GDD, +1.22 off average)
3. February 19th - (191) Pakistan at (114) Lebanon [1:3] (1.9 GDD, +1.22 off average)
FIFA
1. January 18th - (54) Norway at (78) Poland [0:3] (4.2 GDD, +1.58 off average)
2. March 5th - (66) Finland at (46) Hungary [2:1] (2.0 GDD, +1.47 off average)
3. March 3rd - (106) Kuwait at (34) Iran [2:3] (2.6 GDD, +1.30 off average)
SPI
1. (101) United Arab Emirates at (53) Uzbekistan [1:1] (2.4 GDD, +2.27 off average)
2. (25) Ghana at (90) Montenegro [0:1] (4.3 GDD, +2.23 off average)
3. (172) Malaysia at (152) Yemen [2:1] (2.0 GDD, +1.38 off average)
WGF
1. (180) Faroe Islands at (60) Gibraltar [4:1] (7.0 GDD, +5.78 off average)
2. (110) Estonia at (60) Gibraltar [2:0] (4.5 GDD, +2.58 off average)
3. (155) Rwanda at (105) Burundi [1:1] (2.5 GDD, +2.32 off average)
We Global Football had the most atrocious misrank with Gibraltar 120 ranks above Faroe Islands (an equivalent of a 4-0 win) but lost by three goals giving WGF not only a 7.0 GDD but almost six goals off the average. Out of fairness, Gibraltar tied Slovakia back in November 0-0 so I suppose we can understand the high rank. Gibraltar is now 189th on WGF.
I'll try to update again in the summer some time so see how these systems are doing going after the World Cup. If you want to watch them continue to duke it out, keep an eye on the 2014 Royale Google doc. I'll be updating that monthly.