Remember Base Rates!
One topic that is often missed when evaluating a competitive metagame is the importance of base rates. When determining how powerful something is, you need to not just continue the results, but also the results in light of the population distribution.
In other words, if someone sees a certain character or army list. performing strongly, they may assume that that character is especially good. However, this may prove to be an error — it’s very important to consider not just the actual results, but also the results in light of how many people are playing that character/list.
Here’s a basic example: let’s say that there is a game with five characters. Someone holds a big tournament for this game with two hundred players and lots of rounds of Swiss play into a cut to top eight. Once the Swiss rounds are complete, the top eight has four of one character and then one of each of the four other characters.
With just that information, one would tend to conclude that the character with four top eight spots would be stronger than the others. But let’s not be too hasty — in order to draw stronger conclusions, we would need to know what the field looked like. If 196 of the players had all been playing the character with four top eight spots, and only one player each of the other characters, this result would actually represent a massive underperformance by the character who looked to be doing “the best”. On the other hand, if only four players played the character that took four spots and 196 players played everything else, that would be a massive overperformance, even moreso than the numbers would indicate at first glance.
Now, those examples are kind of extreme and silly — let’s take a scenario from real life instead. As some have likely seen here before, I’m maintaining a Results Tracker for Flesh and Blood competitive events. (Much thanks to all those who have helped me with the tracker by submitting events, deck lists, etc.!) I don’t have full playerbase breakdowns for most events, but I did set up another tab in the sheet that goes into those statistics for the events where I do have it (RTN Base Rate Analysis). Here’s what the core statistics look like as of this writing:
As you can see, this sheet tracks the total number of players on a hero as well as the number of top eights and wins those heroes have achieved. It then gives more detailed stats on the number of top eights per entry and the number of wins per entry for each hero. The final section — the percentage of playerbase, percentage of top eight, and percentage of wins — is especially relevant.
In principle, if all heroes were performing the same, we would see the last three percentages as very similar for all characters. Indeed, there are some characters that seem to be performing more or less “on the mark” there — Boltyn, Bravo, Dash, and Dorinthea all have very close playerbase/top eight/win percentages. For some characters the results are a little distorted by low sample size, and Katsu “should be” performing significantly better based on the overall statistics but few events with Katsu winners have given me the full stats.
However, there are two big outliers. First up, Chane is doing great, but he’s also the most popular hero, so maybe we would expect him to be winning more. Well, these statistics show Chane is strongly overperforming his representation — around 20% of the field is on Chane but he is over 29% of the top eight and over 38% of wins! This represents a big advantage for Chane — no other hero is putting up similar numbers at this time. In other words, Chane isn’t just winning thanks to being popular, he’s also extremely strong.
The other big outlier is Prism. Prism is basically the opposite of Chane — despite being very popular, Prism is strongly underperforming her representation. In fact, Prism has the worst wins/entry of any hero with at least one win. To make matters worse, she’s not just underperforming to take the win but actually underperforming a bit at making top eight as well — her large number of top eights looks to be not a function not of being especially strong but rather one of just having a lot of people playing!
Now, the first of these outliers is probably a bit obvious — those who have been following the meta know Chane has been performing well — but the second is a bit of a surprise. Prism is the third-most popular hero in the present meta, so it’s quite surprising to learn that her performance is so weak! To put this into perspective a bit, Prism has more than 50% more players than Dorinthea in these statistics, but only about 40% more top eights, and actually half as many wins. That’s not a typo — the hero with more than 50% more people playing her is actually winning half as often in this sample! Further, this compares Prism to Dorinthea, a solid but not dominant hero — if you instead compare Prism to Chane the numbers look truly dismal, with Chane having about a 5x higher “wins per entry” rate!
What should we take away from this? Well, in terms of practical applications, if you want to win in this meta I would strongly suggest not playing Prism. Chane, on the other hand, is an obvious standout and is winning well above his fair share. That said, Chane isn’t the only option by any means — several other heroes are doing respectably well, including Rhinar and Viserai, who are low sample size but have put up pretty strong showings regardless.
Of course, there are more events to be played and more data to be gathered, this is just a snapshot from one point in the Road to Nationals season. That said, regardless of what heroes might be popular in a particular meta, base rates can be a helpful tool in analyzing things and noticing details that might get lost when just looking at overall results.
(PS: If all goes well, Tales of Aria preview stream on September 8th! Get hype!)
