How do Spiel des Jahres Winners Rate? – Part 3

Setup

Considering the statement made in Part 1 regarding the relevance of the SDJ to the BGG audience, let’s take a look at how the games considered for the SDJ rate on BGG. This rating begins with the Average User Rating, which drives the BGG Rating (which considers the number of User Ratings), and ultimately the BGG Rank (which stacks the BGG Ratings up against each other). Although, BGG Rank is ultimately the most interesting and will have the greatest focus in this article, we will look at some statistics about the SDJ and KSDJ games in the order that the BGG Rank is built: User Rating –> BGG Rating –> BGG Rank. All graphs have been derived from data obtained from BoardGameGeek.com in June and July 2017.

Note: If you are starting your reading with this article, you might want to jump to the introduction here: Industry Focus: Spiel des Jahres & the BGG Community Part 1

Round 1: User Rating

Are the SDJ and KSDJ games the kind of games that BGG users want to play? Let’s take a look at the User Ratings to get an idea. Unfortunately, I have no way to calculate Average User Ratings across the population of BGG listed games for comparison, so this analysis is a bit subjective – how do these games rate against expectations (vs. other games).

First let’s look at the games by Award Category.

Probably no real surprise here, but maybe they all rate higher with the BGG users on average than one might expect. The KSDJ games rate better on average than the lighter, SDJ games by about a full point (precisely 0.91).

Now let’s break that out into how the games fare within the awards, what I have called the Award Level.

There is not a great distinction among BGG Users for the games mentioned for the awards. For the KSDJ, there is a slight preference for the games that were not the Winners. This is in line with popular opinion that the awards highlight games that are less strategic than the average BGG user wants. Though, the difference is not great, we will see more of this later.

Now the average User Ratings of all games mentioned in the awards by Award Category by year.

Not surprising, the KSDJ games as a group rate higher than the SDJ games and there is a general rise in rating over time for both Award Categories. The average rating for the KSDJ games in the high 7’s is quite good and the SDJ rising steadily from 6 to 7 is respectable, but not fantastic. However, since these are averages of up to 16 games, there could be some real gems hidden in all the dross. Here are the best User Ratings in each.

No surprise here. The best rated games are about a half point higher than the average for each year.

Finally, the ratings trend of the Winners only is not too different from the “best” ratings, so apparently the judges generally get it right… (except Pandemic Legacy: Season 1).

Round 2: BGG Rating

The BGG rating is derived from the Average User Rating with the addition of a factor to account for number of ratings. This is to prevent a game that only has a few very high ratings from shooting up the ranking and is presumably done by seeding the calculation with some number of low or middle ratings as a buffer. I believe that the exact formula is still a mystery, but I suspect that a base rating of about 5.5 for 1000-2000 “users”. There is also a minimum number of User Ratings required for a BGG Rating to be calculated – this helps obfuscate the formula.

These charts should not differ drastically from the User Rating charts except for games that have had limited distribution. It is curious to note that a low rated game with few ratings will actually benefit from the BGG Rating calculation; though, very few of the games in this study meet those criteria. Here are the same trends that were presented for User Ratings with BGG Ratings.

Again, the difference here between the two Award Categories is about a full point (precisely 0.93 – a tiny bit, 0.02 greater than the user ratings). The total difference from the User Ratings to the BGG Ratings is a drop of 0.34 points. This indicates that as a group (not individual games), the BGG calculation for ratings is keeping ratings in check, but not causing a great relative difference.

Now, breaking these out into the Award Level.

Again, other than the 0.34 point drop, there is no significant difference from the average User Ratings. The SDJ Award Level popularity is a little shuffled from the raw User Rating and thre is a small but noticeable decrease in the popularity of the SDJ games as they drop in Award Level.

Now a few charts by year.

Not much to note for the total population of games here, but (comparing to the same graph for User Ratings) the BGG Ratings process has caused a slight leveling as expected.

Now, the best of the best.

Similar to the User Ratings, we see greater discrepancies per year (we aren’t leveling by averaging) and the rise over the years is more drastic. This might suggest that the judges are still considering a broad range of games, including some that have a global appeal and some that are specific to the German market.

And, finally, just the Winners.

Due to the small number of data points for the KSDJ, the trend shows a drastic change, but the actual difference is primarily the change in the rating for the 2017 game (really a game system) EXIT: Das Spiel, which had limited distribution at this time and so had comparatively very few ratings.

We will look at the Special awards separately as well, since the nature of the award has changed over the years.

As with the comparisons for game weight in Part 2, we see two different eras of the “Special” Award. With the exception of 1981, when Ra was honored, the Special Award was primarily used to acknowledge lighter, children’s games until the inception of the Kinderspiel Award in 2001. In the four years that the award has been given leading up to 2011 when the Kennerspiel Award was implemented, the award was used to acknowledge heavier games:

• 2006: Caylus, Shadows Over Camelot
• 2008: Agricola
• 2009: GifTRAP (the exception), Space Alert
• 2010: World Without End

Round 3: BGG Rank

The BGG Rank is determined by simply sorting the games by their BGG Rating. Although, this is essentially the same statistic as discussed in Round 2, it has a different effect on the brain so we will take a look at these statistics from a different angle. This is the angle that we will focus most of our attention and rather than focusing on “average rank” which quickly becomes muddy, I have created a few Ranking categories that were convenient mathematically, but also can be described:

• Top 50: These are the “best of the best.”
• 51-200: These are great games.
• 201-500: Many of these were once great games, but have not aged as well as some.
• 500-1000: These get played a lot.
• 1000-2000: OK, but still get played occasionally.
• >2000: Generally forgettable.

A couple notes about my categories:

• Although, this splits the Top 100 that are the basis for the BGG Rankings home page across 2 categories, with the number of high quality games these days, a lot of great games fall into the 101-200 range.
• For perspective, BGG has over 28,000 games listed, so the cut-off at 2000 represents the top 7% of all games listed.

SDJ Considered Games

First, let’s look at the SDJ games (excluding KSDJ) categorized by their position in the awards, the Award Level, and where they rank on BGG. The percentage represents the number of games in that BGG ranking category as a percentage of the games in the Award Level (E.G., All the red “Winners” add up to 100%).

The Winners do tend to rank a little higher than the other games considered for the award – about 20% in the Top 200 vs. about 10% for the other considered games. Here is a look, zeroing in on just the Winners.

Only 6 of the 39 games are not in the Top 2000, which is a pretty good place to be. For reference, on Jan 16, 2018 these were the rankings around 2000: (a mix of decent strategy games and respected family games)

• 1997 - The Witcher Adventure Game
• 1998 - Pairs
• 1999 - Subdivision
• 2000 - Slide 5

KSDJ Considered Games

Now let’s look at the KSDJ, which we appropriately suspect will rank higher in general on BGG since they are heavier games as shown in Part 2 of this series.

The KSDJ considered games are well represented in the Top 50, with 12 games – that’s almost 25% of the top 50 games in just 7 years of awards – and representing 30% of all games considered for the award. Combine that with the 7 SDJ games and the SDJ awards represent 38% of the Top 50 games on BGG. Not a bad showing. A whopping 88% of all games considered for KSDJ rank in the Top 500 games.

The big difference between the SDJ award and KSDJ Award Categories is the representation of the Winners in comparison to the other games considered. In the SDJ games we saw the Winners lead the way by 2x over all others in the Top 200. For the KSDJ we see almost a dead heat at 57-58% in the Top 200, and the Finalists and Recommended games well ahead in the Top 50.

Round 4: BGG Rank by Year

SDJ Considered Games

Games move up and down the BGG rankings over time, but for the most part, newer games rise and older games slide down. Since the SDJ has been awarded since 1979, there are a lot of “old” games in this collection of SDJ Winners and considered games. Let’s take a look at how well the SDJ games have aged as measured by their BGG ranking.

From this graph it appears that some older games have become classics and newer games may not be faring as well as we might expect. As with the comparisons on game weight in Part 2 of this series, it appears that the SDJ games were gaining popularity until the KSDJ was started when there was a slide in the ranking as the SDJ was awarded to very light games and the KSDJ replaced the honors for heavier games. Note: The results for 2017 were captured right at the time the award was announced so we will see in a later article if the ranking improves with the familiarity promoted by the award itself. Note: There are a few games over the years that are not ranked at all (9 of the 238 games in the survey), so these have been removed for the purpose of calculating an average and may have affected the results for specific years. (EX: There were 2 games in 1979, one ranked 989 and the other not ranked).

Narrowing this down to just the Winners each year, we get a better picture of how the award ranks when the obscure family games that never break out of just the German market are removed.

Finally, the rest of the SDJ considered games by Award Level.

Again, let’s look at the Special awards separately, since the nature of the award has changed over the years.

Now let’s ignore the Award Level and look at the best games of all considered by year.

Again, we see a few classics represented and, not surprisingly, newer games generally rank higher.

KSDJ Considered Games

Now on to the KSDJ data. Since the KSDJ has been around for only 7 years it is easy enough to look at the average rank by year and award level. Here is what the highest KSDJ games by year looks like.

As we saw earlier with the aggregate results by Award Level, the Finalists and Recommended games fare at least as well as the Winners of the KSDJ. 2016 appears to be a particularly good year.

Again, let’s look at the best of the best, the highest ranked KSDJ games by year.

End Game

The SDJ and, in particular, the KSDJ games are well respected on BGG and for a small percentage of all games on the site, have claimed high awards in the BGG ranking. While there is definitely a mixed bag of results, the awards represent what the judges believe to be the best family games for a very specific market, but manage to garner wide attention.

Chicken or the Egg?

Which comes first, the quality that appeals to both the BGG audience and the SDJ judges or a BGG ranking that follows the placement of the awards? Note: this would be really difficult to explore since many of the games have no audience outside Germany until after they are recognized by the SDJ and then gain international distribution. However, in our final article to this series, we will compare the ratings of the 2017 SDJ and KSDJ games at the time the awards were announced to their ratings several months later.

Additional parts to this article are available here:

• Spiel des Jahres & the BGG Community - Part 1
• The Full Weight of the Spiel des Jahres - Part 2
• How do Spiel des Jahres Winners Rate? – Part 3
• Buy and Play Spiel des Jahres Games– Part 4
• What Makes an SDJ Winner? – Part 5
• The Impact of SDJ Consideration – Part 6

Your Turn

Flavor Text

I would really like to see how these numbers compare to the complete list of games in BGG, particularly breaking it down by year produced, but there is no way for me to accomplish that (without full access to the BGG database).

A note about the charts: While the differences in the ratings are generally at the top of the chart, I intentionally kept the Y-Axis to the origin to show how those differences are relatively small compared to the scale.

Note: Although this part of the article series wasn’t published until January 2018, the data was collected in June 2017.