How Riot used social experiments on League of Legends players

League of Legends studio Riot Games explains how it has been using a behavior initiative to conduct subtle experiments on its large player base.

3

League of Legends traffics a lot of players on a daily basis--12 million at last count. That player count means it's bound to have some bad behavior, but it also gives developer Riot Games plenty of room for testing. It began a series of player behavior initiatives last year, and has detailed the results.

Ars Technica details the initiative, as explained by Riot Games at an MIT lecture, and panels at both PAX East and GDC.

Riot started with a scheme inspired by our own justice system, the Tribunal. It would but offending players on trial with a community-based jury ruling on what it should be done with them. Riot found that 80% of the time, the community agreed with its own punishments, and the other 20% of the time they were more lenient. That community involvement inspired the proper behavior initiative, which launched about a year ago.

Headed by doctors with PhDs in cognitive neuroscience, experimental psychology, brain and cognitive sciences, and human factors psychology, the initiative made a baseline of what constitutes "bad behavior" using chatlines. Its first experiment was to change the default for cross-team chat, so that players had to switch it on manually. It found a striking 30% difference, even though usage of the feature remained constant. It then found commonalities among the types of words bad players would use (racial and homphobic slurs, mostly), which let them predict which players would be "bad" after a single chat log with up to 80% accuracy.

It's next step was to add more information to players punished by the Tribunal, so they would know what they had done wrong. This showed the doctors a sharp decline in problem behavior after the bans, and what lengths of bans were most effective. Finally, the team decided to reward those participating in the Tribunals by showing individuals their stats, most pointedly how many "toxic days" their rulings had prevented. That led to a 100% increase in Tribunal judge participation.

Since then, Riot has turned its attention toward rewarding good behavior as well, through giving Honors status to players, and "priming" players with messages before matches started. It even offered different colors to attempt to suss out what colors are most effective which which types of messages, but those results have been inconclusive.

The studies on the whole are fascinating if you have an interest in science, so check out the full report for more details.

Editor-In-Chief
Filed Under
From The Chatty
  • reply
    May 16, 2013 1:30 PM

    Steve Watts posted a new article, How Riot used social experiments on League of Legends players.

    League of Legends studio Riot Games explains how it has been using a behavior initiative to conduct subtle experiments on its large player base.

    • reply
      May 16, 2013 9:10 PM

      Yeah, I got my Teamwork/Leadership Ribbon, and was initially PISSED when I got the Helpful one too, cause I'd rather have the Leadership Yellow. But my Yellow still outnumbers ... Green, I think.
      Apparently it's blending your 4 Ratings in some way; like I guess Leadership is Teamwork+Helpful, maybe some Honorable Opponent? I dunno.
      I saw a few Blue Ribbons recently. I think that was "Mentor / Super Helpful".

Hello, Meet Lola