During the SMITE World Championship at Hi-Rez Expo, several games were paused or restarted due to technical issues. This led some of our viewers to question the administrative actions taken during these sets and why some instances warranted a game restart while others did not. In this report, we’d like to take this opportunity to discuss in-depth what happened with each match in question and further explain both the steps we took to resolve those issues in the moment and the steps we’re taking now to ensure that our fans can expect a better viewing experience in the future.
Quarterfinals: What Happened with Dignitas vs Knights?
During the Dignitas vs Pittsburgh Knights match, players were forced to pause multiple times during the match due to their keyboards not interacting properly with the game. At random points throughout their gameplay, the players’ keyboards would stop receiving input from a single key (such as the left/right strafe keys or an ability key), and as such would be unable to change movement directions or fire an ability. While we could quickly fix the issue by restarting the client or restarting the whole PC, it could crop up again any time after the initial “fix” — and it did indeed show up multiple times after that.
Because we had legitimately never seen any issue quite like this before, we did not have default administrative protocol in place for handling such a problem. So our league operations team had the players finish their match, then worked on troubleshooting the issue after games wrapped. We spoke with some members of the dev team and worked with them to investigate further, and we learned that the issue could be traced to one specific brand of keyboard that some players were using on stage.
While we were in the midst of troubleshooting this issue during the Dignitas vs Knights match, some of the players informed us that these exact same keyboard disconnects had been happening intermittently during their recent scrims — but that admission marked the first time an issue with this specific brand of keyboard had been officially reported to our league operations team. After checking all our correspondence records and our internal bug reporting systems for pro players, we found no evidence that this issue had been reported to any of our administrators before the SMITE World Championships.
In addition to thorough build testing from both our QA and administrative teams ahead of a major tournament, we allow our pros ample time to play on that build so they can ensure the build meets their competitive standards and identify any further issues that may not have been replicated during our internal tests. This way, we can ensure that fixes are implemented into these builds before they’re used in live events. Because the keyboard hardware issue was not seen throughout any of our in-studio matches from Phase 2 or the SWC playoffs, player reports are the only way we could have discovered and hotfixed this issue before it happened on the World Championship stage. Unfortunately no such report was made, and at the time of writing we’ve been unable to ascertain why this issue was not forwarded to the league operations staff as soon as it was discovered.
In an effort to solve the problem on stage, we checked all our tournament PCs to see if there were drivers or firmware updates that we needed to install for these keyboards — but this specific brand had a plug-and-play style setup and their only available utility software was for RGB lighting control. The players had not installed this RGB software on their tournament PCs during setup, so we confirmed that program could not be the source of the issue. We also took a look at our servers and found no issues logged during the time of the match. With no drivers or firmware to update and no evidence of a bug in our game files, we couldn’t isolate the exact source of the issue and had to move forward with the assumption that this specific brand of keyboard might intermittently experience problems during gameplay.
After the match, we were able to collect more information from our players on how this keyboard hardware issue presented itself in their scrims on different PC setups, we were able to ascertain that the problem was not specific to the tournament PCs we were using on stage. After we felt like we had as clear of an understanding of this issue as we could given the circumstances, we communicated what we knew to the players before the Dignitas vs Rival match began on Day 2.
Semifinals: What Happened with Dignitas vs Rival?
During the Paladins World Championship semifinals that preceded our Dignitas vs Rival semifinals match, our Paladins pro players experienced the same keyboard issues that we saw in the SMITE quarterfinals the day before, on exactly the same brand of keyboard. We knew that Dignitas had already experienced these issues and would likely be using the same keyboards for their upcoming semifinals matches, so our league operations team had a brief discussion with them prior to their Saturday match versus Team Rival.
In this conversation, we informed Team Dignitas that there was a high probability the keyboard issue would happen again during their game if they continued to use the affected keyboards. We told them that because this issue was occurring at random and couldn’t be isolated or fixed until after the event, we were willing to work with the team as much as possible on any pauses they needed to take if the issue cropped up again. We also let the captain know that we had a quick fix for the issue, which we felt was reliable enough that we could continue the games with minimal downtime. Our admins discovered a keyboard shortcut to “reboot” the keyboard without restarting the client entirely, and they would be able to use that workaround any time players began having issues.
During this conversation, the team captain asked how they should handle pausing for this issue, especially if they were in combat, as combat pauses are not allowed per league rules. We outlined the following procedure in the event that they keyboard bug happened:
- If the player is not in combat, they can immediately have the team pause and raise their hand to get an administrator’s attention.
- If the player is in combat, they must immediately have their team disengage from combat, then pause as usual and raise their hand for an administrator to assist them.
This protocol is consistent with how we’ve handled pauses for technical/hardware issues all year long, and the players indicated to us that they understood this procedure and agreed to follow it in the case of a keyboard issue.
When the match began, we saw the keyboard issue pop up in the early stages of Game 1. While in combat at the enemy speed buff, Trixtank (playing Xing Tian) was unable to use his third ability. Trixtank raised his hand to request administrative attention, but the game had not been paused yet. Because players were still in combat, the pause could not go through. After the players finally left combat, the pause went through and the admin team worked to resolve the keyboard issue using the reboot method outlined above.
After this incident occurred, the league operations team made the decision not to remake the game for a few key reasons:
- Potentially affected players were told beforehand there was a risk with using their hardware.
- The players did not follow the established protocol regarding combat pauses and attempted to flag down an administrator before disengaging from combat and pausing the game.
- The issue could have happened at any point during the match, and could have happened at significantly more impactful moments (ex. a Fire Giant fight).
- According to our ruleset, teams may request a remake if an SPL game if a player fails to load in or disconnects from the game at a point where no gods have been defeated and/or the in-game timer reads ten seconds or less after the spawn gate opens. Any remake outside of these parameters is up to the sole discretion of our administration team. In this case, the player had not disconnected from the game.
Because the team was aware that their hardware could fail or have issues, chose to proceed with potentially faulty hardware, and then did not follow proper combat pause protocol following a technical issues, our administrators chose not to provide a remake.
After Game 1, players were given the option to use a different keyboard if they wanted to remove the risk of issues entirely, and our administrators were willing to allow them up to 30 minutes of extra prep time to get used to a new keyboard, fix all their key bindings, etc. None of the players using affected keyboards chose to use a different one.
Grand Finals: What Happened with SK Gaming vs Rival?
Before we dive into the issues we experienced during the Grand Finals set, we’d like to emphasize that the technical difficulties we saw during the SK vs Rival match were not in any way related to the brand-specific keyboard issue we had encountered throughout the weekend. This was a completely different issue with completely different circumstances, which is why the administration team took different action in this set than they took in other games.
During Game 1 of the Rival vs SK set, there was a pause at 0:53 seconds in-game. PandaCat’s computer had completely crashed, forcing him to disconnect from the game entirely. At the time there was some talk that the crash could be related to a Persephone issue, but the crash did not produce logs so we were unsure if the issue was with the PC or the client. With no further information to go on, we didn’t want to jump to conclusions regarding a god that had been picked for this game and had not caused issues previously at any point during the tournament. So as per league rules regarding a remake when a player DCs, we defaulted to our standard procedure and asked our teams to remake the game with the same picks and bans.
Following the remake we went back into the game, only to see PandaCat’s computer crash once again. This time we were able to recover the crash logs, and the Technical Director of Titan Forge immediately came backstage to review them. While the production and development teams reviewed this crash log, the rest of the team took multiple steps to troubleshoot the issue further:
- Our administrators spoke to Team Rival about what took place in both instances leading up to the crash, with specific emphasis on any interactions they had with Persephone before the disconnect. Both crash instances happened around the same time, and in Game 2 he disconnected into the tower — granting SK Gaming a kill through no fault of Team Rival. We also learned that the bug may have been occurring due to PandaCat’s specific playstyle on Persephone (more details on that below).
- Our operations team looked at the server and confirmed that there was no evidence of a server-side problem. Even so, we switched to a fully operational secondary server to be on the safe side.
- Our IT team completely replaced PandaCat’s computer with a new one in case it was a PC issue.
Unfortunately, none of these steps resolved the problem. After completing our review of the crash log, we learned that Persephone was indeed causing the client crashes. But it didn’t appear to be base Persephone causing the issues; the Withering Bloom skin seemed to be the source of the problem, likely due to a missing skin texture in the game files that was potentially causing problematic interactions with Persephone, especially when paired with PandaCat’s specific playstyle. When playing Persephone, PandaCat will sometimes have multiple Harvest seeds being planted and blooming at the same time — and it seems to be that each time these multiple plants grew to a to their Flower stage, they would cause a client crash.
This explains why we didn’t experience these issues with Persephone until the Grand Finals. While Persephone had been played throughout the tournament, she had not been played with her Withering Bloom skin, nor had she been picked by PandaCat. So the two factors that seemed to contribute directly to the client crash were not introduced until this finals set.
Having isolated the likely cause, we made the decision not to allow any skins for the rest of the finals set, just in case there were other potential texture issues that we hadn’t discovered yet. But because PandaCat had not played Persephone at any other point during the tournament, we couldn’t be certain that the issue was exclusive to the skin he was using. PandaCat had reported that he’d experienced this issue before, and it appeared to be related to multiple plants being planted and blooming at exactly the same time — something only a high level player would be able to do with the right timing to make it work.
We couldn’t say with 100% certainty whether or not the client crashes would persist if PandaCat used the base skin with his usual playstyle, and we weren’t sure we could risk the competitive integrity of the match to test it out. So after some very difficult deliberation, our league operations team also made the call to ban Persephone for the rest of the set. While we understood that this would drastically change the pick/ban phase of Game 1 and perhaps the rest of the set, we believed it was more important for the competitive integrity of the match to ban the character outright.
We explained this decision to both Rival and SK prior to the remake, then continued on with the Grand Finals. After banning Persephone and all skins from play, we were able to continue the finals set without any further issues.
Why was a remake allowed in the finals when it wasn’t allowed in the semifinals? Don’t they both deserve remakes?
While we absolutely understand that our audience might see the keyboard bug and the Persephone bug as equivalent issues, there are a few key differences we’d like to highlight:
- The keyboard hardware issue was related to personal hardware that the players chose to continue using, in spite of being warned about the potential risks. The Persephone bug was a build issue on our end that the players could not have anticipated or preemptively avoided in any way.
- The keyboard hardware issue did not cause a player’s PC to crash and disconnect completely from the game — and it was easily fixed if players followed the correct pause procedure. The Persephone bug did cause PC crashes and disconnects and would disrupt the game regardless of whether or not pause procedure was followed, which is grounds for a remake according to our ruleset.
- In each specific instance of the keyboard hardware issue, it did not grant a kill to the enemy team through no fault of the affected team. The Persephone bug did grant a kill to an enemy team through no fault of the affected team, which brings additional competitive integrity concerns. While we understand that early invades are very impactful in the current pro league meta, and the issue during the Dignitas vs Rival set happened during this invade time, we have to make administrative decisions based on the game state and the competitive precedent as a whole, not solely on what meta is currently active.
As outlined here, there is a clear difference in circumstances between the games affected by the keyboard hardware issue and the games affected by the Persephone bug, which is why our league operations team came to different determinations regarding whether or not the games should be remade.
How Are We Fixing It For Future Events?
It’s not acceptable that these issues occurred during our most anticipated event of the year, so we’ve developed a few action items that will help us implement long-term solutions and hopefully mitigate future issues:
- The League Operations team will be doing more extensive testing on the special builds we use for our major LAN events.
- All pro players for Season 7 will be provided with clear documentation regarding bug reporting to administrators, with explicit guidelines regarding the promptness of reporting bugs ahead of major events and potential liabilities for failure to report known bugs to our administration team.
- The League Operations team will review our policy related to personal hardware issues, and discuss whether a different course of action in future cases like this could better uphold the competitive integrity of our game.
In addition to the changes outlined above, we want to take a closer look at how we treat pauses in pro league matches. Obviously in cases where a player is having a hardware or other technical issue, having to disengage from combat in order to pause can prove difficult to do in the moment, and having to do so can put either team at a disadvantage depending on how that fight is going. With that in mind, we’re considering a few updates to our current procedure.
There is currently a “free pause” system available in-game for custom SMITE matches that allows players to pause at any time, regardless of whether they’re in or out of combat. Our league operations team is looking into using this feature for next season for emergency situations. In the event of any game-breaking issues during a live match, players would not need to wait for combat to end before getting help from an administrator. Instead, they would be able to pause immediately, get the help they need, and attempt to resume the game at exactly the state they were in when the issue occurred. However, we realize that the free pause system could easily be abused without oversight. So there will be very strict parameters around what is considered an emergency situation, and these guidelines will be stated explicitly to the players before the 2020 season begins. Any player that does not follow the correct pause protocol or attempts to abuse the free pause system will be issued heavy penalties.
Once again, we’d like to apologize to our community for the technical issues that affected our World Championship games. We’d also like to apologize to the pro players and teams who were directly affected by these issues. We also appreciate the patience they exhibited on stage while we worked through each issue, and applaud the professional demeanor they displayed throughout this deeply frustrating experience.
As we move into next season, we believe the fixes we’ve outlined above will go a long way in helping us better cope with issues like this in the future, especially in live event environments where the on-site fan experience is a primary concern. We’ll also be keeping a close eye on the feedback we receive regarding this report to see if there are any other potential solutions we could explore.
Last but not least, we truly appreciate our community’s patience and understanding throughout the weekend. We strive to bring our absolute best to the World Championship stage. We hope the information we’ve provided above gives you more clarity about the decisions that were made in the moment, and helps our audience understand that each choice was made with the tournament’s best interests in mind.
If anyone has further questions about the information in this report or other outstanding concerns regarding any of the sets from the SMITE World Championship, please reach out to our esports community manager, TitanAuvey (Twitter: @auverin) to connect with us directly.