Hi everyone, today we wanted to provide an update on our servers and server performance. This should fill in the basics of what’s happened over the last few weeks, how we are handling it, and steps we're taking to prevent these issues from happening again.
(Note: To better understand what’s below: A server is the physical metal box in the datacenter that hosts several dozen server instances. A server instance is the “server” you play on during an Online or Unranked Match, represented by a name seen in the upper-left corner of your game when playing online -- USW734438, USW101-Aftershock, etc.)
After the Tournaments Update was released on April 3, we started seeing a bug that was causing several performance issues in both your game client -- the stuttering you may have seen when opening the Playlist Menu, for example -- and our servers. We fixed the bug in last Thursday’s v1.44 Hotfix -- “Fixed an issue that caused stuttering in the game client when secure TCP sockets are closed down” -- but the change allowed a new server-side software issue to surface.
This software issue, which appears to be unique to one of our server providers, was preventing the normal server instance restarts that happen either 1) on a timer, typically every 1-2 days, or 2) when a server instance hits certain triggers, like using too much of the server’s memory. With the automatic restarts not working properly, the servers took a sharp performance nosedive, leading to “lag” even when your ping was low, and poor performance for all players in a match regardless of their location or platform.
Over the last few days, we have upgraded both the operating system and the control panel software on every single server we have with this provider. 99% of these software upgrades will be complete by the time you read this post today. These upgrades fix the software bug and also give us the level of control we need to better pinpoint and address issues as they come up. The server reboots associated with these upgrades give us a fresh perspective on the server instance data we're collecting from them. We’re also looking into server hardware replacements in some regions, which will take several weeks to complete and should be seamless in regards to server access. While we are confident in these updates and changes we’ve made over the last few days, we are continuing to monitor server performance and will take additional action when needed.
Server issues like these are something we never want to see or have our community experience. It’s not fair to you and “fix your servers,” is not something we want Rocket League to be known for. We apologize for the disruption, and we appreciate everyone for sticking with us and being patient while we’ve worked through the problem. This is something we absolutely take to heart; it’s a top priority at the studio and has been for the last few weeks. Our affected server provider has been working with us around the clock to implement these software and hardware upgrades, and we have an internal group comprised of Infrastructure, Engineering, Analytics, Production and Community team members tracking this very closely. This group is also looking at the new expanded server data we started collecting shortly before the Tournaments Update went live and we hope that this new information will help us better track down and resolve server reports we previously couldn’t confirm.
If you continue to experience issues while playing online, please let us know through our support team at support.rocketleague.com or via @RL_Support on Twitter -- and we will take any details you can provide to inform any changes or additional fixes we need to make. Thanks, all!