In these times of self-isolation, plenty of folks are going back to their roots in the Minecraft-multiplayer scene to make big projects, like for example, the BarelyAlive virtual shows. In this article, I will guide you towards setting up a network that's ready for the masses. Note that this is not relevant at all if you seek to just set up a small server for you or your friends, we are talking about handling thousands of players in a Minecraft music event.
Let's start with basic terminology for those who are completely new.
— Ticks and TPS
A game tick is 1/20th of a second (or 50 milliseconds), everything Minecraft does is scheduled in a tick. From mob-AI updates to interaction handling everything always happens in a set order. This is because the core of the Minecraft server is single-threaded. In the ideal scenario, your server will always run at a perfect 20tps (ticks-per-second). You can best see the 50MS you have per tick as currency your server gets to spend before it starts to get behind on its schedule. If your tick-time goes above the 50MS that you have, you'll see a decrease in total TPS which leads to performance and timing problems.
— Redis
Redis is a caching and message queuing solution used in tons of enterprise applications. Redis is just like your Minecraft server a process that has on your server and demands its own resources. We'll be talking a fair bit about Redis so if you have never heard about it before you should definitely check it out.
— Event
Interaction by the player that causes instructions to be run on the server. There are simple events that listen to player interactions, block placement, etc, but also some heavier events like player movement that fire every tick that a player moves.
More memory (RAM) != Better
The first reaction people have when setting up a server for a larger audience is We'll need much more ram! And while they aren't wrong, they aren't correct in the way they think they are.
Issues with memory usually aren't the cause of the problem but the result of poor planning or development. The thing with any managed language (including Java) is that when you allocate it more resources, it will have to spend more resources to manage these resources. Confused yet? completely understandable. Let's break it down. Your Minecraft server itself probably doesn't need that much ram (assuming that the world where the event takes place isn't insanely big), 8GB for a server with 50 players is fine, but that doesn't mean that you need to upscale to 16GB to take on 100 players.
Your ram will fill up over time, and when it gets to a certain threshold, Java's garbage collector will kick into scan through the memory and delete data it no longer uses, thus freeing resources up again. This is why the memory usage often resembles a sawtooth pattern when being looked at in a system monitor. Sounds great right! Yes, it does, but it seriously poses some issues when you start working with either a really inefficient piece of software that leaks memory (like poorly written plugins) or allocates too much or too little memory. Everything the server does essentially gets paused while the garbage collector is doing its magic, this often isn't more than just maybe a few milliseconds but when you leak memory it can happen far too often, or have too much memory it can take too long to where it becomes noticeable.
What do you need then?
Well, it greatly depends on your setup. Small minigames can easily run on 500MB while your event server might need a whole 16 gigs, you are best of starting at 4/8 and slowly work your way up ONLY IF you are experiencing issues that are memory-related and there is really nothing else you can do (replacing plugins or if you wrote them yourself, fixing them)
Increasing memory should be your last course of action if there's nothing else you can do. It's like fixing a boat with staples, yes, when using enough it will work but you replaced the problem instead of actually fixing it.
Pro tip: Aikar has a great article about tuning the garbage collector for Minecraft
Now that I've told you what not to do with your resources and smashed your dreams and excuses for a 1TB Minecraft server, what else can you do?
In most cases, it's unlikely that one server will be enough since it's clearly not efficient to make one big-ass server and cross your fingers that it'll all work out since there are a bunch more issues like Networking, CPU, etc.
One very common practice is balancing your player base across multiple Minecraft servers (you've seen it before, lobbies, minigames, and whatnot) but you should probably do it too. You could do so by duplicating what's happening across multiple servers and evenly spreading your player base to fill them all round-robin style. One useful tool to do this is Redis. It is mainly known for its incredible caching mechanism but it also does a great job at relaying data between servers, which can be used to synchronize chat, commands, moderation and much more. Minecraft itself doesn't support Redis natively but there are some plugins out there that help you with it (but you are best building something yourself that suits your needs)
If things are getting really intense and you are getting even more than you anticipated to the point where players have trouble connecting or with keeping their connections up you might want to look into running multiple bungeecord servers as well, one ready and public resource to do this is called RedisBungee which also uses Redis to synchronize your bungeecord instances to be one big network. At some point, after hours of constant player logins, Mojang might not be too happy with you and their rate limiter will slow you down or stop accepting player requests, there are multiple workarounds for that too of which you can find one on my GitHub (which is what we used for the BlockParty event).
There's also plenty you can do to look for further problems and diagnose underlying issues. Spigot comes bundled with a timings system which is, in essence, a laid-out map that tells you exactly which events and plugins are listening and doing difficult processing on certain Events and thus might cause your other problems. Before running a public test, you should execute /timings on and then continue as you would, afterwards, do /timings paste and it will give you a URL to a handy-dandy web page telling you where your tick time spent. Look for listings in red/yellow and what plugin they correspond to and look into replacing it or notifying the author of the said plugin.
But then the fun stuff! In my opinion. How are the actual shows ran?
OpenAudioMc has a show editor that can be used to schedule shows with commands, timecodes and manage them all from GUI (start, loop, repeat and all that jazz). The main barely alive event was running based on this system along with some extra add-ons made by Thomas. A Redis feature was later added to synchronize these shows over multiple event servers. This means that one “master” server is sending commands to other servers to replicate what’s happening on the main stage. This keeps timecodes in-sync across the network while keeping us in complete control to halt or jump in whenever needed. We also installed 4 dedicated OpenAudioMc relay servers to allow as many people as possible to connect to the audio client.
This takes care of the show, we also wanted to synchronize the artists across all the servers.
To do so, Thomas and I wrote a plugin that creates a fake player controlled by us via NMS (Also known as an NPC) and distributes player updates (so that’s Chat, Movement, Armor, Interactions, Shifting and more) over Redis to the other servers that update their Display of the NPC. (Unlike OpenAudio, this one currently isn't public)
The Lights (lasers, particles and other effects) were also managed via a custom plugin I made and later expanded by Thomas to include effects like the ender dragon death. This plugin uses a bunch of techniques, math and native Minecraft hooks to create our visuals. It hooked into the aforementioned OpenAudio shows which controlled the effects and contained the animations. The ShowsPlus plugin (as it is called) is currently not public but I do have plans to release it eventually. (it’s not on my high priority list but let me know if you think it should be)
As briefly mentioned earlier in this article, we used a custom build of BungeeCord that caches game profiles (so that's security session checks, player details en skins) in a safe way (once again using Redis) which greatly reduces the strain we put on mojangs servers, our own bandwidth, improves response times for returning players and even allows us to accept players in case of a Mojang service outage where you normally can’t join servers (this fork of bungeecord can be found for free on my GitHub.
This covers most of the basic principles you can use to make everything run as smoothly as possible. This article is not intended as a step by step guide, but inform you about and explain systems that might be unknown to you that will come in handy.
If you have any further questions, tweet me @Mindgamesnl and I'll try to get back to you.
Have fun! :)
