My Experience Hosting an MMORPG on the Cloud
Before we start, I would like to clarify that this is not a hand-to-hand tutorial on how to lift your game server, but rather, a portray of how to take a somewhat monolithic application and modernize it to the edge of technology standards. I would like to reinforce, that software development consists of a lot of setting up and executing repetitive tasks; where you will spend more time on infrastructure and processes, than in actually creating new features.
This article is about investing in your initial infrastructure, so you can use the spared time to implement new features, and more importantly, to interact with your clients. I will talk about game servers, but you can simply see them as any arbitrary application.
Finally, as technology grows, what is being said today, may not be the recommendation for the next hour. However, I am confident that an initial investment in a robust and practical infrastructure, especially in the cloud, may save you a lot of time and money. Honestly, you will keep the programming fun aspect intact. Everyone wants to implement a new app, a new game, a new calculator, but once one starts to face non-related project challenges, it is hard to keep the motivation going.
Tibia
I would like to talk about Tibia, by CipSoft GmbH. Tibia is one of the earliest online multiplayer RPG games, dated back to 1997.
I had my first contact with Tibia when I was about thirteen years old. Tibia was those types of games that you would spend days, even weeks and months, farming, hunting, training, and talking to random players. It was an adventure to build your character and to devote time for it to grow. Likely-wise, sometimes you would lose some of your items and experience points, by either dying from strong monsters, hunted by other players, or even from losing your network connection in the middle of a hunt. A simple death could easily cost you two or three days of active playing. It was an emotional turmoil.
Well, honestly, I never found it so amazing to spend hours killing creatures. What brought me joy, was to facilitate my progress in some way. And you could have called me a cheater, because, I would track my mouse and keyboard strokes to create macros for repetitive tasks, like picking up loot from the ground; I would analyze the packets leaving and entering my network, to somehow, create scripts to mimic them; and in the end, I would eventually join a huge community of alike minded Tibia fans, that together, would lift the grounds for unique open-source Tibia emulators, so players like me, could build our Tibia servers, with our features, quarks, and perks. I am thankful for www.otland.net.
After ten years lifting my Tibia servers, whereas at some point, with quarter-million accounts registered in the database, with a record of over a thousand players playing simultaneously, I am here to share how is it like, to port a 1997-like game, to the cloud.
As a quick background, today I am a software engineer working in a health care company. I hold a master’s degree in scientific and high-performance computing, where I specialized in critical and performance sensible applications. Tibia is the reason for all of that. It is why I have leaned towards programming; toward foreign languages; and toward the overall learning and looking-after mindset.
Tibia Emulators
There are numerous Tibia emulators over there, but we cannot talk about Tibia emulators, without talking about the https://github.com/otland/forgottenserver.
https://github.com/otland/forgottenserver is a Tibia emulator written in C++. It provides all the infrastructure to bind your game-server to your network port, and then respectively exchange gaming packets with players using custom Tibia clients.
If you are further interested in Tibia; or/and knows nothing about C++, nothing to worry about. What makes https://github.com/otland/forgottenserver so attractive, is that it provides so many scripting possibilities, were creating new creature events, global events, talk events, new items, maps, and you name it, can all be done using scripting and markup languages, such as https://github.com/lua/lua and XML.
The Gaming Ecosystem
There are four pillars to lift your game server.
- A computer
- A database
- The game-server
- A website
Then, if you like to make sure you do not screw it up, a good security system! Especially for DDoS protection.
Today, there are Tibia alternative game servers hosting over 3000 players simultaneously. If you would like to serve your content without any delay, with low latency, low downtime, and no crashes, you better get those four points, at least, perfectly.
The Basic Setup
Let’s say you are a bit serious, but not too much. You would like to have your server hosted 24 hours a day, 7 days a week. So your friends, a couple of guys from your city, and maybe one or two peers lost in your country could play. What would that take?
First, you would rent a dedicated or a virtual private server (VPS). There, you have your Windows or Linux environment, with all the necessary packages installed (e.g. boost, Lua, MySQL, etc.).
You would install your database (e.g. MySQL); adjust some IP addresses here and there; some MySQL and other access tokens; and then compile your game server.
Fine! Just a little bit more. You will find yourself a nice open-source https://github.com/slawkens/myaac PHP website — which integrates easily to your game server — , configure your Apache or Nginx web-server, and then you would ensure that the website is publicly accessible and connects flawlessly to your database.
Finally, just to make sure you do not piss all your players off with lags and downtimes, you will probably do some firewalling and secure your database.
Alright. A half-day has passed with the initial setup, assuming that you are an experienced developer and have followed some previously written directions for your targeted Operating System. And I mean it. Things will not compile properly. Your website will not be accessible, or will not connect to your database. You will forget to open some ports to receive network connections, where no one will be able to access the game server. Things will still not compile.
Just so you do not forget, you already have committed a front payment for the dedicated hosting computer you rented! Time is money, and if you spend a week trying to lift your server, it will be a week of loss.
Have I talked about sending new game updates? Updating your website? Updating your hosting computer? Your database? Periodically backing up your database to avoid losses? Analyzing possible crashes? Running a secondary game server to test your latest features? Rolling back versions of your server, website, database, in case all have been wrong? Are you DDoS safe?
Now it comes to 2021. The containers. The cloud.
The Cool Setup
Your cool setup has three goals: automate repetitive tasks; remove OS dependency, and continuously deliver your new features.
Setting up the environment consists of many steps and it is highly application dependent; therefore I will not describe it in detail. I will just cover the services we have used, and depicts how it has helped us solve specific problems. It is important to keep in mind, that sometimes it seems faster just to roll your application in the most obvious and straightforward way. However, much of software development will consist of debugging, processes, and repetitive tasks. If you invest time to ensure you have an initial robust infrastructure, you will not have to worry about anything but new features.
Quick note: we previously mentioned an open-source PHP website for the game server. The problem with PHP websites is that they may not be too flexible. They can be slow, especially if they are tightly connected to a database. We have done it differently. Instead of having a pure PHP website, we have extracted the database connection interfaces and then transformed the previously-mentioned PHP project into a RESTful application. Then, we rolled our own React front-end which HTTP the backend. In this way, we can just freeze the game-server, the database, and the backend within a single EC2 instance, and then roll our front-end application effortless and safely (see more on Amplify on why this is a great deal).
First step: Docker for environment independency
All the portability and maintainability you need will first come from porting your applications to containers. In this step, we ensure that the game, the website, and the database can run on Docker containers. Later on, it will not matter which computer we run our applications, they just need to support Docker.
Of course, containers can introduce overheads, especially for latency sensible applications; however, as we will see, that are optimizations to facilitate and improve it. First, we carefully orchestrate our containers (using docker-compose), to ensure that all the containers can interact with each other (e.g. we need the game server to access the database). Later, we optimize the virtualization overheads, by, for example, configuring containers to use the host’s network. Do not try optimizing before things are fully functional.
At this point, your game server and website are accessible to the public and fully functional.
Second Step: GitHub Actions for CI/CD
There will be constant changes for all game parts, especially the website and the database.
There are dozens of tools for CI/CD environments. We have opted for the simple one GitHub Actions. We already use GitHub for our source control; then, we can just use the GitHub Action feature to ensure that new Docker images are created and pushed to a private repository (hub.docker.com) each time we drive new commits.
Rolling in/out updates will become just as easy as executing one or two commands in the terminal to update running containers.
For example, let’s say I have a new update to roll for the game server. I change the source code and commit to GitHub. The pipeline will be triggered, and it will generate a new Docker image available on Docker Hub. Now, the next time I restart the game server, I can just replace the tag with the new tag, do some cleanup to remove dangling images and volumes, and turn on the game again. 2 minutes have passed. Of course, you can further automate to automatically roll up new image tags for your containers, and do the proper cleanup. However, our game server cannot afford to be turned off at any time.
It is worth noting that depending on the application, you may have tools (e.g. Kubernetes) to run a new container from an updated image tag, while the old container is running. And meanwhile, transfer the traffic to the new container without service disruption. However, because our game server’s internal logic does not support live migrating, we do not bother about it.
Third Step: Amazon EC2 for the Game-Server
After having the game-server portable in a Docker image, this part is easy peasy. The majority of our players are from Brazil. Therefore, we rent an EC2 instance in Sao Paulo, install the Docker environment and then run the game-server and database images, using previously configured arguments. As seen on the docker-compose depicted.
Aside from the portability of using EC2 instances, you will get out of the box: great security configuration; the flexibility to scale down or up your machine; and monitoring.
With AWS, you can set up your firewall rules on external network hardware, so in case of an attack, your CPU is not exhausted trying to process all these random packets.
With the monitoring information you get from free, you know if you lack or overuse current resources, giving you the possibility to make smarter decisions in terms of their allocation. For example, downgrading your CPU when your machine is barely using 1% of it. Or downgrading your 16 GB of RAM, when your highest peak is 3GB. Saving money on unnecessary resources is always a priority.
Besides the aforementioned amazing infrastructure from a Cloud provider, you will easily have the ability to exchange, upgrade/downgrade, or transfer to another country your EC2 machines. As long as you have a static IP address, so you do not mess up your clients' lives trying to connect to your applications, migrating your application to a new machine is as easy as jumping from the first floor.
Fourth Step: Amazon Amplify and Route 53 for the Website
I love it. You do not even need to prepare any GitHub Actions for that. Amazon Amplify integrates to your GitHub repository and takes care of any changes to it. In case there is a new commit, a new build of your website will be built and released online. You will be able to push changes effortlessly.
Besides the amazing feature of continuous integration, Amazon Amplify provides:
Super-low latency! The websites hosted on Amplify are distributed among AWS network centers. This means, that your website will be hosted on multiple continents, seamless from the user’s knowledge. When a new player tries to access your website, AWS will determine the closest hosting center, and then serve the website from there. The whole world besides North Korea will have access to your website from a blink of an eye (assuming you did do a job making sure it is light! Please use async imports on your JavaScript websites, so only the necessary is upfront loaded).
By having a website hosted on a different computer than your game-server, you free the game-server CPU’s resources from handling HTTP requests and possible DDoS attacks. Most of the DDoS attacks will probably target your 80 port, and thus, sharing a website on the same game-server hosting computer is a bad idea. The fewer ports we have open, and the strictly the access it, the better.
With Amazon Route 53, you can buy your domain, link to your AWS Amplify for the domain concerns. Done.
Fifth Step: Amazon S3 for overall storage
We need a place to host our periodic database backups, core dumps for future analysis, among other things. S3 is just cheap and great. You get a couple of GBs for a couple of cents. S3 ensures your data is accessible and safe. S3 has many access control resources to ensure no intruders access your data. Plus, a high-reliability rate, meaning that your backups will always be safe. If one of an S3 instance blows up, your data will still probably be safe, since they are replicated in multiple instances.
Final thoughts
We come up with a super flexible game server, website, and database setup. If today I wish to close my game, tomorrow, setting up again will take about minutes.
The prices for AWS are currently the best in the market. Assuming you do not have any crazy sensible legal contracts for your data, I do not see why you should not give a shot on the Cloud. The setup may seem cumbersome, but again, think about it as a nice foundation that you will rarely have to touch again.
Finally, these Tibia projects are just hobbies and shouldn’t be seen as a way of making extra cash. Although the game servers are open sources and could be legally usable, the original Tibia client and its sprites are not. So if you roll your server, make sure you have your own client and sprites!
I hope you enjoyed the topic, and if you would like to discuss more technical details (or Tibia), I would be happy to do so =)