mineOps - Part 2: Automation with Ansible

In Part 1 of this series we went over manually deploying a Minecraft Server as part of a fictitious business to show their journey of growth through DevOps. In this post we will go over automating all that was done in Part 1 through Ansible. We will also touch on some nice to haves through automation and see where we can make our lives easier for this company.

Installing Ansible

There are a few ways to install Ansible and all of those methods can be found in the Ansible docs: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html

I tend to use pip to install it Ansible, but the package manager of your system will work as well. Here are some quick examples on installing Ansible

• python -m pip install --user ansible
• yum install -y ansible

Setting up our inventory

If you are familiar with Ansible or have read the primer I wrote in the supporting mineOps GitHub repo, you should be aware of what an Ansible Inventory is. A quick overview is just a file containing the servers you wish to manage with Ansible. For the moment, we are going to create this inventory file wherever we are on our "control node" most likely our laptop or workstation for this business.

[minecraft]
mineops-0.kywa.io

So in this inventory file we have our single Minecraft Server called mineops-0.kywa.io. For now our inventory is quite simple, but we will be expounding on this later in this article.

Automating the install

Let's look over what all we did during the manual install process and see about converting it to Ansible.

In order we:

• Installed Java 17 (or later)
• Downloaded the Minecraft Server jar file
• Accepted the EULA for Minecraft Server
• Ran the server

After that we did a few other things such as creating a dedicated directory, a dedicated user, creating a systemd unit to start/stop the server without using up a terminal and creating backups of the Minecraft Server. We will begin by just automating the install going forward from there. Each of the Ansible Modules used in the beginning here will have a link to their documentation in a comment above the task.

The "basic" Ansible playbook can be found in the supporting repo: https://github.com/KyWa/blogs/blob/master/mineOps/files/Ansible/basic_ansible/minecraft-install.yml

As you can see this is fairly straightforward and essentially what we did manually, but all in one Ansible Playbook. We could technically do all of this in a BASH script and it would work just fine, but the issue with BASH scripts comes scalability and extensibility. There are some sysadmins I know who would argue with me to the end of time that BASH scripts can automate with the best of Ansible, but we will see here in a little bit why we would want to avoid that and stick with something like Ansible.

We can give our Ansible playbook a run and see what we get. Again note I'm passing an argument to ansible-playbook to specify my ssh user. This can be done in a file called ansible.cfg and we will get to that later in this post:

ansible-playbook -i hosts minecraft-install.yml -e ansible_ssh_user=root

If all went according to plan (which it should have) Ansible will have completed all the tasks we gave it and a running Minecraft Server should exist at mineops-0.kywa.io:25565. Please note this server isn't publicly accessible and only reachable from inside my home lab so don't bother going to test it out because it will not be there.

We have a working Minecraft Server (again)! Even though we didn't create the playbook in the first part of this blog post, the effort to write this Ansible Playbook to do essentially what we already had through a BASH script was a little on the high side (as is most automation), but the power isn't in automating something you could do manually, the power is in how you can use and re-use that automation. In the next section we will expound upon our Ansible Playbook and break it out into more modular Ansible items such as a roles and inventory vars.

There are a few more things we would need to do for a customer, but we can move along to the more "modular" approach to Ansible and tackle these other needs. To summarize the "basic" Ansible approach we have a playbook, an inventory file and 2 Minecraft specific files we are copying over. All of this sits in a single directory and to make any lasting changes we will have to edit this file or copy it and run it the new playbook. In the next section we are going to break this all apart and spread out most of this into an Ansible Role.

Upgrading our Automation

Using the more "basic" Ansible approach we are able to do quite a bit and get us where we need to be in terms of deploying a Minecraft Server. Now let's say the customer needs to make some changes. Maybe they want the Message of the Day changed for the server, or want to disable/enable PVP. First let's convert our single playbook into a role. This isn't a going to initially grant us any new automation ability, however it does enable us to be able to move things around and fine tune our deployments (especially if growing to more than one customer).

Compare these two directories:

$ tree basic_ansible/
basic_ansible/
├── eula.txt
├── hosts
├── minecraft-install.yml
└── minecraft.service

0 directories, 4 files


$ tree modular_ansible/
modular_ansible/
├── ansible.cfg
├── inventory
│   ├── group_vars
│   │   ├── customer-0
│   │   │   ├── config.yml
│   │   │   └── server-properties.yml
│   │   └── minecraft
│   │       ├── config.yml
│   │       └── server-properties.yml
│   ├── host_vars
│   │   └── mineops-0.kywa.io
│   │       ├── config.yml
│   │       └── server-properites.yml
│   └── hosts
├── playbooks
│   ├── minecraft-backup.yml
│   ├── minecraft-config-change.yml
│   ├── minecraft-install.yml
│   ├── minecraft-restore.yml
│   └── minecraft-uninstall.yml
└── roles
    └── minecraft
        ├── files
        │   └── eula.txt
        ├── tasks
        │   ├── backup.yml
        │   ├── config-change.yml
        │   ├── main.yml
        │   ├── restore.yml
        │   ├── server-prep.yml
        │   ├── start-server.yml
        │   └── uninstall.yml
        └── templates
            ├── minecraft.service.j2
            └── server.properties.j2

12 directories, 23 files

As you can see there is much more going on in the "modular" section. There is one item to point out and that the files under host_vars/servername and group_vars/groupname do not matter. Their names and layout are purely for those who use Ansible to be able to identify which files contain which variables. Ultimately it is the same data as in the basic section, but more spread out and "modular". One of the largest changes is in the inventory section. Our previous example had a single inventory file with nothing special about it and if we attempted to add variables to it, we would quickly lose our ability to easily see what we had. Enter group and host vars. Breaking up variables per group and even per host can help us tackle multiple instances and multiple customers with ease.

$ cat inventory/hosts
[minecraft:children]
customer-0

[customer-0]
mineops-0.kywa.io


$ tree inventory/
inventory/
├── group_vars
│   ├── customer-0
│   │   └── config.yml # Contains Customer specific variables
│   └── minecraft
│       └── config.yml # Contains variables for ALL Minecraft instances
├── host_vars
│   └── mineops-0.kywa.io
│       └── config.yml # Contains Server specific variables

With our example inventory file and the comments of which config.yml file contains what, it should be apparent the benefits of using this model. Going forward with Ansible and having multiple customers, it may happen quickly where you end up with multiple variable files in your group/host var directories. This is expected and placing your variables in files related to what they are will help greatly as scale occurs. Obviously you can throw every variable into a config.yml, but that isn't very descriptive. Minecraft Server may not have many variable files, but in other projects where you would use Ansible, this can grow to some very large files and separating them out will greatly increase your sanity when hunting for them.

One such inventory variable file will be server-properites.yml and as you can probably guess, this will mirror the server.properties file that Minecraft Server creates. Remember when I mentioned things we would need to change for a customer? Well this is where we can do that (as this is really the only area they would want things changed in most circumstances). In the "modular" Ansible directory we can see I have converted the server.properites into a new file called server.properites.j2. Ansible utilizes the Jinja2 templating language and we can make use of it here. As with any variable you can see all of the properties values now equal an Ansible variable:

$ cat roles/minecraft/templates/server.properties.j2
enable-jmx-monitoring={{ enable_jmx_monitoring }}
rcon.port={{ rcon_port }}
gamemode={{ gamemode }}
enable-command-block={{ enable_command_block }}
enable-query={{ enable_query }}
level-name={{ level_name }}
motd={{ motd }}
query.port={{ minecraft_query_port }}
pvp={{ pvp }}
difficulty={{ difficulty }}
~~~~~OMITTED~~~~~

Each of these variables (outlined by their {{ name }} syntax) has a corresponding variable set in modular_ansible/inventory/group_vars/minecraft/server-properties.yml.

$ cat inventory/group_vars/minecraft/server-properties.yml
enable_jmx_monitoring: "false"
rcon_port: 25575
gamemode: survival
enable_command_block: "false"
enable_query: "false"
level_name: world
motd: A Minecraft Server
minecraft_query_port: 25565
pvp: "true"
difficulty: easy
~~~~~OMITTED~~~~~

Handling our variables in this way allows us to push out changes per customer as we will see in just a moment. As we saw in our breakdown earlier of the inventory file hosts up above we see that we have a group called: customer-0. This is a unique group that is part of the minecraft group and will inherit all variables in the inventory/group_vars/minecraft/ directory. However with the way that variable precedence works in Ansible, if we place customer specific variables in their groups_vars or host_vars they will be what is ultimately pushed out to the server. Here is an example with our "modular" Ansible:

$ grep -r motd inventory/
modular_ansible/inventory/group_vars/customer-0/server-properties.yml:motd: "Welcome to customer-0 Minecraft Servers!"
modular_ansible/inventory/group_vars/minecraft/server-properties.yml:motd: "A Minecraft Server"

$ cat modular_ansible/inventory/hosts
[minecraft:children]
customer-0

[customer-0]
mineops-0.kywa.io

In the above output we have 2 motd variables set, only one of them will be used. From what we see in our inventory file, the child group customer-0 will take precedence over the "parent" minecraft group and the server.properties file will have the motd of: "Welcome to customer-0 Minecraft Servers!". This will remain the same for all Ansible you utilize so there is not much more need for examples around this. If you wish to know more about Ansible and variables always check the docs: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html. Another item of note around variables is their names, with some of them being special and reserved inside of Ansible. There is a doc outlining each of them and their use: https://docs.ansible.com/ansible/latest/reference_appendices/special_variables.html

Operating a Minecraft Server

I've created a few tasks in the role minecraft to handle a few nice things for us and our customers. We have a backup, restore and config-change task as well as an unfortunate uninstall if a customer decides to leave us. We can demo these out, but they are fairly straightforward. Let's at least look at the backup task to get an idea of what we are trying to accomplish.

Backing up a Minecraft Server

- name: Mount the Minecraft Server Backup Share
  mount:
    path: /mnt
    src: "{{ minecraft_backup_share }}"
    fstype: nfs
    state: mounted

- name: Backup the systemd servie
  copy:
    remote_src: yes
    src: /usr/lib/systemd/system/minecraft.service
    dest: /mnt/minecraft.service

- name: Stop the Minecraft Server Service
  systemd:
    name: minecraft
    state: stopped

- name: Archive and Backup the Minecraft Server Directory
  archive:
    path: "{{ minecraft_directory }}/*"
    dest: "/mnt/minecraft-backup.tgz"
    format: gz
    mode: 0644

- name: Start the Minecraft Server Service
  systemd:
    name: minecraft
    state: started

- name: Unmount the Minecraft Server Backup Share
  mount:
    path: /mnt
    state: unmounted

In this particular instance we are using an external NFS server to hold our backup data, but this could technically also be Google Drive, a Windows SMB server or anywhere where files are accepted. I will be using NFS as its the easiest to setup and get working in a Linux environment. The task above mounts a share (NFS) stops Minecraft Server, copies the data over to the share along with its systemd service, then restarts Minecraft Server and finally removes the backup share to create an "air gap" of sorts. This last step is important to not leave your backup shares mounted to any system where you hold your backups dear to you. Most Ransomware goes through all mounted directories and disks in Linux and Windows. Keeping your backups disconnected from your production servers will save you heartache, time and money down the road.

Reconfiguring a Minecraft Server

Now that we have a customer with a running server and we've automated its install, they want to make a change. What do we need to change and where do we keep that data? As mentioned previously, most changes will take place in the server.properties file which we have templatized and put on the each server during install time. If a customer wants to make changes away from the default configuration we provide, the customers server should have their own server-properties.yml updated in the host_vars section of the inventory. Let's take a look at this example, the customer wishes to remove PVP from their world because they want to invite a bunch of people, but don't want to have to worry about their friends losing their stuff. Here is what we would add:

$ cat inventory/host_vars/mineops-0.kywa.io/server-properites.yml
pvp: "false"

In the "basic" Ansible to push this change out, we would have to do quite a bit to go and edit the server.properties file on the Minecraft Server and restart the server. Here is what it would look to complete this in the "basic" Ansible fashion:

• Add a task to run the lineinfile module against the server.properties file to change the requested property
• Add another task to the playbook to stop the server and another to start it back

If we add this to our existing playbook we won't disrupt anything, but a lot of steps will have to be gone through again (even if they are already completed) just to make this one change. By using the "modular" Ansible approach via a role we can have a task file for a start of the server such as start-server.yml. This task file can be called from other tasks in a role via the include_tasks: mytask.yml parameter making it far easier to extend your roles with other tasks you have created. This makes it far easier to add in tasks without having your primary playbook grow to unnecessary lengths.

For the sake of reconfiguring a Minecraft Server, we can use the task in the "modular" Ansible role called config-change.yml. Let's take a look at this task:

---
- name: Create ResourcePack directory
  file:
    path: "{{ minecraft_directory }}/resourcepack"
    owner: "{{ minecraft_user }}"
    group: "{{ minecraft_group }}"
    state: directory
    mode: 0755
  when: resource_pack is defined

- name: Create server.properties from template
  template:
    src: server.properties.j2
    dest: "/tmp/server.properties"

- name: Create server.properties from template
  copy:
    remote_src: true
    src: /tmp/server.properties
    dest: "{{ minecraft_directory }}/server.properties"

- name: Stop the Minecraft Server
  systemd:
    name: minecraft
    state: stopped

- name: Starting up Minecraft
  include_tasks: start-server.yml

This task will create a directory for resource packs if this config has been defined, create a new server.properties file from our template, stop the Minecraft Server, put the new server.properties file in place and issue the task to start the Minecraft Server up. The last section as you can see is calling the include_tasks we discussed earlier. This particular task is the same task we use during our main install in the "modular" Ansible configuration.

Adding a new Customer

As a business we've grown and through word of mouth, we have another customer at last! Using our "modular" Ansible we can add in their new group/host vars and update our inventory file and we can provision their new instance. Let's look at the updated inventory file and inventory vars:

$ cat inventory/hosts
[minecraft:children]
customer-0
customer-1

[customer-0]
mineops-0.kywa.io

[customer-1]
mineops-1.kywa.io

$ tree inventory/
inventory/
├── group_vars
│   ├── customer-0
│   │   ├── config.yml
│   │   └── server-properties.yml
│   ├── customer-1
│   │   ├── config.yml
│   │   └── server-properties.yml
│   └── minecraft
│       ├── config.yml
│       └── server-properties.yml
├── host_vars
│   ├── mineops-0.kywa.io
│   │   ├── config.yml
│   │   └── server-properites.yml
│   └── mineops-1.kywa.io
│       ├── config.yml
│       └── server-properites.yml
└── hosts

7 directories, 11 files

We've created our new variables and updated our inventory file, how do we deploy this new customer server? There are a few ways with the first of which being to just run the playbooks/minecraft-install.yml again, but this will attempt to do this for all servers in the inventory file. This isn't going to cause any issues because we have built our Ansible to not do any damaging or disruptive tasks. Ansible itself only "makes changes" to items that require changes. For instance the Install Java task which uses the dnf module will check to see if java-17-openjdk is installed and only install it IF it isn't there already. Now there is a caveat to this, the shell and cmd modules just run whatever commands you pass them so this is just something to be aware of.

Due to the caveat above regarding the shell and cmd modules in our server-prep.yml task file we have a stat task which will verify the existence of the server.jar. This will prevent the task downloading a new server.jar and ultimately ensure we do not replace it with a newer or different version.

To deploy just the new customers server we can run our install playbook with a flag at runtime called --limit. This limits the playbook to just the group or host you specify. So to deploy the new customers server, all we would have to do is run ansible-playbook like so:

ansible-playbook -i inventory/hosts playbooks/minecraft-install.yml --limit mineops-1.kywa.io

You can use the host or the group name inside of --limit, but for making changes to an entire customer's group of servers, you would most likely (depending on the change) run the --limit against their group as opposed to a single host.

Storing our Ansible files

By this point we have our business running with our customers happy and all of these Ansible files (hopefully the "modular" kind), but what happens to these files if your computer crashes? Where are these files being stored? What if the team grows beyond you and you have other people updating these config files? Do you use DropBox or Google Drive and share out these files? You could do that and many businesses get away with this, but there is a better way and that way is through a Version Control System.

There are a few VCS or Source Control systems out there, but there really is only one that people use across the board and that is called git. Git has pretty much taken over every workflow out there and is even used for greater deeds than just versioning source code. This however will be tackled in the next blog post. See you for the next one!

Series Links:

• Part 1 - Manual Minecraft Server Installation
• Part 2 - Automating with Ansible
• Part 3 - Keeping it Altogether with Git
• Part 4 - Containers Have Joined the Party
• Part 5 - Making Containers Highly Available
• Part 6 - ArgoCD to the Rescue