Container Development - Getting Started

Application development is something that can be done so many different ways its hard to keep track of all the tools that are available. Some just use whatever standard toolset is available while others try to find "3rd party" tools for their chosen language to help compile/build their application. And then there is application development which is going to be built and then run within a Container. The tools for the Container landscape are massive and grows more every day. Some tools are there to make life easy and others are there because we needed a new standard for something (relevant xkcd).

I am here to bring some light on a tool that isn't necesssarily meant to be a new standard, but a tool that can help application developers looking to turn their application into something that can be built easily and run in a Container.

NOTE Please note there are some starting knowledge required to make use of this post. If you are unfamiliar with building Container Images, I'd recommend taking a look at the 4th blog post in the MineOps series here for a better understanding: https://blog.kywa.io/mineops-part-4

Building a Container Image

There is really only one way to build a Container Image and that is with the use of a Dockerfile. This file contains instructions on how to build a Container Image. It typically starts with a FROM line which indicates the "base" Container Image to use and then other instructions like COPY or RUN to copy files into the Container Image or run some commands (such as installing packages) in the Container Image. These instructions that get run during build time are what the resulting Container Image will have "in it". The other important part is what does the resulting Container Image do when it starts? This is typically handled by an ENTRYPOINT or CMD (or both) instruction in the Dockerfile (or inherited from the base image).

Let's look at a basic example of a Dockerfile to build a simple application that runs a Python Flask application:

# We are going to start with the official Python image from Docker Hub
FROM docker.io/python:3.8

# This sets our working directory (where the Container will "start" from)
WORKDIR /python-container

# Getting our applicatoin code into the container
COPY . .

# Install our dependencies
RUN python3 -m pip install -r requirements.txt

# Tell the Container what to do when it starts
CMD ["python3", "-m" , "flask", "run", "--host=0.0.0.0"]

The above Dockerfile will result in a Container Image that will start a Flask application:

user@workstation-$ docker build -t pyflask .
[+] Building 3.3s (9/9) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 214B                                                                               0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/python:3.8                                                      0.0s
 => [1/4] FROM docker.io/library/python:3.8                                                                        0.1s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 180.35kB                                                                              0.0s
 => [2/4] WORKDIR /python-container                                                                                0.0s
 => [3/4] COPY . .                                                                                                 0.0s
 => [4/4] RUN python3 -m pip install -r requirements.txt                                                           2.9s
 => exporting to image                                                                                             0.1s
 => => exporting layers                                                                                            0.1s
 => => writing image sha256:7994aab4a965fc6b739456f0c0f738ca3943ccb77f9ff9ad6dbb67aaf82c2514                       0.0s
 => => naming to docker.io/library/pyflask                                                                         0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

user@workstation-$ docker run -d -p 5000 pyflask
8151bd52f0ea875750f7c832de8ab7f37bdd37fef1d657b7b69e6e184070aae9

user@workstation-$ docker ps
CONTAINER ID   IMAGE     COMMAND                  CREATED         STATUS         PORTS                     NAMES
8151bd52f0ea   pyflask   "python3 -m flask ru…"   3 seconds ago   Up 3 seconds   0.0.0.0:62069->5000/tcp   cool_lamarr

user@workstation-$ curl http://localhost:62069
<!doctype html>
<html>
  <head>
    <title>flaskdev</title>
  </head>
  <body>
      <h1>Container Flask</h1>
  </body>
</html

user@workstation-$ docker logs 8151bd52f0ea
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on all addresses (0.0.0.0)
   WARNING: This is a development server. Do not use it in a production deployment.
 * Running on http://127.0.0.1:5000
 * Running on http://172.17.0.2:5000 (Press CTRL+C to quit)
172.17.0.1 - - [27/Jul/2022 12:22:33] "GET / HTTP/1.1" 200 -

This is fine and works well for most people who know how to create a Dockerfile and put the specific instructions in there. What about those who maybe don't know or are unsure of what they need to do to run their application in a Container?

Enter Source-to-Image, or s2i

What is s2i

I am not known for my wordsmithing skills, so I will let the s2i repository speak for itself:

The big tl;dr of s2i is you can build your application with no need to worry about how to actually build it. With s2i you get a few components to handle things for you. The first is an assemble script which handles the building of the application. The next is a run script that handles the running of the application. The run script has logic to essentially see what packages exist and then determine which path to start the application (primarily a Python situation). The logic in both scripts is fairly straightforward and works for most situations, but can also be expanded upon by creating your own scripts which allows you to add extra logic into them. We will look at how to customize this towards the end of this post.

Using the s2i binary

Since we talked about not needing to know how to actually build your application, lets do this without a Dockerfile first. The s2i binary allows you to do a few things. First of which and the most important to us, is a build. It will make use of docker or podman, whichever it finds in your PATH and then run a build with what is specified. The other common alternative, and this is useful for implementing this into a CI/CD system of some kind, is a generate function. This will "do" the s2i process, but only up to generating a Dockerfile which can then be used or customized further.

s2i build

This will handle a couple of things, first it will look at the content its given (either by local directory or a git clone) and determine how it should be built. It will then generate a Dockerfile which it will then build.

The following will build a Container Image from a local directory

$ s2i build . registry.access.redhat.com/ubi8/python-38 somefuns2i
---> Installing application source ...
---> Installing dependencies ...
Collecting click==8.1.2
Downloading click-8.1.2-py3-none-any.whl (96 kB)
Collecting Flask==2.1.1
Downloading Flask-2.1.1-py3-none-any.whl (95 kB)
Collecting itsdangerous==2.1.2
Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting Jinja2==3.1.1
Downloading Jinja2-3.1.1-py3-none-any.whl (132 kB)
Collecting MarkupSafe==2.1.1
Downloading MarkupSafe-2.1.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting Werkzeug==2.1.1
Downloading Werkzeug-2.1.1-py3-none-any.whl (224 kB)
Colleing gunicorn-20.1.0-py3-none-any.whl (79 kB)
Collecting importlib-metadata>=3.6.0
Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Requirement already satisfied: setuptools>=3.0 in /opt/app-root/lib/python3.8/site-packages (from gunicorn==20.1.0->-r requirements.txt (line 7)) (41.6.0)
Collecting zipp>=0.5
Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB)
Installing collected packages: zipp, MarkupSafe, Werkzeug, Jinja2, itsdangerous, importlib-metadata, click, gunicorn, Flask
Successfully installed Flask-2.1.1 Jinja2-3.1.1 MarkupSafe-2.1.1 Werkzeug-2.1.1 click-8.1.2 gunicorn-20.1.0 importlib-metadata-4.12.0 itsdangerous-2.1.2 zipp-3.8.1
WARNING: You are using pip version 21.2.3; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/app-root/bin/python3.8 -m pip install --upgrade pip' command.
Build completed successfully

Same as above, but will pull from a Git repository:

$ s2i build https://github.com/someuser/somerepo registry.access.redhat.com/ubi8/python-38 somefuns2i

If we check our local images, we will see that we have a new image:

$ docker images
REPOSITORY       TAG          IMAGE ID       CREATED          SIZE
somefuns2i       latest       e1f3884208c1   34 seconds ago   861MB

s2i generate

What if you don't want to build the image right away and want to have it handled somewhere else down the line such as some CI/CD tool? You can use the generate function to generate a Dockerfile (but you can call it whatever you want) which can then be handed off to something else (such as buildah, podman or even docker itself).

$ s2i generate registry.access.redhat.com/ubi8/python-38 Dockerfile
$ cat Dockerfile
FROM registry.access.redhat.com/ubi8/python-38
LABEL "io.openshift.s2i.build.image"="registry.access.redhat.com/ubi8/python-38" \
      "io.openshift.s2i.scripts-url"="image:///usr/libexec/s2i"
      
USER root
# Copying in source code
COPY upload/src /tmp/src
# # Change file ownership to the assemble user. Builder image must support chown command.
RUN chown -R 1001:0 /tmp/src
USER 1001
# # Assemble script sourced from builder image based on user input or image metadata.
# # If this file does not exist in the image, the build will fail.
RUN /usr/libexec/s2i/assemble
# # Run script sourced from builder image based on user input or image metadata.
# # If this file does not exist in the image, the build will fail.
CMD /usr/libexec/s2i/run

Using an s2i base image

If you wish to customize the Dockerfile for your build, there are base images that Red Hat has created that can be used for free based on the Universasl Base Image (UBI). These images can be found from the Red Hat Container Catalaog located here: https://catalog.redhat.com/software/containers/search

Although there are plenty of languages that have pre-built s2i images ready for use, we will be focusing on Python. For reference we will be using this Python 3.8 image for our testing. At the bottom of this post you will find links which point to the various files that these images use.

Let's create a Dockerfile to make use of this image (found in the Python SCLORG link down below):

FROM registry.access.redhat.com/ubi8/python-38:latest

# Add application sources to a directory that the assemble script expects them
# and set permissions so that the container runs without root access
USER 0
ADD . /tmp/src
RUN /usr/bin/fix-permissions /tmp/src
USER 1001

# Install the dependencies
RUN /usr/libexec/s2i/assemble

# Set the default command for the resulting image
CMD /usr/libexec/s2i/run

The neat thing about the above is that we don't have to change anything about our code or change anything in our Dockerfile. This Dockerfile will stay the same althroughout our work with Python and never really need to change.

Here is the output of building with an image like this:

user@workstation-$ docker build -t quay.io/kywa/container-development:latest .
#1 [internal] load build definition from Dockerfile
#1 sha256:355e2d734470eaccba3699918560cf74249427fab8fc8285115d7e37b9a2b02d
#1 transferring dockerfile: 452B done
#1 DONE 0.1s

#2 [internal] load .dockerignore
#2 sha256:f4561062f8f809345cf9a5a1d5a6d5423c7a4da4ee4d612192a431a0793e307e
#2 transferring context: 2B done
#2 DONE 0.1s

#3 [internal] load metadata for registry.access.redhat.com/ubi8/python-39:latest
#3 sha256:ce534b0fd64e6c22a3c7a7247a4d676ce56b73f1d40fc33c3418b830c2abbbc9
#3 DONE 0.6s

#5 [internal] load build context
#5 sha256:2a59b27c38b259862c7e3fd8e68c83e6e5cdde8834f720a528ef6dcd51308060
#5 transferring context: 2.47kB done
#5 DONE 0.0s

#4 [1/4] FROM registry.access.redhat.com/ubi8/python-39:latest@sha256:84639af111ab2b3fd7ca325f0912e405c117df31c976446b2a1902948f65a61d
#4 sha256:a46474a8524b7c703c05ae33a46c18a083294a470688705fe751eb7a26b2b9d4
#4 CACHED

#6 [2/4] ADD . /tmp/src
#6 sha256:57bd845075e1b8191c3629d7ad152b942f2c095798c555198a95876d545eb180
#6 DONE 0.0s

#7 [3/4] RUN /usr/bin/fix-permissions /tmp/src
#7 sha256:8189bedac5eec60275eafe95f067ae5eed3d0015100bf6496467d160526f9b0e
#7 DONE 0.5s

#8 [4/4] RUN /usr/libexec/s2i/assemble
#8 sha256:408bdc819e3ee26297474bac423e5a3f0be03ba1399f7b324c96a346a4928790
#8 0.521 ---> Installing application source ...
#8 0.562 ---> Installing dependencies ...
#8 0.877 Collecting click==8.1.2
#8 0.991   Downloading click-8.1.2-py3-none-any.whl (96 kB)
#8 1.048 Collecting Flask==2.1.1
#8 1.063   Downloading Flask-2.1.1-py3-none-any.whl (95 kB)
#8 1.096 Collecting itsdangerous==2.1.2
#8 1.111   Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
#8 1.168 Collecting Jinja2==3.1.1
#8 1.183   Downloading Jinja2-3.1.1-py3-none-any.whl (132 kB)
#8 1.295 Collecting MarkupSafe==2.1.1
#8 1.308   Downloading MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
#8 1.373 Collecting Werkzeug==2.1.1
#8 1.389   Downloading Werkzeug-2.1.1-py3-none-any.whl (224 kB)
#8 1.435 Collecting gunicorn==20.1.0
#8 1.449   Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
#8 1.561 Collecting importlib-metadata>=3.6.0
#8 1.572   Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
#8 1.586 Requirement already satisfied: setuptools>=3.0 in /opt/app-root/lib/python3.9/site-packages (from gunicorn==20.1.0->-r requirements.txt (line 7)) (50.3.2)
#8 1.645 Collecting zipp>=0.5
#8 1.659   Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB)
#8 1.729 Installing collected packages: zipp, MarkupSafe, Werkzeug, Jinja2, itsdangerous, importlib-metadata, click, gunicorn, Flask
#8 2.113 Successfully installed Flask-2.1.1 Jinja2-3.1.1 MarkupSafe-2.1.1 Werkzeug-2.1.1 click-8.1.2 gunicorn-20.1.0 importlib-metadata-4.12.0 itsdangerous-2.1.2 zipp-3.8.1
#8 2.235 WARNING: You are using pip version 21.2.3; however, version 22.2.2 is available.
#8 2.235 You should consider upgrading via the '/opt/app-root/bin/python3.9 -m pip install --upgrade pip' command.
#8 DONE 2.3s

#9 exporting to image
#9 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#9 exporting layers 0.1s done
#9 writing image sha256:a4e6b75c423b51b4708f5ab99bfc7178e5fe4ac0c37035b869b695e344e532d6 done
#9 naming to quay.io/kywa/container-development:latest done
#9 DONE 0.1s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them

Customizing s2i

One of the other neat components of s2i images is that when it goes to build or run, it detects if your repository has its own .s2i/ directory and will use whatever scripts you have in there. Sometimes your build may need something other than just language libraries/modules. What if your application needs something else to run, such as ODBC drivers for a database connection? This is how we would do this, by customizing s2i with our own scripts that can extend upon what already exists.

If you are already using the s2i scripts to build and run your images, you do not have to actually change anything and can keep what is already there. Let's see how to do that:

user@workstation-$ tree
.
├── Dockerfile
├── .s2i
│   └── bin
│       ├── assemble
│       └── run
├── app
│   ├── __init__.py
│   ├── routes.py
│   └── templates
│       ├── base.html
│       └── index.html
├── main.py
├── requirements.txt
└── wsgi.py

[[In]] our repository, we have our main application code as well as a .s2i/bin directory with 2 files in it, assemble and run. These 2 files have different uses but are both picked up by the ENTRYPOINT scripts of the existing image. These 2 files will be run at build and run time (assemble during build and run during runtime), but you can also have these files just be specific customizations you want and then have them run the main s2i scripts after or before the customizations. For the assemble script, it could look something like this:

user@workstation-$ cat .s2i/bin/assemble
#!/bin/bash

# insert code sause for OS based depends
# this file can be used in place of a Dockerfile
# to patch s2i

echo "I'm a customization before the actual application build"
echo ""

${STI_SCRIPTS_PATH}/assemble

The above s2i/bin/assemble script will run during build time and will go and get some RPM file and install it, but right after will carry on with the rest of the s2i assemble process.

And the same is done for the run script:

user@workstation-$ cat .s2i/bin/run
#!/bin/bash

# this file can be used in place of a Dockerfile
# to patch s2i

${STI_SCRIPTS_PATH}/run

There is no real limit to what you can customize, just the limits of your imagination. You could even not call the existing run script and could have it do something very specific:

user@workstation-$ cat .s2i/bin/run
#!/bin/bash
export SOMEVAR=prod
python3 app.py

The above is a horrible example, but gets the idea across you could do whatever you need/want to do for your application. Just note this is going to be run during a step where you are a "mortal" user. If you are needing to complete tasks at an OS level, you will need to make use of a Dockerfile where you can specify the USER.

Using these custimzations looks like this:

user@workstation-$ s2i build ./ registry.access.redhat.com/ubi8/python-39:latest
I'm a customization before the actual application build

---> Installing application source ...
---> Installing dependencies ...
Collecting click==8.1.2
Downloading click-8.1.2-py3-none-any.whl (96 kB)
Collecting Flask==2.1.1
Downloading Flask-2.1.1-py3-none-any.whl (95 kB)
Collecting itsdangerous==2.1.2
Downloading itsdangerous-2.1.2-py3-none-any.whl (15 kB)
Collecting Jinja2==3.1.1
Downloading Jinja2-3.1.1-py3-none-any.whl (132 kB)
Collecting MarkupSafe==2.1.1
Downloading MarkupSafe-2.1.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
Collecting Werkzeug==2.1.1
Downloading Werkzeug-2.1.1-py3-none-any.whl (224 kB)
Collecting gunicorn==20.1.0
Downloading gunicorn-20.1.0-py3-none-any.whl (79 kB)
Collecting importlib-metadata>=3.6.0
Downloading importlib_metadata-4.12.0-py3-none-any.whl (21 kB)
Requirement already satisfied: setuptools>=3.0 in /opt/app-root/lib/python3.9/site-packages (from gunicorn==20.1.0->-r requirements.txt (line 7)) (50.3.2)
Collecting zipp>=0.5
Downloading zipp-3.8.1-py3-none-any.whl (5.6 kB)
Installing collected packages: zipp, MarkupSafe, Werkzeug, Jinja2, itsdangerous, importlib-metadata, click, gunicorn, Flask
Successfully installed Flask-2.1.1 Jinja2-3.1.1 MarkupSafe-2.1.1 Werkzeug-2.1.1 click-8.1.2 gunicorn-20.1.0 importlib-metadata-4.12.0 itsdangerous-2.1.2 zipp-3.8.1
WARNING: You are using pip version 21.2.3; however, version 22.2.2 is available.
You should consider upgrading via the '/opt/app-root/bin/python3.9 -m pip install --upgrade pip' command.
Build completed successfully

Conclusion

This has been a brief look at a technology for building and running Container Images that is outside the norm, but can provide some value to those who may not want or need to know "what" exactly is going on when building an image. I would however highly recommend that you do understand what is actually going on so that if the need arises when things get more complex.

Links

• s2i - https://github.com/openshift/source-to-image
• s2i Deep Dive - https://www.youtube.com/watch?v=flI6zx9wH6M&ab_channel=OpenShift (Somewhat older, but still valid)
• SCLORG - https://www.softwarecollections.org/en/
• GitHub SCLORG - https://github.com/sclorg

The following are for reproduceable Docker images in the following languages. Note there are others available, but these are the most common. Check the link above for PHP, Ruby and others.

Python

https://github.com/sclorg/s2i-python-container

NodeJS

https://github.com/sclorg/s2i-nodejs-container

.NET

https://github.com/redhat-developer/s2i-dotnetcore

Short History of CI/CD Tooling

For those that don't understand what Groovy is (the syntax used for Jenkins Pipelines), I'm happy for you. For those who don't know the person or the film the above image is from, please stop whatever it is you are doing (including reading this) and go watch Army of Darkness.

Before we dive into Tekton, let's take a quick look at "the big one" that has been used for many years now, Jenkins. Sure there are other tools out there that have been used over the years, but Jenkins has probably been the biggest one and had the widest use for the longest time. So what is Jenkins? Jenkins is an open-source "automation server", which really translates to a platform based on Java that is extensible with many plugins to do numerous things. It could be run anywhere and years ago would have many VMs dedicated to running large instances for an organization, or you could have it running in your Kubernetes cluster in containers.

One of the things that made Jenkins popular was a platform that was "extensible", but allowed you to have a someone custom execution environment. For the most part it was basically the ability to run shell commands with some plugins that took arguments. Here is a brief example of what a Jenkins pipeline would look like to build and deploy a new application:

pipeline {
   agent any
   stages {
       stage('Build') {
           steps {
               sh 'make'
           }
       }
       stage('Test'){
           steps {
               sh 'make check'
               junit 'reports/**/*.xml'
           }
       }
       stage('Deploy') {
           steps {
               sh 'make publish'
           }
       }
   }
}

As you can see from above, its somewhat logically laid out and clearly defines each stage of the pipeline. You may notice though that this basic pipeline is essentially running shell commands and nothing really special. The agent line up above specifies which "instance" to run on. One of the great things about this is you could have certain agents that had different tooling and were lighter weight. The big thing here I want to point out is the syntax of Groovy, which looks like JSON. Nothing is wrong with JSON except for the humans having to write it. Just to be clear, you can technically get this out of the Jenkins UI, but that isn't very declarative or repeatable, so we're going to focus on the manifest pipelines like above.

JSON style formatting is great for machines, but for humans it isn't exactly easy to read/write. Now the above is somewhat easy, but its only 1-2 line objects for each stage. As this would grow, or having other types of components in the various stages, the above would go from somewhat readable to a large collection of curly braces. Here is roughly the same pipeline, but in Tekton format:

apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: my-pipeline
spec:
  workspaces:
  - name: shared-dir
  tasks:
  - name: build
    taskRef:
      name: app-build
    workspaces:
    - name: source
      workspace: shared-dir
  - name: test
    taskRef:
      name: app-test
    workspaces:
    - name: source
      workspace: shared-dir
  - name: deploy
    taskRef:
      name: app-deploy
    workspaces:
    - name: source
      workspace: shared-dir

NOTE: I want to call out in the above is that it is 5 lines longer than the Jenkins pipeline and you are getting technically the same pipeline of tasks in the end.

For those who are familiar with Kubernetes manifests (and I hope you are at least somewhat, since we are looking at a tool/platform for Kubernetes), you will notice that the Tekton example above is just that, a Kubernetes manifest whereas the Jenkins file further above is "its own thing". Tekton itself is a "cloud-native" platform (aka Kubernetes native) that installs and becomes an extension of the Kubernetes cluster itself. The huge benefit here on its own over Jenkins is that, you can use your existing tools to work with Tekton since its "in cluster" and native.

Enough looking at Jenkins compared to Tekton, let's actually start digging into Tekton itself. If you do not understand the Tekton Pipeline example above, fear not, we will be doing a deep dive on that very shortly. For all examples (except the pseudo-code here and there) the manifests can all be found in the supporting repository for this post:

GitHub - KyWa/tekton-is-groovy

Contribute to KyWa/tekton-is-groovy development by creating an account on GitHub.

GitHubKyWa

A Brief overview of Tekton

If you are familiar with Ansible (this is not a CI/CD tool, although some try to use it for that purpose), there are some super easy comparisons to make so you can grasp the components of Tekton quite easily:

A lot of the components of Tekton can easily be translated to Ansible items (and almost easily to Jenkins items). For those who are unfamiliar with Ansible, then this is a great little listing of some of the components of Ansible. I do also have a primer written for Ansible in a previous post during the MineOps series that can be read if you are curious about Ansible.

I try to make the comparison to Ansible due to the nature of how Tekton works as its very "Ansible-esque". You put your tasks with their parameters and send them off to do whatever it is you want. Ansible has the ability to only run certain tasks "when" something occurs or if a certain variable is set, but they are run in order from top down through the playbook or "pipeline". In Tekton you also get a when field, which functions exactly like you would expect, but you also get a different field you can add, runAfter. This allows you to run certain tasks only after others have run. Most devs will probably build out their Pipelines in order top down, but it does give you the ability to only run certain tasks after others have run (such as a test cant be run until the build has been run).

Tekton Component: Task

Let's take a look at what a Tekton Task looks like so we can better understand what a Pipeline is and how it works.

---
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
  name: basic-task
spec:
  params:
    - name: URL
      type: string
      description: "What URL you wish to curl"
  steps:
  - name: curl-something
    image: someimage:tag
    command:
    - curl
    - $(params.URL)

In the above manifests, we have very few fields, but they are all very self explanatory. No need to explain anything outside of the spec field except that the apiVersion will most likely eventually update to no longer be a v1beta1 endpoint, so just keep that in mind.

A breakdown of what we have in the Task is fairly straightforward as this is a very basic Task. Let's cover the fields we have and what they do:

• params - This is where parameters for the Task can be specified as far as what it can make use of. You must specify at least the name and type fields, but you can also specify a description and default field. The default field is where you can specify a default value if none are provided when the Task is run (either from a PipelineRun or TaskRun, but more on this later)
• steps - Determines the steps the task needs to take to complete. A note here, a Task can have many steps and each step is a container in the Pod that is created for the Task
  • steps.name - This names the step in the Task. In the above example the container name would be step-curl-something
  • steps.image - Which container image to use for this Task (we will explore this a little bit more)
  • steps.command - Just like a container manifest you specify what command to execute when the container starts up, or in this case the step. You could also use script instead of command, but again, more on this later just know that neither is required as the image may have its own command you want to use

One of the unique things you may have noticed is in the command arguments a very "variable" looking line that shows $(params.URL). This is one of the components of Tekton that allows you to pass in parameters to the Task and not even just lines in the command field, you can also use this for the image line and others. Tekton has a webhook that validates all of the objects passed in prior to creating the Task. This gives you the ability to "templatize" your Kubernetes manifests (at least Tekton manifests) by having some fields be dynamic. One unique way to utilize this feature is by having the image line(s) of a Tasks step(s) have a tag that is specified by a parameter. An example of this, would be having a Task which does "something" for you, but could be re-used for a different purpose if a different image/tag was used (maybe testing different versions of NodeJS). You can keep the same Task with the same test logic, but you can have the image used be different when you run it depending on the NodeJS version you are looking to test.

There is no way we can go over all of the specifications for a Tekton Task in this post, but here are 2 resources that may help for a more detailed dive into the components of a Tekton Task:

• Tekton Task docs
• An exploded view of a Task with all options and comments

Tekton Component: TaskRun

A TaskRun is just what it sounds like, it runs a Task. Thankfully it doesn't get much more straight forward than this, so lets take a look at what a TaskRun looks like to run our simple curl task above:

---
apiVersion: tekton.dev/v1beta1
kind: TaskRun
metadata:
  name: basic-task-run
spec:
  params:
  - name: URL
    value: https://github.com
  taskRef:
    kind: Task
    name: basic-task

The TaskRun looks very similar to our Task (being how basic and minimalistic it is) and we can see it has a params spec just like a Task does. The params field(s) here are used to "pass" values to a Task that is being run through the TaskRun object. If a TaskRun is created without having all the params required by the Task that do not have default values set, the TaskRun will fail due to missing `params. It is important to know the Tasks you are attempting to run, although that should go without saying.

Since we mentioned having your Kubernetes manifest templatized, it would be important to point out you can use a podTemplate in a TaskRun (and PipelineRun) objects allowing you even finer control. This gives you the ability to call out specific things such as a dnsConfig or other Pod specific fields you cannot add to a Task. We wont really get into that, but its important

Tekton Component: Pipeline

Pipelines are what make Tekton fun to use as you don't have to declare how you want all of the Tasks to run and in what order. A Pipeline can be described by the following:

• A collection of Tasks to be run
• The order in which said Tasks run
• What parameters it provides to the Tasks when they are run

I said the word "run" quite a bit there, that is because everything in Tekton has to be "run" in some capacity (with the "technical" exception of Triggers).

A Pipeline looks very similar to a Task in its structure, but is typically "trimmed" down somewhat in comparison. Unlike Tasks however, a Pipeline doesn't necessarily run through its list of Tasks serially. Below we have an example Pipeline to run our basic-task to curl:

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: basic-pipeline
spec:
  params:
    - name: URL
      type: string
      description: Which URL to curl
  tasks:
    - name: get-url
      taskRef:
        name: basic-task
      params:
        - name: URL
          value: $(params.URL)

Now some of this may look like a "duplication" of what is in the Task and you'd be right to say that at first glance. A single Task and a Pipeline really don't mix as Pipelines are meant to be a compilation of Tasks, so at first glance you seem to have duplicates of parameters. In reality you are specifying what parameters a Pipeline accepts (similar to a Task), but these parameters can have their own names unique to the Pipeline. When you pass in a parameter to a Task in the tasks field of the Pipeline you will pass the parameter name the Task needs, but you can give it a predefined value, or as in the example above, give it a value equal to the Pipelines parameter. This is done through Tekton's variable substitution which looks like $(params.URL). We could change the Pipelines parameters to something completely different and as long as we tell the Task what its parameter is getting its value from, it doesn't matter what we call it. Although obviously it would be smart to keep the names somewhat relevant for the sanity of you and your team later down the road.

So let's look at the "final core" component of Tekton, the PipelineRun.

Tekton Component: PipelineRun

Much in the same as the relation of a Task and a TaskRun, so to is a Pipeline and a PipelineRun. There is one difference however in that a Pipeline doesn't get "run" in the same sense. A Pipeline is just a collection of Tasks so when a PipelineRun runs a Pipeline it is going to create the TaskRuns for those Tasks with only the Pipeline being a reference for the creation of these objects.

---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  name: basic-pipeline-run
spec:
  params:
  - name: URL
    value: https://github.com
  pipelineRef:
    name: basic-pipeline

The above is a PipelineRun for the Pipeline we are using for our example, basic-pipeline. What this manifest is going to do is generate TaskRuns from the tasks in the Pipeline after generating all the variables being set by the Pipeline and then passing them to the TaskRuns. The order of object creation from running a Pipeline looks like this:

Task <- Pipeline <- PipelineRun -> TaskRun

Remember that we said each TaskRun generates a Pod to do its "tasks" and each step is a Container in that Pod. This last bit is important when planning your Pipelines if you need to carry over "data" from one Task to another. This will be handled in a later section of this article for workspaces.

Installing Tekton

As with most "add-ons" to Kubernetes, Tekton can be installed very easily through a single command (assuming you have some level of cluster-admin privileges):

kubectl apply -f https://storage.googleapis.com/tekton-releases/pipeline/previous/v0.35.1/release.yaml

With that you will install all the components for Tekton to run in your cluster. After a few moments (or minutes depending on network speed), Tekton will be installed in your cluster. At the time of this writing, v0.35.1 is the latest release of Tekton so that is what we will be using. To look into other installation methods, the official Installation Documentation is your best place to look.

By default the above installs into the tekton-pipelines namespace, but you could modify the YAML manifest prior to applying it, but the default namespace should be fine for most teams so that is where we will work out of. In a vanilla install of Tekton, you really only get the controller and the validating webhook (which converts the Tekton manifests as stated previously). There are no Tasks, ClusterTasks or Pipelines for you to utilize out of the box. There are a few ways (outside of creating your own) to get some base Tasks in your cluster. One of those is by using the Tekton CLI, tkn. The CLI is not required for use with Tekton, but does make some things easier such as installing Tasks from Tekton Hub.

Tekton CLI Install

Installing the CLI is OS dependent and there are even other ways outside of the usual "install the binary" for tkn (the CLI for Tekton). You can technically make it a plugin for kubectl if you choose, but that is outside the scope of this article. For OS specific information, check the official Tekton CLI documentation. I've already got tkn installed (via brew) so let's see the above in action.

Installing Tasks from Tekton Hub

The commands run above were:

tkn hub install task git-clone -n tekton-is-groovy
kubectl get task -n tekton-is-groovy

And now we have the git-clone Task in our namespace so we can make use of it later (which we will). The -n is not necessary, but does make it easier to install Tasks to a specific namespace.

Installing Tekton Tasks without tkn

The above can also be done without the tkn CLI utility by running:

kubectl apply -f https://raw.githubusercontent.com/tektoncd/catalog/main/task/git-clone/0.6/git-clone.yaml

As stated before, tkn isn't required to do anything, but when getting started or just testing things quickly, it can make life a little easier. We won't be diving into tkn too much, but for some of our initial Task and Pipeline testing we will use it here and there. We will also be looking at YAML manifest equivalent's of the tkn commands just for comparison sake. Obviously if you are going to have CronJobs or some sort of "runner" handle your Tekton runs, then the CLI may make sense to make available to a runner. The choice is up to you and your team in how you want to make use of Tekton.

Real World Tekton Usage

What we've gone over so far has been pretty basic, so basic in fact, the name of our objects all start with basic (a terrible joke I know). We haven't even scratched the surface of what makes Tekton a great CI/CD tool. Before we dive into some of the cooler features, lets look at some Tasks and a Pipeline that do a little bit more than curl a URL. Let's setup a common pipeline to build and rollout an application.

Build and Rollout

For those who are unaware, you can build container images on Kubernetes. If you are familiar with Red Hat OpenShift (or its upstream project OKD), then you've most likely already been using the BuildConfig object's and this will be nothing new. What about those who are using a vanilla Kubernetes cluster and don't have the BuildConfig object in their cluster? Who would even need to build an image on Kubernetes when you can just run Docker locally? Well obviously, your organization isn't going to use your laptop for its pipeline's (hopefully that isn't the case), but also more importantly, there are organizations where you aren't allowed "normal" developer tooling on your machine for "security" reasons. Whatever the case, let's look at building a container image with Tekton.

NOTE: Tekton itself does not possess any magical ability to build Container images. We are just using Tekton to do things you could normally do in Kubernetes, but inside of a Tekton Task and Pipeline. Many have used "Docker in Docker" in their CI/CD pipelines over the years to accomplish this, but still is foreign to many people I've met and talked with over the years.

Building a Container Image in a Container

The above may sound silly and redundant, but its extremely powerful and important for modern CI/CD pipelines. Since everyone has an executive who say "We need Kubernetes now!", obviously you are dealing with containers whether you want to or not (I'm kidding of course). And because of that, you should try to use tooling that is native to whatever platform you are running on (which I hope is Kubernetes if you're reading this article). "Docker in Docker", or dind for short, is something that has been around for years at this point, but as with most things, there are different options and arguably better tools to use for some tasks. Enter Buildah.

Buildah is a project managed by the Containers organization and unlike docker (the application) it seeks to focus on one core task as opposed to "everything" involved with containers. Buildah has many features that expound upon building container images that are OCI compliant and will run in a "Docker" environment. We are not going to go into a deep dive of Buildah here, but we are going to use Buildah to build our container image. To build a container image with Buildah, the flags are very similar to how you would build an image with docker or podman:

# Docker
docker build -f Dockerfile -t quay.io/your-repo/some-image:tag

# Buildah
buildah build -f Dockerfile -t quay.io/your-repo/some-image:tag

Historically docker only looked for a file called Dockerfile when running a build, but this is no longer the case these days (thankfully). Tools like buildah and podman allow you to make use of any file as long as it is in the traditional Dockerfile format and since their inception. A Task to use buildah looks pretty straight forward and we will use the script field as opposed to command as it gives us slightly better control over the step in our Task:

  steps:
    - name: build
      image: quay.io/containers/buildah
      workingDir: $(workspaces.builder.path)
      securityContext:
        privileged: true
      script: |
        #!/usr/bin/env bash
        buildah --storage-driver=overlay build --no-cache -f $(params.DOCKERFILE_PATH) -t $(params.REGISTRY)/$(params.REPOSITORY)/$(params.IMAGE):$(params.IMAGE_TAG) .
      volumeMounts:
        - name: varlibcontainers
          mountPath: /var/lib/containers

Its pretty straight forward, but the core things here is that our image we are building, we are going to name and tag with params for the Task (not shown, but you can see their names so that is all that really matters). There is a follow up task, which runs buildah push, which will push the image to the container registry that is specified through the params. One thing you may notice is the field that is granting the privileged securityContext. This is required to use parts of the filesystem that containers need to "run". There are ways to do this in a "rootless" or privilege-less environment, but for the scope of this we are going to focus on using this method.

We have a method of building a container image, but where is our Dockerfile going to come from? Typically this is going to be in a Git repository somewhere (or some other source control mechanism) and so long as its publicly reachable that is is all we need to do. So we need to pull down our source code along with the Dockerfile so the application can be built. If we follow best practices, we would have one Task acquire our source code, another handle the build and finally a task to rollout our application for testing (which would be its own task, but outside the scope of what we are looking at here). Since each Task is a new Pod, how is the Task that gets our source code going to make it available to our buildah Task? The typical obvious answer is a volume created through a PVC, but Tekton has something "better" that can be used. It's time to look at workspaces.

Workspaces

Tekton allows you to make use of workspaces, which as stated above can function just like a volume would in your Kubernetes Pods. When using a workspace it functions quite similarly to a volumeMount in how its used inside of a Task step, but you can have it be a dynamic path without needing to "hard code" the path inside of the container. Let's take a look at how you would use a workspace and then we will look at declaring one:

apiVersion:
kind: Task
metadata:
  name: some-task
spec:
  steps:
    - name: persist-code
      image: quay.io/example/someimage:latest
      script: |
        cat $(workspaces.mydir.path)/somefile.txt
  workspaces:
    - name: mydir

The Task above will run a very basic step that runs cat on somefile.txt which is in the workspace named mydir. The path in which that lives inside of the actual running container will be (by default) /workspaces/mydir. Obviously at the end of the day everything boils down to a mount path in the container, the one big bonus of using workspaces over a normal volumeMount is that you do not have to worry about where things are mounted. You can just use $(workspaces.NAME.path) and it will use the actual path in its place during the steps of your Task.

Something to remember with workspaces is that just like volumes this doesn't have to be a PersistentVolumeClaim, it could be anything you would normally mount to a container such as a Secret, ConfigMap or even an emptyDir. For each Task there may be different reasons to use one over the other, but the main benefit is that you don't have to specify a path for your steps and can just rely on the variables as shown above.

For actually declaring workspaces and how they work is quite simple and does give you some flexibility (again its almost exactly like a volume in the configuration):

workspaces:
  - name: some-name
    description: What goes here or lives here
    mountPath: /some/alternative/mount
    optional: true
    readOnly: false

The only field above that is mandatory is the name field, all else is optional. Now what this workspace is depends on how its mapped from a TaskRun or a PipelineRun (which creates a TaskRun for each Task in a Pipeline). You have quite a few options (again just like a volume):

workspaces:
- name: pvc-workspace
  persistentVolumeClaim:
    claimName: my-pvc

- name: volumeclaim-workspace
  volumeClaimTemplate:
    spec:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi

- name: emptydir-workspace
  emptyDir: {}

- name: secret-workspace
  secret:
    secretName: my-secret

- name: configmap-workspace
  configMap:
    name: some-configmap

Above is pretty much the options you could/would use with a Tekton workspace and for those who have used Kubernetes for some time, looks identical to what you would/could do with a volume. I keep bringing this comparison of volumes and workspaces to show they roughly the same with the main benefit of being more declarative with workspaces for your "pipelines" as a whole.

After all of that explanation, here is a quick tl;dr on volumes vs workspaces:

With that covered, let's see using workspaces in action.

I've created a simple Pipeline, which we will extend later to our complete build and rollout Pipeline, but for now we can use it to test the workspace feature for getting our source code and then building an image with it.

---
apiVersion: tekton.dev/v1beta1
kind: Pipeline
metadata:
  name: build
spec:
  workspaces:
    - name: ssh-creds
    - name: build
    - name: registry-credentials
  tasks:
    - name: fetch-repository
      taskRef:
        name: git-clone
      workspaces:
        - name: ssh-directory
          workspace: ssh-creds
        - name: output
          workspace: build
    - name: build
      taskRef:
        name: buildah
      runAfter:
        - fetch-repository
      workspaces:
        - name: builder
          workspace: build
        - name: registry-secret
          workspace: registry-credentials

For ease of readability I took out the params to focus on the tasks and workspaces and how they all interact. The git-clone task (which we pulled in earlier through the tkn CLI) really only needs one workspace and that is output which is where the Git repository will be cloned into. If the repository is private however, you will need to provide credentials for the git-clone task to be able to pull from that repository. We will be using SSH based authentication, but this will be defined in the PipelineRun and only the ssh-directory workspace is needed for the task declaration in the Pipeline.

The buildah task we've created, which is based on this one from Tekton Hub, technically only needs a workspace that contains the code to build a container, but does require some form of credential for pushing the built image to a container registry. Let's see our PipelineRun to get some more clarity on how the workspace declaration works:

---
apiVersion: tekton.dev/v1beta1
kind: PipelineRun
metadata:
  generateName: build-pipeline-run-
spec:
  params:
    - name: SOURCE_URL
      value: git@github.com:KyWa/kywa-website.git
    - name: IMAGE
      value: "kywa-website"
    - name: IMAGE_REGISTRY
      value: docker.io
    - name: IMAGE_REPOSITORY
      value: mineops
    - name: DOCKERFILE
      value: "Dockerfile"
  pipelineRef:
    name: build
  workspaces:
    - name: ssh-creds
      secret:
        secretName: ssh-auth
        items:
          - key: ssh-privatekey
            path: id_rsa
    - name: build
      volumeClaimTemplate:
        spec:
          accessModes:
            - ReadWriteMany
          resources:
            requests:
              storage: 2Gi
    - name: registry-credentials
      secret:
        secretName: docker-hub
        items:
          - key: .dockerconfigjson
            path: auth

This Pipeline is being run with 3 total workspaces:

• ssh-creds - This is a Kubernetes Secret which contains SSH credentials
• build - A volumeClaimTemplate which will request a PVC be created and used for source code storage
• registry-credentials - Another Kubernetes Secret containing a .dockercfg for the image registry credentials

Let's give it a run and see what it looks like:

Now that we see how we can build a container in a container, the last "task" would be to see about using this image we built to update an existing Kubernetes Deployment with the image digest for it to rollout. This Pipeline can be quite powerful as being able to handle automatic rollouts for a Deployment is a great start to getting towards a fully automated CI/CD pipeline.

Planning our Build and Rollout

This Pipeline is only adding a single task to what we just did previously with our buildah Pipeline. The manifests used for the "build and rollout" can be found in the supporting repository for this post.

Running our Build and Rollout

With our Pipeline defined, we can in theory actually go to run it with a PipelineRun object:

This is a very basic CI/CD pipeline, but could be easily (and should be) extended with another task after the deploy to run some tests and once that is done "prod" could be rolled out with that image on a successful test. Tekton really can do whatever you want it to without much effort. It all comes down to "how do you want to do it?".

Dashboard (BETA)

If you're tired of the terminal and wan't to see a GUI element, I have just one thing to say to you, let's see what the Tekton Dashboard looks like!. Didn't think I was going there did you? Although the terminal is the greatest place on Earth (sorry Disney, also not sorry), there are times when you just want to get some visibility outside of kubectl.

The Tekton Dashboard has some pretty nice features and is great for visibility. The project can be found in its GitHub repository and is installed much in the same way Tekton itself is, the trusty kubectl one-liner:

Much like with other "web" services running in Kubernetes, you will need some form of Ingress to access it. That is outside the scope of this tutorial, but was covered in a previous post during the MineOps series. Already having an Ingress setup for this cluster, we can now view the Dashboard. Again at the time of this writing, the Tekton Dashboard is still in "BETA", but it is quite feature rich and can do pretty much all the things you'd expect it to do (view and run Pipelines!):

Conclusion

There are many other features of Tekton, but this guide was meant to be a light primer and get you aware of this powerful tool with its capabilities. One of the other powerful features of Tekton is Triggers (as one would expect from a CI/CD tool), but we will not be covering that in this post. I do recommend reviewing Tekton Triggers as they can really help flesh out an amazing CI/CD pipeline for your organization by enabling the running of Pipelines with events such as a Git commit being pushed into a branch among other things.

Just remember one thing, Tekton is groovy.

ArgoCD for mineOps

Since it seems we are now using Kubernetes and our CEO is happy and doesn't appear to be playing buzzword bingo anymore, its time to do something for us to make our lives easier. After spending some time scouring the internet, it looks like we may have found what we need, ArgoCD. It seems like we've heard it come up in our Twitter feeds, Reddit and wherever else dope technology is talked about. It seems like it handles GitOps, whatever that is, but most importantly it keeps manifests applied and correct in our Kubernetes cluster(s). But what does that really mean? Let's take a tour of what ArgoCD is and how it is going to help us manage all of these customer Minecraft Servers running on Kubernetes.

What are we trying to solve?

Since we now have our customer Minecraft Servers running on Kubernetes and we can create new ones with kubectl create, do we wan't to keep up with a kubecofig for each person on our development team, what do we do? We can keep doing this, but it gets tiring having to get more kubeconfigs for each new team member and really doesn't make auditing super easy (really borderline impossible in a vanilla Kubernetes cluster). There may be some other items, but here are our core focuses on what we are needing for our sanity:

• An easy way to deploy new customer Minecraft Servers
• Not require giving new team members a kubeconfig file
• Some form of auditing of who created/deployed what

How does ArgoCD help us?

Thankfully based on the items we have determined above, ArgoCD can handle most of these items (and more) without really having to change much of what we do. That is fine and dandy saying that, but what exactly does ArgoCD "do"? Argo itself is actually just the name of a group of projects (and a company I guess?) with ArgoCD being one of those projects. The CD stands for Continuous Delivery, which is something that up until now "we in this company" haven't really ever dealt with. In short it takes changes you've made (ideally in a Git repository somewhere) and then... continuously delivers/deploys them. On the surface it is actually quite simple, but the how and when is what is important. So that seems to lead mostly to an answer for the first item in our list, but does leave us still with the "how" of it. In Part 3 we moved all of our configuration files into Git and they are just sitting there doing nothing. What if they could "do" something? This is where ArgoCD comes in and the "how" of the CD portion of its namesake. ArgoCD can be utilized to implement a neat framework that some may consider a "buzzword" (we can use those too if we want Mr. CEO) called GitOps, but it is so much more than just a buzzword. It is honestly an amazing way of handling the automation of your CI/CD changes and really shines in a matured DevOps culture, which hopefully by now we can consider ourselves that.

For the issue of getting new team members access to Kubernetes to be able to run kubectl commands against, we don't even need to bother with that. ArgoCD itself (typically) lives in the Kubernetes cluster as its own application and is checking a Git repository (or even a Helm Repository for those who've gone a little further in their Kubernetes maturity) for any changes. Since every team member in theory who is going to be making changes, would use Git, nothing needs to be changed in terms of an access model to our Kubernetes clusters. ArgoCD only cares about what is in the cluster and what is in Git. You don't even really need to grant anyone access to ArgoCD itself unless there is some form of an issue.

With the previous question answered, we technically also have our answer as to what we need for some form of audit. Git handles auditing very, very well, so well in fact as stated before, you can literally use the word blame to find out who changed what in a file. Between this existing functionality of Git and using Pull Requests that require approvals/reviews, we can easily audit what changes actually go into our cluster. On top of those things, ArgoCD itself has a pretty amazing view of who did what right from the UI, which is really just reporting what Git says.

We've roughly got our what we believe we need to make this all work right out of the box with ArgoCD and Git. And we've got our "buzzword" GitOps, which is something we are really going to focus on. In a perfect world, no one would have direct access to our Kubernetes clusters and we would have everything audited through Git, applied via ArgoCD and happily running with 0 manual intervention. We can absolutely get here and there is nothing stopping us from doing that outside of a little growing pain.

The "Endgame" of GitOps

Without violating an Hollywood IP, the "End Game" of GitOps is quite simple, Git is the single source of truth. In theory its simple, in practice it can be challenging. Thankfully with ArgoCD it becomes quite doable. How this would look fully implmeneted is this:

• k8s Cluster is created
• ArgoCD is installed and configured
• kubeadmin config is stored in some secrets management tool that no one can access directly
• Any and all changes to the cluster are made through ArgoCD
• No one uses kubectl to get or do anything

For most of us who are admins, or who used to be admins, the idea of not having any visibility into our clusters or workloads sounds horribly scary. There are times where it can be for sure, but so long as there is a good Git review process and competent people are writing code changes, it doesn't have to be. With ArgoCD (as we will see) you have quite a bit of visibility into what is going on with your cluster(s) and if you have done some Day 2 activities such as having monitoring for your cluster, you don't really "need" kubectl. I will admit though not having direct access does make for a nerve racking scenario sometimes, but if done correctly, implementing GitOps will keep manual "tweaks" from being performed without anyone's knowing.

For more information on GitOps, check out this post from GitLab.

Installing ArgoCD

As discussed during the outline of ArgoCD I mentioned that ArgoCD typically lives in the Kubernetes cluster where it is deployed. There are other methods where you could technically have a "central" ArgoCD that manages multiple Kubernetes clusters. We will not be focusing on that method as it requires quite a bit more setup and there is nothing lacking if we have 1 ArgoCD instance per Kubernetes cluster (especially since we aren't growing past this during the mineOps series).

Raw Manifests

Ideally the best way to install ArgoCD is up to the team who is going to be doing the actual install. The method we are going to pursue is through just normal YAML manifests. I would personally use Helm to deploy this, but learning Helm isn't a necessary for the mineOps series, but a follow up post may be in the works. The fastest way to get the ArgoCD manifests is to obtain them from the Argo Project's GitHub repository for ArgoCD. In this repository there are many great things to look at, but for the most part we are primarly focused on one directory and that is the manifests directory which contains the most obvious of YAML files, install.yaml. We breifly mentioned this in Part 5, but kubectl can be used to target not just local files, but also URLs as well. It looks just like how we would normally apply a manifest, so lets get it going:

$ kubectl create namespace argocd
namespace/argocd created
$ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/master/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/applications.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/applicationsets.argoproj.io created
customresourcedefinition.apiextensions.k8s.io/appprojects.argoproj.io created
serviceaccount/argocd-application-controller created
serviceaccount/argocd-applicationset-controller created
serviceaccount/argocd-dex-server created
serviceaccount/argocd-notifications-controller created
serviceaccount/argocd-redis created
serviceaccount/argocd-server created
role.rbac.authorization.k8s.io/argocd-application-controller created
role.rbac.authorization.k8s.io/argocd-applicationset-controller created
role.rbac.authorization.k8s.io/argocd-dex-server created
role.rbac.authorization.k8s.io/argocd-notifications-controller created
role.rbac.authorization.k8s.io/argocd-server created
clusterrole.rbac.authorization.k8s.io/argocd-application-controller created
clusterrole.rbac.authorization.k8s.io/argocd-server created
rolebinding.rbac.authorization.k8s.io/argocd-application-controller created
rolebinding.rbac.authorization.k8s.io/argocd-applicationset-controller created
rolebinding.rbac.authorization.k8s.io/argocd-dex-server created
rolebinding.rbac.authorization.k8s.io/argocd-notifications-controller created
rolebinding.rbac.authorization.k8s.io/argocd-redis created
rolebinding.rbac.authorization.k8s.io/argocd-server created
clusterrolebinding.rbac.authorization.k8s.io/argocd-application-controller created
clusterrolebinding.rbac.authorization.k8s.io/argocd-server created
configmap/argocd-cm created
configmap/argocd-cmd-params-cm created
configmap/argocd-gpg-keys-cm created
configmap/argocd-notifications-cm created
configmap/argocd-rbac-cm created
configmap/argocd-ssh-known-hosts-cm created
configmap/argocd-tls-certs-cm created
secret/argocd-notifications-secret created
secret/argocd-secret created
service/argocd-applicationset-controller created
service/argocd-dex-server created
service/argocd-metrics created
service/argocd-notifications-controller-metrics created
service/argocd-redis created
service/argocd-repo-server created
service/argocd-server created
service/argocd-server-metrics created
deployment.apps/argocd-applicationset-controller created
deployment.apps/argocd-dex-server created
deployment.apps/argocd-notifications-controller created
deployment.apps/argocd-redis created
deployment.apps/argocd-repo-server created
deployment.apps/argocd-server created
statefulset.apps/argocd-application-controller created
networkpolicy.networking.k8s.io/argocd-application-controller-network-policy created
networkpolicy.networking.k8s.io/argocd-dex-server-network-policy created
networkpolicy.networking.k8s.io/argocd-redis-network-policy created
networkpolicy.networking.k8s.io/argocd-repo-server-network-policy created
networkpolicy.networking.k8s.io/argocd-server-network-policy created

Based on the output above, we have just created quite a few new objects in our cluster. For the most part this is just a bunch of RBAC rules to allow the various compontents of ArgoCD to do all of the things it does. The rest is really just the actual deployments of the ArgoCD components of which there are a few. We will break down each component just a little bit, but knowing each component isn't 100% necessary for using and working with ArgoCD so we will only go over the main ones that you'd realistically ever need to look at:

• ArgoCD Server - This handles the UI elements and is where the API server lives that hanldes the "running" of ArgoCD
• ArgoCD Application Controller - Continuously monitors running Applications and compares their state with what is cached in Git
• ArgoCD Repo Server - Manages the repositories and provides a local cache of each Git repository holding application manifests

Accessing ArgoCD

Now that we've covered the various components that we've installed, it is time to actually "see" and use ArgoCD, but with all things Kubernetes that have some form of GUI component, we will need some form of Ingress to access it. To access the web front end of ArgoCD, we will need an Ingress object to route our traffic to ArgoCD. Here is a simple manifest for such an Ingress object. Please note you do not actually need to use the web front end to make use of ArgoCD as it will work on its own, but if you wish to use the argocd client or view from the web front end to manage things, you will need an Ingress object (or a Gateway/VirtualService etc...). You could also technically use the kubectl port-forward function, but that isn't very realistic or scalable since we are using this in a cluster used by many (in theory).

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: argocd
  namespace: argocd
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
spec:
  rules:
  - host: "argo.kywa.io"
    http:
      paths:
        - path: /
          pathType: Prefix
          backend:
            service:
              name: argocd-server
              port:
                name: https
  tls:
  - hosts:
    - argo.kywa.io

NOTE Some of these annotations can be removed by changing the configuration of ArgoCD's settings in the ConfigMap argocd-cm, but are required for a "default" installation of ArgoCD along with ingress-nginx.

ArgoCD is deployed and seems to be up and running, we can get to the web application, but we don't have any login credentials. By default ArgoCD creates an initial admin password which can be found in a secret in the namespace that ArgoCD is installed in. To view the Secret and get the password to login initially here is a quick handy one liner:

$ kubectl get secret -n argocd argocd-initial-admin-secret -o jsonpath='{.data.password}' | base64 -d
Z1Ha6-z8QVR2jC0b

Later in this guide we will be doing things in a more declarative process and will not be needing to obtain this secret manually or even use it potentially for that matter. Now let's try to login to our newly installed ArgoCD instance.

Using ArgoCD

Now that we've logged into the UI, we are met with an amazingly detailed blank screen that tells us "No applications yet". So let's breakdown what an Application is real quick. An Application is a CustomResource provided by the ArgoCD controller which manages an "application". For some an "application" could be a Helm Chart, a Kustomize "application" or a directory of manifests. We will be looking at 2 of these style Applications with a primary focus on the directory style as it is probably the more useful for us deploying Minecraft Servers. Since ArgoCD is being used as a GitOps tool, we need to get ArgoCD to talk to Git somehow and from the Applications page we are on, there is no obvious path to this.

Creating an Application

With ArgoCD there are many ways to create an Application and we will outline the 3 most used (and currently the only 3 known) methods for creating an Application in ArgoCD.

ArgoCD UI

First up is the ArgoCD UI which we are already in, so this makes the most sense to do first. Let's see what we get if the click the "CREATE APPLICATION" button in the middle of our screen.

After passing in some information such as which Git repository to target and the name of our Application we get more changes than we were probably expecting. As we can see the UI has changed drastically from whwen we first logged into ArgoCD. There are now filters and statuses we can view, with labels and all sorts of things to click on. Since we have added an Application to our cluster and we can view what the Application is doing by clicking on it from the ArgoCD UI. This first Application is really nothing more than "applying" itself based on what we gave ArgoCD during its creation. Upon clicking the Application called "argocd-mineops", we are taken to a different view where we can see the name of our application pointing to another item named "argocd-mineops". This "other" object sharing the name is showing that the Application we created is applying manifests in the files/ArgoCD/basic directory.

Declarative

As with literally everything in the Kubernetes world, it can all be boiled down to a manifest and ArgoCD is no different. The same application we created from the UI can also be created through a Kubernetes manifest. Here is what it would look like and can be applied via the normal kubectl create -f method as with everything:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: argocd-mineops
  namespace: argocd
spec:
  destination:
    server: https://kubernetes.default.svc
  project: default
  source:
    directory:
      jsonnet: {}
      recurse: true
    path: files/ArgoCD/basic
    repoURL: https://github.com/KyWa/mineOps.git
    targetRevision: master

ArgoCD CLI

And for the "last" option to work with ArgoCD is the argocd CLI. This may be useful to some, but once you have ArgoCD setup and your Git repository the way you and your team want it, you may not have much reason (initially) to use the argocd CLI. It does have some neat features for more advanced usage, but for what we will be doing, it isn't needed. To get the argocd CLI installed check the official ArgoCD docs.

Let's at least see how to use it though and as with most tools you will need to authenticate so you can actually communicate:

To create the same argocd-minops Application as above, we would run the following:

As you can see, the argocd CLI is pretty straightforward and has very logically named flags for the various commands, but takes way more typing to get an Application created than having a manifest to apply or using the UI. The option is there though for those who wish to use it.

With an Application defined and the ability to check its current status we have done the core of what you will ever need to do with ArgoCD. However you may have noticed that our Application showed "Out of Sync" and that it had a Yellow color as opposed to Green. Most of us know that Green is good and Yellow is typically a warning of some kind. This is still the case with ArgoCD and we can dig into why that is in a later section.

Using the ArgoCD UI

Now that we've created an Application and have it in our ArgoCD instance, what do all the fancy UI elements mean? Thankfully they are quite self-explanitory, but we can look at them for a little bit. On the top row of the default layout we have a few buttons to look at:

• NEW APP - This takes you to the same screen we were on previously when adding our Application
• SYNC APPS - Initiates a manual sync on all Applications (or a selection)
• REFRESH APPS - Tells ArgoCD to check in with Git and refresh the state of Desired for the selection Applications. This is normally done on its own every 5 minutes by default

When using the UI, the latter 2 are probably the only buttons you will click from the main Application page. Things get a little deeper once you click on an Application and can see more choices.

• APP DETAILS - Takes you to the Application itself to view its status and information
• APP DIFF - Also takes you to the above page, but specifically to the DIFF tab
• SYNC - Same as the SYNC APPS button from the main Application page, but only for this Application. Can also choose which resources to sync
• SYNC STATUS - Looks at the sync status of all objects in the Application
• HISTORY AND ROLLBACK - Shows the history of the Application and each Git Commit that was targeted. Rollback is only available for non Automated sync policies
• DELETE - Should be self-explanitory, but deletes the Application (and its resources if prune is enabled)
• REFRESH - Also the same as the REFRESH APPS button from the main Application page

The UI does have some handy features, but as stated previously, once everything is "up and running" there may not really be a need to revisit the UI unless an issue occurs. And even still, the Pod logs for each component will tell you what the issue is, but may be easier to track down via the UI.

ArgoCD Sync Features

Previously we added an Application to ArgoCD that immediately showed "Out of Sync". That isn't because our application was extremely complex or that our co-worker was updating the same manifest behind our back, but it was because of something that ArgoCD does on its own we didn't account for. The way in which ArgoCD manages the state of Applications is by checking its Live state against the Desired state. The Desired state is what we have in our Git repository, but ArgoCD also does some magic on its side after it gets the manifests from Git. ArgoCD, when it creates/applies manifests, adds labels to identify that it is managing those objects and updates the Desired state to reflect that. When the Live and Desired states do not match, we are "Out of Sync", but since our Live manifest is the one out of sync, why doesn't ArgoCD do something about it?

Manual

For those who looked closely at what what our UI showed when creating our argocd-mineops Application, you will remember that the "Sync Policy" was set to manual and no options were checked just as in the screenshot below:

The Manual sync policy is there to be more of a "hey things are good, bad or mostly ok" and does not do anything outside of warn the user. In some cases it has merit, but really defeats the purpose of ArgoCD and GitOps in general in my own personal opinion.

Automated

An Automated sync policy is where ArgoCD really shines as it will enforce any deviation from the Defined application and the Live state of the Application in the cluster. Essentially how this works is if you had an object in your Kubernetes cluster, lets say a ConfigMap, defined in Git, if someone modified the ConfigMap via kubectl, ArgoCD would immedietly detect the change and apply the Desired manifest. This is extremely powerful and very handy, but there are some things to be aware of. If you put your Kubernetes cluster resources inside of Git (certificates, CRDs, etc..) ArgoCD will enforce them as they are, but there may be times where you run into an issue where Kubernetes updates a specific resource. It could be something as simple as a label or annotation, or it could be a block of configuration, but if ArgoCD detects a deviation from what is defined in Git it will continually re-apply the Desired manifest, sometimes leading to ArgoCD having issues or the cluster being put into a "unique" state. These "differences" were thought about when ArgoCD was designed and thankfully can be handled easily in the Application specification through an ignoreDifferences block. The official ArgoCD docs outline how to use it and when to use it, but the quick and dirty is that you can use it in the Application spec or at a global level (which can be handy depending on the type of object).

The 2 main options when selecting an Automated sync policy that we will care about for now are the 2 check boxes beneath AUTOMATED when choosing it from the drop down. These 2 options are:

• prune - Tells ArgoCD to delete objects no longer defined in Git (disabled by default)
• selfHeal - Enforces Desired to Live self healing (semi-disabled by default)

With an Automated sync policy, by default the manifests will not be force synced without selfHeal being set to true except in a few scenarios:

• An automated sync will only be performed if the Application is OutOfSync. Applications in a Synced or error state will not attempt automated sync.
• Automated sync will only attempt one synchronization per unique combination of commit SHA1 and application parameters. If the most recent successful sync in the history was already performed against the same commit-SHA and parameters, a second sync will not be attempted, unless selfHeal flag is set to true.
• If selfHeal flag is set to true then sync will be attempted again after self heal timeout (5 seconds by default) which is controlled by --self-heal-timeout-seconds flag of argocd-application-controller deployment.
• Automatic sync will not reattempt a sync if the previous sync attempt against the same commit-SHA and parameters had failed.
• Rollback cannot be performed against an application with automated sync enabled.

As you can see the Automated sync policy on its own will handle most of the common use-cases, but there is typically 0 reason to not set selfHeal to true as it really does enforce the configuration. The only time you may wish to consider not setting selfHeal to true is when initially testing new items and do not want to cause issues (as seen above with differences).

In YAML form an automated sync policy will look something like this:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true

There isn't much else to outline here on the Automated sync policy for our uses, but there are plenty more definitions for sync options that can be read up on here: https://argo-cd.readthedocs.io/en/stable/user-guide/sync-options/)

Repository Setup

So we've got a decent understanding of ArgoCD, how it works and what it targets to enforce a desired state. So let's look at what our setup looks like under the guise of our Minecraft Servers as we will have a few variations here very shortly. Our first configuration will be what we've already seen, a single Application that targets a directory in a repository and then recursively enforces the manifests in that directory. We will call this the "basic" setup of our Git repository. This is without a doubt the simpliest setup we can use, but does have some shortcomings. With our one Application we will get what we need with our customer Minecraft Servers being applied, but we have one Application that will grow in size and not be very "modular". Let's look at what this looks like in the UI using a single ArgoCD Application to manage all of our customer deployments.

As you can see by the very zoomed out screenshot, we have our one Application enforcing all 3 of the customer Minecraft Servers, but thats it, it is one single Application with one Status. This gives us much less insight into what is actually going on, what if a customer Minecraft Server gets changed by someone on our team, or what if there are issues with one of the manifests? It is very difficult (assuming at a much larger scale than 3) to track down which instance is having issues. This is where the concept of "App of Apps" comes in. You have an ArgoCD Application manage the state of other ArgoCD Applications. Let's see what this looks like from a realish scenario.

App of Apps

App of Apps does create more initial overhead and requires a little bit of design work, but the payoff is huge in the long run as it allows us to have an Application per customer (in our case). In a more real world Kubernetes scenario, the possibilities are endless and the benefits remain the same. Maybe you have one Application with its sole purpose to keep all cluster configuration Applications in place. Or you could have one Application manage an actual application such as Hashicorp Vault or a VS Code Server group for your developers.

For our test Git repository, here is what the layout will look like:

$ tree appofapps/
appofapps/
├── apps
│   ├── customer-0.yaml
│   ├── customer-1.yaml
│   └── customer-2.yaml
├── argocd-mineops.yaml
└── mineops-servers
    ├── customer-0
    │   └── mineops-0
    │       ├── mineops-configmap.yaml
    │       ├── mineops-deployment.yaml
    │       ├── mineops-namespace.yaml
    │       ├── mineops-pvc.yaml
    │       └── mineops-service.yaml
    ├── customer-1
    │   └── mineops-1
    │       ├── mineops-configmap.yaml
    │       ├── mineops-deployment.yaml
    │       ├── mineops-namespace.yaml
    │       ├── mineops-pvc.yaml
    │       └── mineops-service.yaml
    └── customer-2
        └── mineops-2
            ├── mineops-configmap.yaml
            ├── mineops-deployment.yaml
            ├── mineops-namespace.yaml
            ├── mineops-pvc.yaml
            └── mineops-service.yaml

Our "main" Application is the argocd-mineops.yaml manifest which watches the apps/ directory and enforces the manifests there. The apps/ directory itself contains its own Application for each customer which is watching the respective directory under mineops-servers and enforces those manifests. Below is what each Application is targeting so you can see the "tiered" approach:

# argocd-mineops Application
spec:
  source:
    path: files/ArgoCD/appofapps/apps

# customer-0 Customer Application
spec:
  source:
    path: files/ArgoCD/appofapps/mineops-servers/customer-0

From a UI perspective, lets take a look at how this plays out:

We ultimately still get the same result, but having each customer in its own Application it makes it easier to group and single out a specific configuration. There are some who say it would be fine from having a single Application manage an entire Git repository full of manifests, but this approach is cleaner and allows for more granular and potentially less destructive changes in the event of an accidental Git deletion.

Extras and Nice to Haves

We've gone over just about every core thing in ArgoCD that we will need to manage our customer Minecraft Servers and everything to get going with your own ArgoCD instance for managing Kubernetes clusters. There are a few things that we didn't look at during Part 5 of the MineOps series and that is Kubernetes Secrets because we didn't have a need for them. Well, we will be looking at them here as they are important in the grand scheme of Kubernetes management.

Secrets

A Kubernetes Secret is basically the same thing as a ConfigMap with one exception, they are base64 encoded. Spoiler time for those who don't know, but encoding != encryption. They do serve a purpose however and that is being a separate and unique object that can be managed via RBAC so other's cannot actuall view the data inside. Almost all Kubernetes native applications make use of Secrets. They can be blank objects that get updated by something in the cluster for dynamic secrets that are more secure as they are dynamically generated and updated not by a user. Well if we are going to add Secrets to Kubernetes and store them in Git, this presents a huge issue. Any base64 encoded value can be easily obtained by simply running the following:

$ echo "c2VjcmV0cGFzc3dvcmQ=" | base64 -d
secretpassword

With that in mind, how do we store a Secret in Git if it is easily retrieved? Well, you don't technically because of that very reason, however there are many amazing tools to make this not so much of an issue. Bitnami Sealed Secrets is a pretty awesome one as it allows you to encrypt values and store them in Git which can then be decrypted in cluster via kubeseal. Another great one is Hashicorp Vault which does things a little different and has way more overhead than the Bitnami option. We will not be playing around with either tool as we have no real need to store Secrets in Git, but it is important to know how you would handle these in your own real world scenario. ArgoCD can make use of both of the aforementioned tools and decrypt the Secret values and inject them into the cluster without anything being exposed in Git or otherwsise, so have fun and choose a tool that works for you and your team.

As for actually creating Secrets, there are a few ways. The easiest is through kubectl create secret as this allows you to "target" a file and create a Secret with its data being the value and the filename being the key. You can also create it declaratively through a manifest and this can be done in two different ways actually. The first of which is creating it with pre-encoded values and you specify it like so:

kind: Secret
data:
  type: Z2l0

Or you can have Kubernetes encode it for us by using the stringData type:

kind: Secret
stringData:
  type: git

$ oc get secret somesecret -o yaml
kind: Secret
data:
  type: Z2l0

There are a few other ways, but these are the more common ways of creating Secrets.

Private Repositories

What if our Git repository isn't public? How does ArgoCD get the credentials to communicate with a private repository or one behind an organization that isn't public? We know that ArgoCD manages objects through an Application, but nowhere during its creation did we get a place to put in credentials of any kind. This is where we get to explore the ArgoCD UI a little more and look at a way to do this declaratively through a Secret manifest. The reason its done through a Secret is that is how ArgoCD stores private repositories.

GUI

Since we are probably still in the UI somewhere, lets see where we go and how we add in a Private Repository, which are just listed as Repositories inside of ArgoCD.

Now that we've created a repository (please note I did not include a username/password as I'm too lazy to blur out the data) we can see how ArgoCD stored that in our cluster:

$ kubectl get secret repo-1593074683 -o yaml
apiVersion: v1
data:
  project: ZGVmYXVsdA==
  type: Z2l0
  url: aHR0cHM6Ly9naXRodWIuY29tL0t5V2EvbWluZU9wcy5naXQ=
kind: Secret
metadata:
  annotations:
    managed-by: argocd.argoproj.io
  creationTimestamp: "2022-03-19T01:11:24Z"
  labels:
    argocd.argoproj.io/secret-type: repository
  name: repo-1593074683
  namespace: argocd
  resourceVersion: "545945"
  uid: 73dc7ef2-fadf-4279-9e3c-ba69cea10445
type: Opaque

Knowing this is how ArgoCD stores it, that leads us into the other way to add a private Git repository to ArgoCD.

Declarative

Using the previous example as our "base" and using a different repository style (git vs http) we can create a new Secret which contains a repository configuration for ArgoCD to use.

apiVersion: v1
stringData:
  name: argo-repo
  sshPrivateKey: ssh-rsa AAAA0192k301j0192j3019j2301923-----
  type: git
  url: git@github.com:KyWa/mineOps.git
kind: Secret
metadata:
  name: mineops-repo
  namespace: argocd
  annotations:
    managed-by: argocd.argoproj.io
  labels:
    argocd.argoproj.io/secret-type: repository
type: Opaque

The data inside is obviously important, but what is key here is the label argocd.argoproj.io/secret-type: repository as this is picked up by ArgoCD and knows that this is a "repository" for it to use. The rest of the data is self-explanitory and using a Git repository and connecting over SSH. NOTE If you are using a private Git repository over SSH, your ArgoCD Applications will need to reflect this as well.

Let's create the repository from our Secret and see if ArgoCD picks up the Git repository:

And that is the other method for creating a private Git repository for ArgoCD. Now obviously for similar reasons as mentioned before, you wouldn't store this in Git itself because its completely readable by all (at least those who know how to pass a value to base64) so a different solution for storing this will be required for the sake of security.

Wrapping Up

Hopefully throughout this post you can see that ArgoCD isn't all that complicated and for most teams, this is the core use of it, but can easily be expounded upon as teams grow. There are other alternatives (or even pairings) such as Flux, but that is outside the scope of this series.

I hope you've enjoyed this blog series on MineOps and exploring the growth a company may see moving up the DevOps tool chain. There may be a follow up post to cover a few other items in small detail, but this final post wraps up the official series. It is possible that I may go over this in a livestream and go through the entire series, step by step for those who wish to take part and see all of this happen in real-time.

Series Links:

• Part 1 - Manual Minecraft Server Installation
• Part 2 - Automating with Ansible
• Part 3 - Keeping it Altogether with Git
• Part 4 - Containers Have Joined the Party
• Part 5 - Making Containers Highly Available
• Part 6 - ArgoCD to the Rescue

Before this post begins, I would like to apologize for the long delay in getting this part of the series done. There were a few challenges in the way I was attempting to write this and life got in the way as well. But we are done with the Kubernetes portion of the series and I hope you enjoy it.

Since Containers were such a hit (and presented us with many challenges), the CEO has unleashed the new buzzword he heard (and says we need for some reason), Kubernetes. In Part 4 we took a long dive into Containers and how to do just about everything with them. In this part of the series, we will see how to make better use of Containers and handle some of the shortcomings of single node Container Hosts.

What is Kubernetes?

Kubernetes is a collection of open-source software that handles the deployment and management of containers. This may or may not be a 1:1 copy of the info page of https://kubernetes.io, but it is honestly the best "simple" answer. Kubernetes is wildly complex beyond most peoples imaginations. For most people though, it doesn't have to be.

One of the many great things about Kubernetes is the scheduler, which handles the placement of applications on nodes in the Kubernetes cluster. How it does it is quite interesting and has some pretty deep engineering designs under the hood that I am quite literally not capable of explaining. So I'll just let Kelsey Hightower explain it in a simple elegant outline.

I've heard it said that Kubernetes is really just the "next Linux" and truth be told having been working with Kubernetes for quite some time now, I have to agree. Linux gave the data center its freedom to do many things and Kubernetes has now done this for the "Cloud" (and data centers). Many long time Linux admins and engineers will easily see the comparisons of Linux and Kubernetes

Do We Need It?

As we go to talk with our CEO about Kubernetes and if we really need to go down this route, we are told: "don't ask silly questions, just implement this buzzword so we can keep up with all the new tech". Since it seems we have no choice lets dig in and find out what Kubernetes is going to be able to do for us that Containers couldn't on their own and try to find the benefits.

With Containers we had some interesting challenges around where to run containers and having to deploy them individually, which we could do with Ansible just fine already. Containers didn't really improve anything for us outside of learning a new technology. And that is because deploying and managing containerized applications is not an easy task, but thankfully that is quite literally what Kubernetes does and does it well.

If you watched the video above where Kelsey Hightower explained Kubernetes to folks at PuppetConf, you should have seen the concept of how Kubernetes schedules containers and frees up resources for "completed" workloads. With running Minecraft Servers, we typically wont see the latter part of that, but the former is really what we are looking for.

One of the biggest problems with running Containers, is just that, running them. Not from the perspective of docker run mycontainer, but running multiple workloads and managing multiple servers that run Docker/Podman and managing the Containers they have. This is something we didn't really touch on in the last article, but should have been obvious in the way in which we had to deploy Containers with Ansible (please note it would have been the same even if we had done it manually). Kubernetes at the end of the day is a scheduler, and the issue we had was scheduling our customer Containers at any form of scale.

No time like the present, so lets dig in.

Basics of Kubernetes

Just like Linux there are many "distributions" of Kubernetes and even a Linux from Scratch style approach. So what is a distribution of Kubernetes then? Since Kubernetes is grouping of software, there are specific groupings that can be had. Inside these groups of software remain some common and core components which we can dig into below.

Despite the complexity of Kubernetes and all the software that it makes up, for those who deploy applications on it or manage it, we can describe it quite easily. Just like when describing the "cloud", Kubernetes can be broken down into the 3 core pillars:

• Compute
• Storage
• Networking

Now how each of these is done can vary greatly, but each of these has a common item that can be used for each of them. Let's go over these a little:

Compute

In Kubernetes the smallest compute resource you can create is a Pod, which is comprised by a minimum of one Container. They are called Pods and not Containers because a group of whales is called a "pod" and if you noticed the logo for Docker is a whale. So there is some fun/silly trivia since when Kubernetes was created, Docker was really the only Container game in town at the time.

You can create Pods on their own or use other types of objects in Kubernetes to deploy and manage pods. We will be going into great detail here later, but just know the most common is called a Deployment.

Storage

Storage is always a fun concept as this is where our data is stored. It is even more interesting in Kubernetes as there are mechanisms to automatically provision storage for you depending on where you have Kubernetes installed and what you have installed on top of it. In the big 3 cloud providers, you get automatic provisioning of storage utilizing each of the big 3's "cloud storage" options.

There are other ways to utilize storage inside of Kubernetes. You can connect to NFS servers, use paths from the host, or even mount "files" directly into the Containers! And yes for the people running Windows file servers (or Samba for some reason), you can even use SMB/CIFS if you wanted to, just know that the node running Kubernetes will need a driver installed for SMB/CIFS.

Networking

For most SysAdmins and "DevOps" people alike, networking is the great equalizer (or the great headache depending on how you feel about it) when it comes to technologies. Kubernetes has some very complex networking going on under the hood, but unless you are trying to get into engineering or becoming a maintainer, we don't need to dig that deep.

Cluster traffic is handled by objects called a Service. The core usage of a Service is "intra-cluster" communication, but depending on the environment can spin up Load Balancers external to the cluster allowing access to the compute resources behind the Service.

To get HTTP traffic into a cluster (so you can see your pretty web apps) you will need something called an IngressController. We will dive into Ingress Controllers later, but just know they are an additional item in Kubernetes that gets traffic into a cluster through an object called Ingress.

There is also another method that is can be used called Gateways. This one is extremely important to many because it goes a step further than HTTP traffic by allowing us to ingress TCP traffic easily. There are other methods to ingress TCP traffic, but they are clumsy and cumbersome in comparison to Gateways.

Sadly however, we cannot get the full k8s Ingress experience with Minecraft Server as it uses its own version of the TCP protocol so even a Gateway cannot help us here. Thankfully there are still ways to run and use Minecraft on Kubernetes. Remember those cumbersome methods I just mentioned? Yeah thats us now.

Not so basics of Kubernetes

For us, this section isn't entirely as important as understanding the basics of Kubernetes and how we can use it, but we will outline a lot of the components and how they really bring it all together.

The core of Kubernetes is the components that run in whats called the "Control Plane". This is aptly named as it controls all of the cluster and its components and sometimes items external to the cluster (depending what components are installed). A Control Plane node is a node in the cluster that runs one or more of the components that make up the control plane. Let's outline these components below:

Worker Node Components

• kubelet - The Kubernetes agent that runs on the node
• kube-proxy - Handles cluster network rules at the node level
• Container Runtime - We should remember this from Part 4 of this series

Control Plane Node Components

Control Plan nodes have all of the components of a Worker Node including:

• kube-apiserver - This is the front end of the Control Plane
• etcd - High performance key-value store that contains all cluster data
• kube-scheduler - Handles the scheduling of Pods on Nodes
• kube-controller-manager - Control loops that watch the state of the cluster

And here is a nice little visualization of the components from Kubernetes.

Building on top of Kubernetes

One of the great things about open-source software like Kubernetes is the ability to have customizations and distributions tuned to your liking. Out of the box, Kubernetes provides a pretty awesome platform to run containerized workloads, but does lack a few things that require other services outside of the cluster to fully function. Remember how we built out Minecraft Server image? Can't do that in a Kubernetes cluster out of the box and must use either your own Docker/Podman/Buildah instance to build these images. What about Security? Kubernetes has some basic security, but really doesn't provide any extensible security features outside of some very basic features.

OpenShift from Red Hat handles these shortcomings of a vanilla Kubernetes cluster by providing a much more complete Kubernetes package that you can deploy in all the major cloud providers or on-premise. If you are more of the DIY support group team, you can deploy OKD, the upstream project of OpenShift.

Here are just a few of the "add-ons" that OpenShift and OKD provide over a vanilla Kubernetes installation:

• Extensible security through SecurityContextConstraints utilizing SELinux
• A Container Build platform
• An internal Container Registry
• Cluster authentication built in (OIDC, LDAP and more)

Getting access to a Kubernetes Cluster

A few years ago, getting access to a Kubernetes cluster was pretty much left to the big 3 cloud providers, but now we have many other methods and the ability to easily spin up our own cluster locally on our machine. For serving customers, we obviously aren't going to focus on any of the options for running a local cluster, so let's keep our focus to cloud providers. And now for the big players in the space:

Vanilla Kubernetes Clusters

• GKE - Google Kubernetes Engine
• EKS - Elastic Kubernetes Service
• AKS - Azure Kubernetes Service

OpenShift Platforms

• Amazon
• Azure
• IBM Cloud
• OpenShift Dedicated
• Self-Hosted

Self-Hosted, Smaller Cloud, Development

• DigitalOcean
• CRC
• Minikube
• Docker-Desktop

And now for the "Linux from Scratch" style Kubernetes cluster if you are so bold, https://github.com/kelseyhightower/kubernetes-the-hard-way. This repo is a project known as possibly the best deep dive DIY guide to Kubernetes to date. It is not for the faint of heart, but is truly an amazing guide to really building your own Kubernetes cluster.

There is even an upstream software called KubeVirt which lets you create Virtual Machines and treat them as Kubernetes Objects. Red Hat has a product that handles this through OpenShift and you can read more about that here. We will for sure not be diving into this during this series, but is something to keep in mind as we go throughout our DevOps journey.

Using Kubernetes

When it comes to actually using Kubernetes, all you really need is the ability to authenticate with your cluster, usually through a kubeconfig and a client to connect with, typically kubectl.

Note: The next few examples outline minikube as the nodes and not a cloud provider. All commands and demonstrations will be the same between Kubernetes clusters unless otherwise noted.

Kubeconfig

Let's take a look at the kubeconfig our Cloud Provider gave us when the cluster was provisioned. This file should be placed in ~/.kube/config on your machine as kubectl by default will pick it up natively, or can be set through ENV vars (KUBECONFIG) or with the --kubeconfig flag for kubectl. I'm a big fan of not having to modify anything I don't have to, so mine goes into ~/.kube/config. Let's look at what it we have:

Depending on the Cloud Provider and how they grant access, the kubeconfig may look slightly different, but the core of each is the same. The breakdown of the sections is fairly straightforward:

apiVersion: v1
clusters:
- cluster:
    certificate-authority: /minikube/ca.crt
    server: https://172.16.171.129:8443
  name: minikube
contexts:
- context:
    cluster: minikube
    namespace: default
    user: minikube
  name: minikube
current-context: minikube
kind: Config
preferences: {}
users:
- name: minikube
  user:
    client-certificate: /minikube/client.crt
    client-key: /minikube/client.key

• clusters - Lists the clusters and their information. certificate-authority, server and name are what commonly fit into these fields
• users - Outlines username a token or certificate belongs to
• contexts - This groups Kubernetes access data in an easy name. It combines data from the clusters and users sections to determine your "context" and is used to set the current-context which is where kubectl will talk with

Getting kubectl

If your cluster was deployed in the cloud, chances are you will have a kubeconfig file prior to getting the kubectl binary, so let's go ahead and get the kubectl binary so we can actually communicate with Kubernetes.

For Mac users, you can install through a few different methods:

### Via Homebrew
$ brew install kubernetes-cli

### Direct from Kubernetes.io
$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/darwin/amd64/kubectl"

Linux users can obtain it through curl as well and is the easiest method:

$ curl -LO "https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl"

And for other platforms and methods, you can always view the kubectl installation documentation at https://kubernetes.io/docs/tasks/tools/#kubectl.

First Commands with kubectl

So we have a kubeconfig file and kubectl installed, now its time to try to see what happens if we issue a command to Kubernetes.

$ kubectl get nodes
NAME           STATUS   ROLES                  AGE   VERSION
minikube       Ready    control-plane,master   11m   v1.23.1
minikube-m02   Ready    worker                 11m   v1.23.1
minikube-m03   Ready    worker                 10m   v1.23.1

Well it looks like we've connected and got some data back, but lets break down the command we ran to get an idea of what is going on.

• get - this is one of the many commands or "verbs" for kubectl
• nodes - the target object of the get command

The output we received from Kubernetes tells us that it has 3 nodes, they are all "Ready" (the kubelet on each node is up and connected with the Kubernetes API), what their roles are (not entirely important for most users), how old they are and what version of Kubernetes they are running.

Some of the most common verbs you will use with kubectl are:

• get - Displays a resource (has many flag options)
• describe - Shows specific details about a resource(s)
• create - Creates a resource (or from a file with -f)
• apply - Applies a manifest
• delete - Deletes a resource (or from a file with -f)

With some of the common verbs out of the way, lets see what all else we can find in the cluster.

$ kubectl get namespaces
NAME              STATUS   AGE
default           Active   25h
kube-node-lease   Active   25h
kube-public       Active   25h
kube-system       Active   25h

$ kubectl get all -n kube-system
NAME                                   READY   STATUS    RESTARTS   AGE
pod/coredns-64897985d-p7j5s            1/1     Running   0          25h
pod/etcd-minikube                      1/1     Running   0          25h
pod/kindnet-47kjw                      1/1     Running   0          25h
pod/kindnet-pknbj                      1/1     Running   0          25h
pod/kindnet-rdq4h                      1/1     Running   0          25h
pod/kube-apiserver-minikube            1/1     Running   0          25h
pod/kube-controller-manager-minikube   1/1     Running   0          25h
pod/kube-proxy-4pgbq                   1/1     Running   0          25h
pod/kube-proxy-n4ksg                   1/1     Running   0          25h
pod/kube-proxy-smjzx                   1/1     Running   0          25h
pod/kube-scheduler-minikube            1/1     Running   0          25h
pod/storage-provisioner                1/1     Running   0          25h

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   25h

NAME                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR            AGE
daemonset.apps/kindnet      3         3         3       3            3           <none>                   25h
daemonset.apps/kube-proxy   3         3         3       3            3           kubernetes.io/os=linux   25h

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   1/1     1            1           25h

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-64897985d   1         1         1       25h

The first command above looked for something called a namespace which is a Kubernetes object that groups resources. In a normal Kubernetes cluster a namespace could contain a single application, a teams applications, a customer's applications or any number of combinations and doesn't really matter. The only thing to note about namespaces is that the objects in a namespace are "sectioned" off in that namespace and cannot be accessed by other objects in other namespaces except where explicitly granted via RBAC or some other mechanism (more on this later).

For the second command kubectl get all -n kube-system, the -n flag specifies to run the command against the namespace, kube-system (we can see that is a valid namespace from the first command). With the get command, you can specify what you want specifically (such as with get namespaces) or you can specify all which returns "most" objects in a namespace. When I say most, I do not mean that it will only return some of the objects, what I mean is that Kubernetes and kubectl have a specific set of objects in the namespace it displays. There are other objects that can be "within" a namespace that can be viewed by calling them out specifically via the get command. The focus of get all revolves around mostly the pods, what manages those pods (such as Deployments, ReplicaSets and DaemonSets) and Services.

If you are ever curious about all the namespace level objects and want to actually get all of them, here is a handy kubectl command you can run, but be warned, this really does get every single object that exists in a namespace.

# Put this in your .bashrc or .bash_profile
# Function is run like this: kubegetall NAMESPACENAME

kubegetall () {
for i in $(kubectl api-resources --verbs=list --namespaced -o name | grep -v "events" | sort | uniq); do
  echo "Resource:" $i
  kubectl -n ${1} get --ignore-not-found ${i}
  echo ""
done
}

For an example of the output of the above command, check out this GitHub Gist. And note the output in that Gist is on a minikube cluster with practically no additional Kubernetes addons (again, more on this later). Hardly any of those outputs in a namespace are needed for normal cluster/namespace operation, but do come into play occasionally. You will most typically get the specific object you need as opposed to getting everything.

The commands above, excluding the kubegetall function, are the most common that you will use, but we need to start diving a little deeper into Kubernetes objects and some more verbose commands.

Kubernetes Objects

All Kubernetes objects can be viewed much like our kubeconfig was and in different ways. Let's take a look at one object, the kubernetes Service in the default namespace.

NOTE: Just a quick discussion, default is just what it sounds like. If there is no context set, kubectl will always target the default namespace. It is not best practice to put your applications here, but there are many guides out in the Internet that "teach" you to just put your applications here. Please do not do this and utilize namespaces as they are there to save you. If you feel like blowing away your namespace with a kubectl delete all, well that will also delete the kubernetes Service which could have less than desirable side effects.

$ kubectl get svc kubernetes -o yaml
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: "2022-01-26T03:48:38Z"
  labels:
    component: apiserver
    provider: kubernetes
  name: kubernetes
  namespace: default
  resourceVersion: "207"
  uid: 91e5f024-5267-4210-a595-3f6368db0c0b
spec:
  clusterIP: 10.96.0.1
  clusterIPs:
  - 10.96.0.1
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: https
    port: 443
    protocol: TCP
    targetPort: 8443
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

The command above was a normal kubectl get command, but we've added a new flag, -o. More info on the -o flag will be next. All of the information above is broken down into a few "core" fields that are common with almost every single Kubernetes object. Let's break down the "core" fields here to get a better understanding:

• apiVersion - Specifies which apiVersion the object is requesting/using from the Kubernetes Controller(s)
• kind - This identifies what an object is and also identifies what the get command would look for
• metadata - There are many sub fields in here, but the most common to be aware of are:
  • name - the actual name of the object
  • namespace - where the object "lives" (cluster level objects do not have a namespace attribute)
  • labels - key/value labels for adding attributes to an object
  • annotations - similar to labels, but are typically used by Operators and additional Kubernetes Controllers that "watch" for objects with specific annotations
• spec - The spec are is where the real identify of the object comes from. This section is where most Kubernetes objects differ from each other as each object has its own unique specification. Just know that the fields contained within the spec section are what makes up the configuration of an object
• status - Status is not something you can "apply" to an object as this field is managed by Kubernetes itself and will be updated live with various attributes depending on the Kubernetes Object. Examples would include a Deployment status of availableReplicas, conditions which shows the last time an object was modified/updated.

With that massive wall of text out of the way, the rest of the Kubernetes objects you deal with will ultimately be the same except for the spec field. Knowing these "core" fields will make looking at and creating Kubernetes manifests much easier. If you ever find yourself unsure of an object or the fields it can contain, kubectl has an amazing built-in function called explain which does just as it sounds. I will not put an example in the blog here, but I will post a GitHub Gist you can review. Just know that each object follows the format of a json style path: object.spec.configitem1.configitem2

Now for some more information regarding the -o flag from above. The -o flag stands for output and here are some of the common options you will most likely use:

• -o yaml - Prints out the YAML of a Kubernetes object(s)
• -o json - Same as above, but in JSON format (best when piped to jq imo)
• -o name - Prints the object(s) without any other data except type/name such as pod/etcd-minikube
• -o wide - Prints out similar to with no -o flag but provides more details in line. For pods this includes the node they are running on
• -o jsonpath - A little more advanced, but can be used to get specific fields from an objects manifest. Example: kubectl get pod etcd-minikube -o jsonpath='{.spec.containers[].image}' will return the image: line of the manifest. This can be done with get -o yaml | grep, but is much cleaner

So we've looked at kubectl get and added the -o flag, but there is another command that gives more or less the same data, but in a different format and that is kubectl describe. This command gives all the usual manifest data from kubectl get -o yaml, but formats it differently and provides us with a new item to look at called events:

$ kubectl describe svc kubernetes
Name:              kubernetes
Namespace:         default
Labels:            component=apiserver
                   provider=kubernetes
Annotations:       <none>
Selector:          <none>
Type:              ClusterIP
IP Family Policy:  SingleStack
IP Families:       IPv4
IP:                10.96.0.1
IPs:               10.96.0.1
Port:              https  443/TCP
TargetPort:        8443/TCP
Endpoints:         172.16.171.129:8443
Session Affinity:  None
Events:            <none>

As you can see if you were to compare the above output to the kubectl get svc kubernetes -o yaml you would see we haven't really "gained" any data except a new field called Events which shows the events an object has. For a Service, there probably won't ever be many (except in possible some bad scenarios), but for a Pod, this will show the Pod being scheduled, NIC assignment, pulling the image for the containers to run and starting each of the containers. The events are very useful for troubleshooting, but you do not need to see them via kubectl describe as they are an object you can also just kubectl get events. One other item to note on describe vs get is the labels section. If you compare the two outputs, you will notice that describe gives us labels with a key=value as opposed to a key:value.

Which brings me to the next example regarding Kubernetes Objects and that are selecting objects based on their labels. This next example is really more beneficial in namespaces with many applications living in it, but the core of what it does can be seen here. Since we saw that some objects have labels, lets use a label (that we would have either known about or obtained via describe or get) and use it as a selector via another flag -l:

$ kubectl get all -n kube-system -l k8s-app=kube-dns
NAME                          READY   STATUS    RESTARTS   AGE
pod/coredns-64897985d-p7j5s   1/1     Running   0          3d10h

NAME               TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns   ClusterIP   10.96.0.10   <none>        53/UDP,53/TCP,9153/TCP   3d10h

NAME                      READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns   1/1     1            1           3d10h

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-64897985d   1         1         1       3d10h

With -l being a selector, we chose k8s-app as our label with the value of kube-dns. This shows all objects that share this label and help us identify all the components that make up the kube-dns application. This will be extremely handy to remember when working with Kubernetes and multiple applications as it allows you to grab just the objects that share the label. This can also be run with the get all -A instead of the -n namespace-name flag and will append the namespace of each object before the object:

$ kubectl get all -A -l k8s-app=kube-dns
NAMESPACE     NAME                          
kube-system   pod/coredns-64897985d-p7j5s 

NAMESPACE     NAME              
kube-system   service/kube-dns 

NAMESPACE     NAME                     
kube-system   deployment.apps/coredns 

NAMESPACE     NAME
kube-system   replicaset.apps/coredns-64897985d

NOTE: On the above command I removed all data after the NAME field for readability, but it prints out the same as the prior command. You can even chain other flags onto the selector such as -o wide or -o yaml, but note the -o yaml will print all of those objects into a List which is typically unpleasant to scroll through, so I would recommend piping that to a file to review.

Where do objects live?

So we have looked at Kubernetes Objects and how to get and describe them, but we haven't really discussed "where" they live. Now so far every single object has been a namespace level object, but there are objects in a Kubernetes cluster that are not at a namespace level and live at the cluster level. These objects have a flag in their definitions which state namespaced: false. For a list of Kubernetes objects in your cluster which can be created/applied you can run the kubectl api-resources --verbs=list command to identify them. You can also separate them into namespaced=true or namespaced=false to show which objects are for which area of a cluster.

As you can see there are quite a few objects that can be created in a namespace, but hardly any of these will be used by most people directly, but each of them have their place and provide great value as you grow in your Kubernetes knowledge and administrative capabilities. Thankfully this output shows you the apiVersion and kind of the objects to help you identify and create your own if you are having issues.

Cluster Level Objects

Now lets dig into the non-namespaced level objects commonly known as cluster-level objects. Depending on the cluster there may be less or more of these, but the common ones are here:

As you can see in the output above its no different than the namespace level objects with the exception of the NAMESPACED field being false. Whichever cluster you are working with, you can use the kubectl api-resources command to identify which objects the cluster can have. This really comes handy in debugging as you get into more advanced Kubernetes objects, which in fairness most of the complexity comes from the Networking components as they can vary greatly depending on what has been installed on the cluster.

The more common cluster level objects that you really need to know about are, nodes, clusterroles, clusterrolebindings, and potentially ingressclasses. Nodes are important just to know the state of the cluster, clusterroles and clusterrolebindings are for managing RBAC in the cluster.

There really isn't much more to discuss about objects in the context of describing them. It is now time to start looking at some of the objects we will need to use and be aware of for deploying applications.

Deploying an Application

So we know about Kubernetes objects and how to look at their specifications to determine what they are and do, but we haven't seen anything outside of what Kubernetes itself needs. Just like in the Docker part of this series, we did a basic "hello-world" which actually provided some value as it outlined quite a few unique attributes of containers. There is not an equivalent in Kubernetes, but thankfully we don't really need one. As mentioned before, Kubernetes can easily be broken down into the 3 main components of a "cloud" (Compute, Storage, Networking), so let's look at manifests that generate those components. All manifests can be found here in the support GitHub repo for MineOps.

Instead of diving straight into a Minecraft Server deployment, we will look at a VS Code Server to get a feel for more Kubernetes components. Typically you would name your manifest files after what they are, but for the sake of comparing the 3 main components, I've named them after what they "do".

Compute Manifest

Inside of the compute.yaml file we have a single object which is our Deployment. A Deployment is responsible for, well, deploying a Pod which can have any number of Containers in it. This particular example only has one Container which uses the image for a VS Code server that can run in a browser. This image is hosted in Quay.io and when the Pod goes to spin up, the kubelet on the node where the Pod has been scheduled will basically run a docker pull to get the image so it can then start the Container. A few items of note in this manifest, are the volumeMounts, volumes, and env sections.

• env - This provides environment variables to the Container and in our case passes in a PASSWORD to login to the VS Code Server
• volumes - Volumes outline which objects are required to "mount" to a Pod. This can be a persistentVolumeClaim (which we will be using and discussing), ConfigMap, NFS share, iSCSI volumes, Secrets and quite literally anything. Note that volumes are declared at the Pod level and not the Containers
• volumeMounts - Identifies where a volume should be mounted inside a container. Depending on the type of volume used, if you were to run df inside the Container, you may see it as a literal mount like any other mount would be

Storage Manifest

Our storage manifest is quite simple as you can see in storage.yaml. There is a single object called a PersistentVolumeClaim which will, based on its spec, request 5Gi of storage. Now the "how" it does this is unique for each platform, but a PersistentVolumeClaim will request a PersistentVolume from the cluster of the same size as the requested Claim and if one does not exist will look to the StorageClass to create one. A StorageClass is a provisioner that resides in a cluster to reach out to a storage platform and create a share/volume/disk which is then present in the cluster as a PersistentVolume. A cluster can have more than one StorageClass and one of these can be set to the default which will be where all PersistentVolumes are created from, unless in the PersistentVolumeClaim you specify the storageClass to use. Now if there is no StorageClass in a cluster, you will have a PersistentVolumeClaim perpetually stuck in a Pending state.

Most if not all clusters deployed in the cloud will always have a StorageClass, but there will be specific use-cases where you may want a different type of underlying storage. Sometimes you may be running a database which may not like being on Amazon S3 File storage and would be better suited performance wise with block storage. These choices will change depending on the application and the team running it and whatever architectural and platform decisions are made.

There are some storage platforms that do some "interesting" things as it relates to mounting the share/volume/disk to the running Pod. Some workarounds may have to be made to your application, or your deployment (which we will go into later) to have the storage work as expected. This is not the case with all storage platforms, but some do have permissions that make life interesting.

Network Manifest

The networking.yaml file has 2 objects inside of it, a Service and an Ingress. This manifest is broken up by the YAML syntax of --- which creates a "break" and tells the renderer (in our case kubectl) where the next object begins.

This is one of a few ways to use a single file to house multiple objects, the other being an actual List. Here is a quick snippet of a List:

apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Service
  ~~~~~
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  ~~~~~

There are times depending on the client using the manifest where a List may be helpful, but for now its just another little bit of information and using the --- to break up objects works fine.

We explained earlier that a Service is for the most part meant for "in cluster" communication. A Kubernetes Service can actually have many different forms and how the work depends on the underlying platform they run on (see why some people say Kubernetes is hard/confusing?). The Service can have a few different types each doing something different with the most commonly used (and "default" type) being the ClusterIP. A Service with a ClusterIP basically gets a non-routable IP address (well, its routable, just only in the cluster) and is used as a front end for other services/applications in the cluster to be able to resolve the Service. Each Service, regardless of type, gets a DNS name in the cluster. For our example Service in the networking.yaml file the cluster DNS name would be: code-server.code-server.svc.cluster.local or even code-server.code-server. This comes in quite handy when building out multiple applications as you don't really have to worry about DNS, just your Service names and namespaces.

Even though we didn't cover this in the Containers part of this series, you will eventually run across someone using docker-compose or just docker and something called link and/or bridges. What that accomplishes is allowing Containers to "know" about each other and be able to communicate. A Kubernetes Service, which gets a DNS entry the entire cluster can use by default, your applications can find other applications via their Service name without any real changes or worrying about hostnames and DNS. Since networking is a pretty huge part of most applications, lets dig into this concept just a little further, even though our Minecraft Server wont really be utilizing this, it is extremely important to understand for working with Kubernetes.

Let's take a basic application as an example such as Wordpress. Wordpress has a web front end in PHP and needs a database as its backend (typically MySQL). In a normal installation on a VM or something similar, you have to give Wordpress the DNS name or IP of the database for it to be able to connect to. Depending on how and where you deploy Wordpress this could change drastically and be something you have to "hunt down" every single time you deploy a new Wordpress instance. In the Kubernetes world, your database will almost always have a Service regardless of any other factors making it simple and easy to deploy multiple Wordpress instances across a Kubernetes cluster.

# Wordpress Deployment snippet
spec:
  env:
  - name: MYSQL_SERVER
    value: mysql

Now for the important and "complicated" topic of Ingress. Ingress is an object created in a Kubernetes cluster that allows traffic into the cluster from outside of the cluster. Services do not do this, but technically can enable external traffic into the cluster, but in a very different way. Services that are of the type: LoadBalancer and on a supported platform, creates a Load Balancer on the platform with a Public IP Address that then connects it to the Service. This can work for some use-cases, but is very much a one to one of IP<->Service. Using Ingress allows for far fewer external IPs (in theory only 1 for most) and allows you to use DNS names to route traffic into the cluster. The IngressController will then pass the traffic to the Services based on the host defined in the rules section.

Quick example to highlight this and how it works:

You have an IngressController with an external Load Balancer with an IP of 20.192.1.254, you create an external DNS record of *.kube.example.com which resolves to the IP of 20.192.1.254. Inside of your cluster you can create an Ingress object that has a configuration like so:

~~~
spec:
  rules:
  - host: "code.kube.example.com"
~~~

The above snippet should more or less outline what this is going to do, but basically the Ingress object allows the IngressController to do this:

• Listen for traffic coming in over the DNS/IP of 20.192.1.254
• Check rules for existing Ingress objects
• Route external traffic into the cluster to the Service with the matching rules for the host rule in the Ingress object
• Service then passes traffic to the pod(s) matching its labels

That was a rough outline of what Ingress is and does, but there can be multiple IngressControllers that allow for different features and your Ingress objects can specify which one to use via an annotation. For us we will not be using Ingress, but there are other ways to do this which we do not need to go into here, but it is important to know the common ways to get traffic into your cluster.

Applying the Manifests

So now that we did a semi-deep dive on the manifests and what they are going to do, lets go ahead and try to create them and see where we can get. There are a few ways to do this, but the easiest way to apply multiple manifests is to have them in a directory and target that directory with kubectl using the -f flag which specifies a "filename". This filename can be a file, directory or even a URL (which is handy if you have a manifest on the Internet somewhere and wish to use it without having it locally). You can apply multiple manifests, but only through targeting a directory (sadly you cannot apply multiple individual files).

So we applied our manifests and now have a working application in our Kubernetes cluster as seen by going to code.kube.example.com. Nothing is really too special about this except that we have a running application. Its just VS Code  in a browser (a really cool project) running in a container. Now if we look at a few things we can see what all we created in these manifests:

$ kubectl get all
NAME                               READY   STATUS    RESTARTS   AGE
pod/code-server-7c444dc854-9bfcw   1/1     Running   0          12h

NAME                  TYPE        CLUSTER-IP       EXTERNAL-IP   PORT(S)    AGE
service/code-server   ClusterIP   10.102.116.112   <none>        8080/TCP   12h
service/kubernetes    ClusterIP   10.96.0.1        <none>        443/TCP    41h

NAME                          READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/code-server   1/1     1            1           12h

NAME                                     DESIRED   CURRENT   READY   AGE
replicaset.apps/code-server-7c444dc854   1         1         1       12h


$ kubectl get ingress
NAME          CLASS    HOSTS                   ADDRESS          PORTS     AGE
code-server   <none>   code.kube.example.com   172.16.171.146   80, 443   12h

$ kubectl get pvc
NAME          STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
code-server   Bound    pvc-e443ccf4-6bcf-47cd-a998-88d007eed995   5Gi        RWO            standard       12h

In the compute.yaml manifest we had a Deployment, but we now also see a replicaset and a pod. Without going into a super deep dive, a Deployment is more of a "specification" of how pod(s) should be created and managed. A Deployment will generate a ReplicaSet with the specification and it's the ReplicaSets job to manage the state of the Pod. If the pod is deleted, the ReplicaSet will request a new one be created. Nearly all "compute" resources in Kubernetes follow this model to a degree. And to add fun to that, the pod we ultimately got has 1 Container in it. A pod can have multiple containers as we mentioned earlier in this blog.

NAME                               READY   STATUS    RESTARTS   AGE
pod/code-server-7c444dc854-9bfcw   1/1     Running   0          12h

The rest of the objects are all there and we can see details about them (which should match what we have in the manifests we applied). For the most of our use-case, this is just about all we really need to know about Kubernetes manifests and applying them as everything else you will ever deploy in a Kubernetes cluster follows basically the same logic.

Minecraft on Kubernetes

Now that we've gotten a somewhat decent outline of what Kubernetes is and what it looks like to deploy an application, its now time to actually deploy Minecraft Server on Kubernetes.

Building our Manifests

As with our example deployment of the VS Code Server above, we will need manifests to create and deploy our Minecraft Server, but instead of creating these objects in the default namespace, we will create them in a namespace dedicated to our application. We can have many Minecraft Servers per namespace or one per. Since we our now moving to putting Minecraft Server for our customers in Kubernetes, we will be using a namespace per customer to logically break up the applications (and more easily identify what goes where). So lets create our first namespace prior to deploying these manifests.

$ kubectl create namespace mineops-0
namespace/mineops-0 created
$ kubectl config set-context --current --namespace mineops-0
Context "minikube" modified.

We also ran another command and that was to set our current context to allow any manifest we apply to go into that namespace. The previous statement is true unless the manifest you apply has its metadata set to specify a namespace like this:

metadata:
  namespace: someothernamespace

The above snippet for whatever is in it will be applied only to the namespace someothernamespace regardless of the context of your kubeconfig. For our manifests we will be omitting this (to be explained later) so we will need to ensure we change our context as we did above.

Service (Networking)

Now for the best part, but also the "worst" part about this whole endeavor. There is nothing stopping us from running Minecraft Server on Kubernetes, however there is one issue I outlined earlier that we will run into. Since Minecraft uses its own version of the TCP Protocol, we are unable to use any typical Ingress or Gateway method to route traffic to a specific Container. We are going to have to do essentially what we did during all previous iterations of deploying Minecraft Server for our customers and that is 1 Server per IP. Sadly there is no known way around this (much of the reason this article took so long to come out was trying to engineer a way around it). So lets get on with what we can do and go from there. Here is what our Manifest will be and keep in mind this assumes that our Kubernetes cluster is running on a platform/provider that can provision IPs via a Load Balancer service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: mineops
  name: mineops
spec:
  ports:
  - port: 25565
    name: tcp-minecraft
    protocol: TCP
    targetPort: 25565
  selector:
    app: mineops
  sessionAffinity: None
  type: LoadBalancer

The above Service will act as a normal service except that because its type is LoadBalancer it will trigger the underlying platform to provision an IP address and "map" it to this Service as we will see in a moment. There is not much else to cover here for the networking portion so lets move on to storage.

PersistentVolume (Storage)

For nearly all Kubernetes clusters, you are going to have some dynamic storage provisioner in the cluster and can be verified by checking for a storageclass like so:

$ kubectl get storageclass
NAME                 PROVISIONER                RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
standard (default)   k8s.io/minikube-hostpath   Delete          Immediate           false                  6d5h

Depending on the cluster and where it lives, the name and provisioner fields will most likely have different data inside of them, but for the most part they will all function the same. Similar to our previous "storage" manifest we will just be creating a PersistentVolumeClaim which will check to see if there is a volume matching our request and if not, go create one (using whatever provisioner is default or selected).

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mineops
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 5Gi

Nearly all storage providers will be able to work fine for our scenarios, but there are some providers that work far better than others. Block level storage will grant us the best speeds for I/O of the Minecraft Server and is what we should look to use. If the platform we deploy to has the option of choosing a StorageClass and one of them is block, go for that one (which is what we will do). The StorageClass can be specified in the PersistentVolumeClaim manifest by adding this line to it:

spec:
  storageClassName: standard

Block storage is amazing with its I/O capability, but there are some minor things we need to work around prior to it working as some storage providers do. We will go over and discuss this in the next section.

Deployment (Compute)

All workloads on Kubernetes come from somewhere and no different than the VS Code Server example was used previously, we will be using a Deployment. The manifest will look something like this:

---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: mineops
  name: mineops
spec:
  selector:
    matchLabels:
      app: mineops
  replicas: 1
  strategy:
    type: Recreate
  template:
    metadata:
      labels:
        app: mineops
    spec:
      containers:
      - image: docker.io/mineops/minecraft-server:prod
        imagePullPolicy: Always
        name: minecraft-server
        ports:
        - name: minecraft
          containerPort: 25565
          protocol: TCP
        volumeMounts:
        - mountPath: /minecraft
          name: minecraft-data
        - mountPath: /minecraft/server.properties
          subPath: server.properties
          name: server-properties
      volumes:
      - name: minecraft-data
        persistentVolumeClaim:
          claimName: mineops
      - name: server-properties
        configMap:
          name: minecraft-server

For most platforms this will be just fine and work as intended, however for us using block storage for that sweet read/write performance, we are going to run into something else that needs to be added to the Deployment. It is a new concept called an initContainer, which just as its name sounds will be run on "initialization" of the Pod. initContainers can be extremely powerful, but just know they are not a "one and done". Every time a Pod starts up that has an initContainer, the initContainer will run. So if you have in the future an idea for an initContainer to handle some "pre Pod work", just know that it will run every single time the Pod is spun up.

initContainer

As mentioned earlier when discussing the manifests, some storage platforms have small challenges to overcome. For both the built-in storageClass of Minikube and many cloud provider default block storage class, we must implement a workaround for our Minecraft Server deployment. The issue with these block storage providers is that they mount the volume as root and our Container does not run as root because that is a very bad security practice. If we use the default storageClass of each of these providers, we will run into a permission denied issue when attempting to write to the PersistentVolume. Our workaround will involve an initContainer which we are going to use to "fix" our permissions prior to starting the main Minecraft Server Container. It follows many of the same spec outlines of our normal Pod as we can see here:

      initContainers:
      - name: data-permission-fix
        image: busybox
        command: ["/bin/chmod","-R","777", "/minecraft"]
        volumeMounts:
        - name: minecraft-data
          mountPath: /minecraft

The above code will go into our Deployment at the same "level" in the spec as the containers field. We could technically implement something like this into a startup-script called by the ENTRYPOINT of our Dockerfile, but there is no real need to do this, when its only needed on some platforms.

"Ingress" for Minecraft

As I mentioned in the description of the Service manifest above, Ingress will have to be handled as a 1 IP to 1 Minecraft Server. In almost all other Kubernetes Ingress related scenarios this is not the case regardless of application type. Nothing wrong here, but ideally you would want to reduce the number of IPs for your services and be able to have your Ingress Controller or ServiceMesh be able to Ingress traffic and route it for you.

Getting Configuration data into a Pod

Since we have yet to discuss ConfigMaps lets do that now. ConfigMaps can be used to store data in a "key:value" style specification. How you use them is up to you, but for us we will be using ConfigMaps to store our server.properties configuration file. This will be a somewhat different change from our Containers part of this series, where we had a local directory on a server with a server.properties that we pushed out via Ansible and maintained in Git.

If you noticed in the Deployment manifest above, we are mounting a ConfigMap as a volume for our Pod, but how do we get one and whats in it? We can create configMaps as any other object by having a manifest to create it with, or we can do something pretty neat and that is having kubectl "ingest" a file as its manifest as we can see below:

$ kubectl create configmap minecraft-config --from-file=server.properties
configmap/minecraft-config created

And if we were to look at our ConfigMaps via kubectl get -o yaml we would see that we have something that looks like this (some lines omitted for readability):

$ kubectl get cm -o yaml minecraft-config
apiVersion: v1
data:
  server.properties: |
    debug=true
    enable-jmx-monitoring=false
    rcon.port=25575
    gamemode=survival
    enable-command-block=false
    ~~~OMITTED~~~
kind: ConfigMap
metadata:
  name: minecraft-server
  namespace: mineops

One thing to note is the field above data is something new and can basically be viewed as the spec for certain types of objects with ConfigMaps and Secrets being some of the main ones. Everything will boil down to a key and a value regardless of the size of the value. Since we didn't go over many items for ConfigMaps here are a few examples (excluding what we see in the above snippet).

data:
  somevar: 42
  
  myscript.sh: |
    #!/bin/bash
    echo "hi"

NOTE: Although we didn't go into Secrets in this series, Kubernetes Secrets play a big role in storage sensitive data in a Kubernetes cluster and follow the same key:value model as ConfigMaps with one exception, the values inside must be base64 encoded and have no trailing newlines. More information on Secrets can be found here.

Deploying Minecraft Server

Now that we have all of our manifests its time to deploy our manifests into our cluster. Just as with every other manifest we can just run kubectl apply or kubectl create to get our manifests into the cluster. So here we go!

Looks like we've got everything created in our namespace and the Minecraft Server Pod seems to be up and running. Before we connect lets dig a little deeper and see whats going on with our Pod just for some more context and a bit more visibility using Kubernetes. We will watch the logs as we connect using kubectl since we aren't on a VM and don't have normal access to the local log file.

Connecting to our Minecraft Server

So now comes the best time of our deployment and that is to see if it works as we anticipate. We will have kubectl logs -f PODNAME running alongside so we can see the status of the Minecraft Server as well as verifying that we've connected.

Whelp, looks like that is another one down in the history books. We've officially done what our CEO has asked of us and deployed our Minecraft Server to Kubernetes. How do migrate our customer's existing data to Kubernetes from our existing Container infrastructure? In the next and final section of this post we will look at some of the extra things that never get talked about in a real world Kubernetes scenario.

Migrating to Kubernetes

Since we don't have direct access (at least over normal methods such as SSH) to the nodes in Kubernetes and our Minecraft Server data will live in a PersistentVolumeClaim from a storage provider, how do we get data into this PVC? As with most things, there are many methods and some may be cleaner than others. For us, we are going to open a simple HTTP Server that will serve the Minecraft Backup's we are taking (we do take backups right?). With that open, we will be able to retrieve the backup tarball from a Container using curl, extract it and then go stand up our Minecraft Server.

Exposing our backups

The method in which we are going to do this may be frowned upon by some, but since it is just a Minecraft Server backup and no company intellectual property, it isn't really that big of a deal. For more sensitive data that would be migrated, we would come up with a more elegant solution that involves either HTTPS or SSH with keys of some kind (or even an Amazon S3 bucket through Minio).

Thankfully Python has an extremely simple HTTP server that can be stood up and accessed easily. From wherever we are hosting our backups, we would go into the directory where backups are stored and then run the following (for Python 2.X the command would be: -m SimpleHTTPServer).

$ python -m http.server
Serving HTTP on :: port 8000 (http://[::]:8000/) ...

With that up, we can now use curl to grab our Minecraft Server backups, but we will need a Container in our cluster to do this. I've created a new set of manifests for deploying customer Minecraft Servers in a Kubernetes cluster. There are 2 manifests, a customer-init and a customer-run.  The customer-init manifest will deploy a Pod that we can then use kubectl exec -it customer-init -- bash to get into an interactive terminal to the Pod and then run our curl and tar commands. If we were mass migrating customers, we would do something a little different, such as have a Pod for each customer that would run a script to curl the backup and then extract it for us as we spin up each namespace (and we might do this in the next part of the series). For showing a "one off", this will do, but we will need to ensure when we do this, we have a clean environment, so we will start with a fresh namespace that has no running Minecraft Server (as if we were migrating a customer initially). We will have the PVC get created along with our Pod to be able to ensure a PVC is created in our namespace to be able to get data into it.

$ kubectl create namespace mineops-0 && kubectl create -f customer-init.yaml -n mineops-0
namespace/mineops-0 created
persistentvolumeclaim/mineops created
pod/customer-init created
$ kubectl exec -it -n mineops-0 customer-init -- bash
[root@customer-init /]# cd /mineops-data
[root@customer-init mineops-data]# curl -sL -o mineopsbackup.tgz http://somefileshare.com/backups/customer-0-server-1.tgz
[root@customer-init mineops-data]# tar -xf mineopsbackup.tgz
[root@customer-init mineops-data]# exit
$ kubectl delete pod -n mineops-0 customer-init
pod "customer-init" deleted

We have now manually obtained our one customer backup and extracted it into the PVC in their namespace (mineops-0). The only thing left to do is create the rest of the objects so the Minecraft Server can spin up and the customer can start playing. Once the objects have applied and we get the Public IP for the Service, DNS can be updated to target that Load Balancer.

$ kubectl create -f customer-run.yaml -n mineops-0
configmap/minecraft-server created
deployment.apps/mineops created
service/mineops created
$ kubectl get svc -n mineops-0 mineops
NAME              TYPE           CLUSTER-IP       EXTERNAL-IP      PORT(S)           AGE
service/mineops   LoadBalancer   10.245.225.155   143.244.222.81   25565:31568/TCP   9h

The pod should be up and running and checking the logs should show that:

$ kubectl logs -n mineops-0 mineops-64d95d95f4-kbpqf
[20:58:27] [Worker-Main-2/INFO]: Preparing spawn area: 42%
[20:58:27] [Worker-Main-2/INFO]: Preparing spawn area: 42%
[20:58:27] [Worker-Main-2/INFO]: Preparing spawn area: 42%
[20:58:27] [Server thread/INFO]: Time elapsed: 65986 ms
[20:58:27] [Server thread/INFO]: Done (66.234s)! For help, type "help"
[20:59:16] [Server thread/WARN]: handleDisconnection() called twice
[21:24:58] [Server thread/WARN]: handleDisconnection() called twice
[22:04:00] [User Authenticator #1/INFO]: UUID of player Hardcoreqq is fc138322-655d-4534-b351-7cb178e7c3da
[22:04:00] [Server thread/INFO]: Hardcoreqq[/10.108.0.3:35082] logged in with entity id 80 at (163.5, 67.0, 238.5)
[22:04:00] [Server thread/INFO]: Hardcoreqq joined the game
[22:04:06] [Server thread/INFO]: Hardcoreqq lost connection: Disconnected
[22:04:06] [Server thread/INFO]: Hardcoreqq left the game

Wrapping Up

We've covered just about everything we need to know for using Kubernetes to deploy workloads (mainly Minecraft Server instances) for our customers and how to migrate them from somewhere else. There are many other topics in the Kubernetes world, but as long as you remember that each of them boils down to one of the big 3 (compute, networking and storage), it will make your lives all that much easier.

There are other areas that do relate to deploying Minecraft Servers on Kubernetes that we did not go into too much depth such as managing multiple namespaces for customers, but we will be discussing that in greater detail in the next and final part of this series. And remember whenever you have manifests or configuration files, always put it in Git!

Up Next

Since we have pretty much concluded our deployment to Kubernetes and have a path forward, our CEO seems to be pretty happy that all of his buzzwords have been implemented. We on the other hand now have to manually create namespaces and manifests as opposed to how we used to do all of this with Ansible even when using Containers. So it seems we still have some work to do, but fear not, there are similar tools for automating workloads on Kubernetes and we will be going over one of the major ones in the next part of this series.

Another item we will look at is how to we update the server.properties and restart our Minecraft Server to take in the new changes? Can we do it automatically or will we have to run kubectl commands? Find out the answer to this and more in the next and final installment of the mineOps series!

Series Links:

• Part 1 - Manual Minecraft Server Installation
• Part 2 - Automating with Ansible
• Part 3 - Keeping it Altogether with Git
• Part 4 - Containers Have Joined the Party
• Part 5 - Making Containers Highly Available
• Part 6 - ArgoCD to the Rescue

The mineOps series is still ongoing and Part 5, Making Containers Highly Available, should be released here in the next two weeks or so. Following that the final part in the series, ArgoCD to the Rescue, will be out probably a week after. This final post will conclude the series and follow more or less a typical journey a company and its engineers may take moving through the technology stack.

On the horizon the next few posts are in the wings and should pair well with those who followed along with the mineOps series or for those who are already on this journey. The upcoming posts will focus on Tekton and its use as a replacement for existing pipelines in your workflows specifically targeted at replacing Jenkins. Please note that is me saying that and not a tagline from the Tekton group.

And the next article coming after that will be focusing on Terraform, another great product from Hashicorp that has been used to great success for many companies since it was released. At first Terraform will appear to be no different than Ansible, however there is much more to it than this and we will dive into those similarities and differences.

In Part 3 we started to use Git to track our files for the company and then put these files in GitHub as our remote repository. Business is good, we've got Minecraft Servers running and happy customers, but trouble is on the horizon. Despite things going well, our CEO has heard about a buzzword called "containers" and its our job to make this a reality. So let's start this journey of containerizing our Minecraft Servers and ultimately cut-over our customers if we can make it work.

What is a Container

Currently our company (in theory) is using virtual machines to host Minecraft Servers for our customers which is using a technology called "Virtualization". It has been around for quite some time and has some amazing adoption as it really allows you to utilize the most out of your physical hardware. Containers are not virtual machines. This is a common misconception by most people making the jump to Containers, but is quite untrue. I know people in my industry who know this yet continue to treat their Containers like virtual machines and I hope during this guide we can cover why you wouldn't do that and how easy it is to go down that road.

Another misconception is that Containers are Docker Containers or that Docker is the only Container "type" out there.  Docker has become synonymous with Containers just like the term SLI did for multiple GPUs (even if using AMD Crossfire) or Kleenex just meaning a tissue. This is unfortunate as there are other Container engines out there and this is something else I hope to dispel in this post.

We don't have to go over the entire history of Containers and what they are in reality (kernel namespaces which have been around in Linux since the turn of the millennium), but know that UNIX had their own equivalent (for the most part) developed back in 1979 and live by the early 80s. I will simply what they are for everyone's sake. Containers in theory are just another PID/Process on a server. Obviously there is a bit more to it than that, but it's best to think of them as processes. You can and will have more processes in side this Container, but the core of the Container should be thought of as a single process. Failing to do this will lead to bloat and have VM sized Containers and defeating a lot of the advantages to using Containers.

The large benefit of Containers over Virtual Machines is the overhead. A Virtual Machine is an entire Operating System with all of its libraries and software making its "package" the size of, well , an Operating System. If you wanted to run an Apache Web Server (httpd for the cool kids) on a Virtual Machine you would have to have an entire Virtual Machine or a physical server with Apache installed and then your website/web application along with it. A Container on the other hand can be as small as the actual application and data itself and nothing more. In the example of Apache, this could be as small as 10MB as opposed to the same VM with Apache installed being probably 4GB+. The benefits of containers is far more than just the size of the "package" (Virtual Machine vs Container Image) and hopefully we will see this benefit here shortly as we go through this guide.

Container Images

In the previous sentence I mentioned a new concept called a Container Image. If a Virtual Machine is comprised up of an entire Operating System and its installed software, a Container Image is just a packaged application and its dependencies along with information on what processes it runs when launched. Container Images typically tend to be quite small, but when starting out most people end up with larger Container Images as they learn how to build their own. We will go over building our own Minecraft Server Container Image later in this guide.

One interesting thing about Container Images (and other things in the modern tools and applications) is the concept of a tag. With Container Images since they are just a packaged application you can use tags to identify what version of your application a Container Image has. The format looks like this: minecraft-server:1.18.1. If a tag is not specified (both when building an Image or running a Container) the tag called latest will be used in its place. We will go over tags and latest later in this guide.

For Container Images, all we need to know now is that its a packaged application, which we can tag to show the version inside. For us we will end up building a Minecraft Server image.

So what is Docker then?

Previously we stated that Docker != Containers and that Containers != Docker, but anytime we look at something related to Containers we see Docker. Why is that the case if the concept of Containers has been around longer than Docker the company? The reason Docker is so synonymous with Containers is that it was because of Docker (both the company and the application) that brought Containers to the forefront of IT nearly a decade ago now. A talk was made and people started using Containers through Docker (the application/engine) and here we are today, however Docker isn't the only Container "runtime" in town.

Without doing a large deep dive on the concept of what a Container Runtime is, essentially its the "low level" program that runs Containers. Now we will discuss what a Container Engine is. Container Engines are applications we humans can use to manage, package and run containers (Container Engines come with their own Container Runtime). Docker is one of these Container Engines (and arguably the most popular by far) and for sure the one most widely available for Desktop Operating System usage. But in the wise words of Yoda...

Podman is fairly new to the scene of Container Engines, but it comes packed with a ton of features. For most beginners though, these features wont be realized until growing a little more in your journey with Containers. When it comes to developing Container Images and running Containers both Podman and Docker ultimately do the same thing, but how they do it is slightly different.

Docker runs in a daemon which whereas Podman has a daemon-less architecture. On the surface this may not seem like something important or meaningful, but it does grant some nice functionality mostly in the security space. Since Docker runs as a daemon (and that daemon runs as root) the containers it spins up run as the root user. Podman since it doesn't have a daemon it runs as, can spin up Containers as the user who spun up the Container as opposed to the root user. Docker did however recently add in this functionality, but Podman did it first and has been pushed as one of the primary features.

There is no right answer and for the sake of running and building Containers either will work for what we are doing here. Now for most people Docker Desktop is going to be the easiest option to get up and running as it can be installed on the two major Desktop Operating Systems whereas Podman can really only be installed on a Linux based OS. For our tests we will be using Docker, but the commands between Docker and Podman will be nearly a 1:1 mapping. Now let's get started actually using Containers!

Installing Docker

As with most applications we need to get it installed before we can use it. For Docker Desktop all you need to do is visit https://www.docker.com/products/docker-desktop and click the link for your specific Operating System. Note if you are using a Linux based OS for your Desktop or using a Linux based server, installing Docker may be a little different depending on your OS. Here are some links for each of the major Linux distributions and how to install Docker on each of them:

• Fedora
• CentOS 8
• Debian and Ubuntu variants

Then just go through whichever installation method your OS uses and you are pretty much done. Just a side note for Mac and Windows that this installs a minimal Virtual Machine (yes I see the irony that it uses a Virtual Machine on an OS to then use Containers) for Docker to run in.

If you are using Linux and wish to not go through the various install guides for Docker, you can install Podman through your package manager with the package name being: podman

How to use Containers (Docker Desktop)

Once Docker Desktop is installed you can open a Terminal (Mac) or Command Prompt/WSL (Windows) to start using Docker. If you are on Linux, most likely you already have a Terminal open because you are awesome.

All commands will be run as docker or docker.exe on Windows, although I believe Docker Desktop on Windows if you have WSL installed you can just run docker from WSL.

For those curious about Podman or are going to use Podman all the commands we will go over will be a 1:1 with Docker. If we do run commands that are not 1:1 I will be sure to call them out explicitly.

Common Commands

• docker run httpd:latest - Runs a container from the image httpd and tag
• docker stop <container-name> - Stops a running container
• docker rm <container-name> - Removes a stopped container
• docker ps -a - Lists all containers running or stopped
• docker images - Lists all container images
• docker start <container-name> - Start a stopped or created container
• docker logs <container-name> - View the logs of a container

Running a Container

As with most training we must do the inevitable "Hello World" so lets just get it over with. I do see the irony of constantly saying no to typical "Hello World" tutorials, but this one actually helps quite a bit as you will see. From your terminal throw this command into it and hit enter:

$ docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:2498fce14358aa50ead0cc6c19990fc6ff866ce72aeb5546e1d59caac3d0d60f
Status: Downloaded newer image for hello-world:latest

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

See that wasn't so bad now was it? Heck this version of a "Hello World" even told us what it did, which is way cooler than most "Hello World" tutorials. As we can see in the output the Docker command we ran did the items above:

• docker contacted the Docker daemon (remember it runs as a daemon)
• Docker daemon pulled the hello-world image from Docker Hub. How and why it did that will be explained later, but notice the first few lines after the command ran and see if you can gather what occurred.
• The Docker daemon created a new container from the Image it pulled and executed a command to spit out the output we saw
• This output was streamed via stdout to our terminal

Awesome, so we are done now right? We have now run a Container to tell us it ran a Containerized application so lets tell the CEO to get off our back. Sadly I think we all know we aren't there yet. Let's try something else with our docker run hello-world command to showcase something closer to what we will be doing in the "real world". We will be running the same command and passing in a -d flag which runs the Container in a "detached" state. Without this flag our Container will run and any stdout, stderr or output the container creates will be spit out to our terminal. Detached Containers however (which is how all Containers are run in the real world) have a slightly different output. Let's check it out:

$ docker run -d hello-world
6730c8a06b4b68c078d949cd230e44f301b6f356a3403628e75cb2ff7e5c38f1

$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

$ docker ps -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
6730c8a06b4b   hello-world   "/hello"   3 seconds ago   Exited (0) 3 seconds ago             priceless_khorana

$ docker logs 6730c8a06b4b

Hello from Docker!
This message shows that your installation appears to be working correctly.

To generate this message, Docker took the following steps:
 1. The Docker client contacted the Docker daemon.
 2. The Docker daemon pulled the "hello-world" image from the Docker Hub.
    (amd64)
 3. The Docker daemon created a new container from that image which runs the
    executable that produces the output you are currently reading.
 4. The Docker daemon streamed that output to the Docker client, which sent it
    to your terminal.

To try something more ambitious, you can run an Ubuntu container with:
 $ docker run -it ubuntu bash

Share images, automate workflows, and more with a free Docker ID:
 https://hub.docker.com/

For more examples and ideas, visit:
 https://docs.docker.com/get-started/

A few things were done here, but its important to see the flow and how we got there.

1. We ran docker run -d hello-world and got a long hash (kind of similar to the Git commit hashes we saw in the previous post), but no output like before
2. docker ps showed no running containers
3. docker ps -a which lists all containers showed us a container with the Exited status. Exited is an ok status and a good thing in most cases. More on this later
4. Through docker logs we were able to see the output that container gave us the first time we ran it.

Since we ran this Container in a "detached" state the logs weren't just shown in our terminal, which for the sake of finding out whats going on may seem bad. When it comes to running containers, you want them to run in the background and not keep a terminal actively streaming output (because it can get noisy and wastes your ability to use the terminal). We've seen some basics around running a Container, how to run them in a "detached" stated and get logs from Containers.

Some of this may seem silly, but we will pull all of this together later on in a following section.

Working with Images

In the first run of the docker run hello-world command we saw some output from Docker:

Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
2db29710123e: Pull complete
Digest: sha256:2498fce14358aa50ead0cc6c19990fc6ff866ce72aeb5546e1d59caac3d0d60f
Status: Downloaded newer image for hello-world:latest

When you attempt to run an image, Docker will check to see if it held locally in its image storage, if it does not find that image it will attempt to pull it from a "Registry" which in the case of Docker will only ever be Docker Hub. I do want to point out that Podman gives you the ability to set "Registries" to search from and pull images from. Let's not get to hung up on "Registries" just yet as we have a whole section dedicated to them coming up.

Notice how we ran docker run hello-world and the output said Unable to find image 'hello-world:latest' locally? The :latest tag (which we briefly went over) was appended to the image we attempted to run, why is that? If a tag is not specified when doing anything at all with Containers, the Container engine will use the tag latest to specify the "latest, last" tag of the requested image. Nothing really more to say on that except possibly that using latest may not always have the best results. It is best to always specify the tag you wish to use to ensure you have a repeatable environment. What if an image you rely on gets a new major update and the latest has some major changes you weren't expecting? Let's be honest, we should all have a dev environment to test things like this prior to pushing to production, but let's be even more honest and admit hardly anyone does that. By the end of this series we will be able to do that and remove a lot of potential headaches in our lives.

Container Images like hello-world exist to be consumed or expanded upon through a Build (next section we get all up in that). The hello-world image is quite basic and doesn't do a whole lot, but it does help us see what is going on. So we know a Container Image is just a packaged application with a command it runs when a Container starts with that image, but how do we get these images? Docker will try to pull any images it doesn't have when you try to run a Container, but you can pull Images on your own if you wish to do things with them. The docker pull command does just this and if you aren't sure what to pull you can run a docker search minecraft to see what's out there. This command will pull down an image to your local Docker (or Podman) image store, let's see it in action (side note I have an alias for docker image as dimage because typing is hard):

In the previous commands we viewed our local images, searched for any images at Docker Hub containing the name minecraft, we pulled down an image named itzg/minecraft-server and then pulled the image to our local image store (notice the line at the beginning that said Using default tag: latest). Take note that unlike the hello-world image there is a prefix name, itzg with a trailing / before minecraft-server.

We will dig more into this in the "Storing Images" section in a moment, but for now all you need to know is that image minecraft-server we pulled came from the repository of itzg hosted in the Docker Registry of Docker Hub. There is one final command we will run to work with Images (outside of running or building them) and that is docker inspect. Note that Podman has an inspect as well, but there is a powerful tool called skopeo that can do inspect on remote images without having to pull them. Let's look at what inspect does on the hello-world image as opposed to the itzg/minecraft-server image we just downloaded.

As you can see, there is quite a bit of output from docker inspect that we don't really need to go over here, but as you start to use Containers more, seeing what is in an Image and what makes it tick can be quite helpful. Now that we've pulled an Image from somewhere else that was built already, let's see what it takes to build our own Image.

Building Images

We know how to get images via pull or indirectly through run, but what if the image we want (or think we want) doesn't exist? Enter Image Building 101 and it all starts with something called a Dockerfile and yes it is capitalized. Let's take a look at the Dockerfile of the hello-world image which can also be found here.

$ cat Dockerfile
FROM scratch
COPY hello /
CMD ["/hello"]

The Dockerfile above is quite simple and straight forward. Most people even those not in "tech" should be able to somewhat decipher what is going on here. Just as a side note, Dockerfiles have their own syntax which is broken down like this: INSTRUCTION argument. But let's break down the one we have:

• FROM scratch - FROM tells Docker/Podman which Image to use as the "base" for this build. The scratch image is the tiniest of images and is nothing more than an executable environment for, well, executables.
• COPY hello / - COPY does exactly what it sounds like. It copies the file hello to / in the Image being built. Normal COPY behavior is to look in the current directory where you initiated the docker build. We will for sure dive into this in just a moment
• CMD ["/hello"] - CMD is what we spoke about earlier about instructions on what to do when an Image is run. In this case its running /hello or in a different way of looking at it as ./hello.

We are going to build our own Dockerfile for Minecraft Server and we will use all of the above Dockerfile Instructions and showcase some new ones. Let's get our Dockerfile sorted for a Minecraft Server. We know that we need Java, so we will need to look for a Container Image that has Java in it. We know that we need at least Java version 17 and we can do this via docker search or checking through the GUI of Docker Hub. Either way will work, but for looking through Images it may be easier to use the GUI at Docker Hub as the docker search command doesn't have the ability to look through tags.

As you can see, there were other options newer than 17, but just like I mentioned a while ago, using latest or the newest version of something may not always be a good idea. We know for a fact that openjdk-17 works for our server and our customers, so lets begin with openjdk 17. You may have also noticed there were multiple tags/versions of the same openjdk version such as "oraclelinux", "slim" and may have even noticed there were some Images with a windows architecture:

We most certainly do not want Windows based Containers for multiple reasons the most obvious being they wont run on our platforms and the second being of their size in comparison. This is something to pay attention to when looking for Container Images to utilize.

So let's start our Dockerfile with the Image and tag we found:

# Our base/starting Image
FROM openjdk:17.0.1-jdk-oraclelinux8

# Creating a dir for Minecraft Server
RUN mkdir /minecraft

# Telling Docker this is where we "live" currently. Similar to our "current directory" in Linux/Unix, all further commands "take place" in WORKDIR, which for us is "/minecraft"
WORKDIR /minecraft

# Copying over our files (eula and server.jar) to our /minecraft dir which is known due to the previous WORKDIR Instruction
# We will talk about server.jar being here in a moment
COPY eula.txt .
COPY server.jar .

# And the final line of our Dockerfile is "what to do when I'm run". Which this should look very similar to what we did way back in our manual setup of Minecraft Server.
CMD ["java", "-jar", "server.jar"]

We have our Dockerfile (with some in-line comments to help identify some of the options) and we can technically go ahead and do a docker build, but first lets take a look at our directory.

$ ls
Dockerfile  eula.txt    server.jar

We have our eula.txt and the server.jar which I downloaded manually. We can have our Dockerfile go and do this for us, but it gets very complex in a Dockerfile if we start doing large commands inside of it (think of our long curl command to go grab the latest Minecraft Server .jarfile). There are ways to make this work which we will look at later. So let's try to build this Image and see what happens:

Let's look at the command that we ran to build this Container Image and examine each component:

$ docker build -t mineops:17 .

• docker build - This first portion tells Docker we are going to build a Container Image from a Dockerfile
• -t mineops:17 - We have told Docker the Image that is built is to be named mineops with the tag :17. This tag was chosen to represent openjdk17
• . - This tells Docker the location of our Dockerfile. You can specify where your Dockerfile is and in a later article in this series we will discuss doing that. For now, building where you are works fine

So with that explanation out of the way, it looks like we built our Minecraft Server Container Image. Now the real test, does it work?

Looks like it does and we are one step closer to actually having a production ready Minecraft Server Container Image made. The command that was run to start this image was:

$ docker run -d --name mineops -p 25565:25565 mineops:17
$ docker logs -f mineops

A few new flags got introduced here via --name and -p. The --name flag starts the Container with a specific name, in this case it was mineops as opposed to the random generated name given by Docker. The -p flag exposes a port (or port range if needed) from the Container to the Docker host (our local machine in this instance). As we know from the first couple of installments of the mineOps series, Minecraft Server defaults to port 25565 to receive incoming connections. With -p 25565:25565 the syntax is host:container in how the ports are exposed/mapped. If we wanted to have our customers access their server over 25575 instead, we could either change the server.properties file (not immediately easy to do in a Container) or we could just change the port the Minecraft Server is exposed on the host.

One final thing to note here and this is a big one, Minecraft Server did create the usual files we've seen time and again inside the Container when it started. However Containers by default use "ephemeral" storage, which means that the files/data inside the Container are lost when the Container is deleted. We will be looking how to remedy this and keep our data towards the end of this article.

Storing Images

Now that we have built a Container Image and it is on our local machine, I guess we can just ship our machine to the customers that need a Minecraft Server right? Nonsense, but it is a funny meme about how Docker was born. So if we aren't going to send customers our laptops to use, where do we put a Container Image so our servers that have Docker/Podman installed can run it?

Remember how I mentioned Docker will pull Images from a Registry? Well those images got there somehow and now we will learn what the somehow is along with how the itzg/minecraft-server image had itzg/ in front of it, but the hello-world image did not. So what is a Container "Registry"? Much like a Git repository, a Container Registry stores Container Images and the various tags of those images. As you can imagine given that our one mineops:17 image we built is ~500MB, a Container Registry if it holds all of these images and versions/tags it would need a massive amount of storage to "contain" all of this right? Well, obviously it does need a decent amount of storage, but thanks to some fancy built in technology called "layers" not every single version and layer is uniquely stored on disk somewhere.

Layers, much like Container Images (and Git commits), have what is known as a hash or SHA. Each unique Layer is stored, but Images (and their versions/tags) that share certain layers do just that, share it. When you pull down a specific tag of an Image from a Registry and you already have an Image locally that shares a few of the layers (rhyming is fun eh?) only the differencing layers are pulled down to give you the new Image tag you pulled. Let's see this in action:

In that example, we pulled the Container Image nginx:1.21.4 and nginx:latest to showcase the idea of reusable/shared layers. There was one line after pulling the second image that stated Already exists. These Images only shared the first layer with each other (the "base" layer), but some images may share more than others. Typically in your own builds quite a few of the layers will be shared. Each layer comes from an "Instruction" i.e. RUN yum install java in a Dockerfile, so the changes they are unique is higher, but not in our own environment as we release newer versions of our images.

So for our Image we built, mineops:17, we need to push this somewhere like Docker Hub. As a side note for my personal Image Registry, I use Quay.io as it works for me and provides features I like that Docker Hub doesn't. For this series we will just use Docker Hub, but I would suggest checking out Quay.io as it has some pretty awesome features once you get to using Containers more often.

First and foremost we will need a login as it isn't anonymous. When you create your account at Docker Hub, the username you choose will be the prefix name of your repository, just like what we saw with itzg/minecraft-server. A Container Image Location is typically broken down like this:

docker.io/itzg/minecraft-server:latest

docker.io = Container Registry

itzg/minecraft-server = Container Image Repository

:latest = Image Tag

Once you've created your account and chosen your username, we can get started. I created a new account on Docker Hub for this with the name "mineops". The image we built previously was also named "mineops", so to avoid a duplication, we are going to do some magic with docker tag.

$ docker images mineops
REPOSITORY   TAG       IMAGE ID       CREATED       SIZE
mineops      17        e4b0329ca176   9 hours ago   517MB

$ docker tag mineops:17 mineops/minecraft-server:17

$ docker images
REPOSITORY               TAG  IMAGE ID       CREATED        SIZE
mineops                  17   e4b0329ca176   9 hours ago    517MB
mineops/minecraft-server 17   e4b0329ca176   9 hours ago    517MB

Notice how we have a new Image in our list, but every other bit of information is the same? All we did with the command docker tag was create a "pointer" locally. Let's try to login and see if we can push an image to Docker Hub.

We can pull Images, run Containers, build Images, push Images to a registry and we've just about covered every use-case for normal Container functionality for developing with Containers. Now there is one last thing to discuss before move on to building production grade Container Images for our customers to run their Minecraft Servers on.

Earlier I had mentioned about targeting a specific tag and not latest because you can ensure the version you are targeting will work for your use. Well if we build another image with some changes and tag it as :17 just like we did before and push it to a registry, its no longer the same image, but it is the same tag. This isn't something you will typically see happen with "Official" images from big name development companies, but it is something to be aware of.

There is a workaround for this and that is using the hash/SHA (formally known as the "digest") of the Image you know is good. You can get the digest of your specific Image (if its on your local image storage) by running a simple command:

$ docker inspect mineops/minecraft-server:17  | jq -r '.[].RepoDigests'
"mineops/minecraft-server@sha256:0e5242fbae2abf68df82f14a19d80f5e64d1afcc8241c84e2fe8e554831076d7"

You can use this digest in your docker run commands as well as in your Dockerfiles in the FROM line just as with any other image. You can basically think of the @sha256:longhash as a git commit hash since it targets a specific and unique point in history for the Image (just like a Git commit). This is just something to think about as you begin to use Container Images and not accidentally cause an outage because someone pushed to the same tag as before.

Making the CEO Happy

The CEO has given us our time to learn about Containers and wants us to start onboarding customers and move existing customers to Containers. Obviously we have a little bit further to go, especially as he also that we should migrate our customers to Containers. The last thing he left us with was to not waste too much time on this effort. During a conference he heard a new buzzword that he wants to drop on us and some of you may know what that buzzword is (hint: it starts with a K).

Getting Existing Data into the Container

We have a functional image via mineops/minecraft-server:17 we can use to test a few items before we move to our "final" Image, so let's get to it. We need to see how to get Docker to target existing data and "mount" it into the Container. Thankfully there isn't much to this and is a simple flag known as -v. This flag, similarly to the -p flag from above, mounts a host path or a Docker Volume into the Container. There are reasons to use either option, but for now we will focus on the host path variety as its easier to manage and "see" normally. Docker Volumes do have their place, but given that we know the CEO wants us to move to something else soon, we will stick with the easier of the two.

Our current customers have servers deployed through Ansible on a few servers such as mineops-0.kywa.io. These Minecraft Servers have their data stored in /minecraft and we are aware of all the files contained within, but if we are going to migrate customers we need to get this data into a Container. Since we are taking backups through Ansible lets go grab a copy of mineops-0.kywa.io and bring it to our local machine. Once we have the /minecraft directory contents on our local machine, we can begin testing out the -v flag for docker run.

And as we can see by using tail on the log of the local file (not the Container log) we can verify we are using this local data and not the ephemeral storage of the Container. So now we have a way to keep our data and still back it up in a normal fashion (aka its on a filesystem we can see). If we were going to use Docker on its own, we would absolutely use Docker Volumes for this, but we aren't, so no need to really dive into this.

Automating Containers

Since we know how to use existing data for our Minecraft Servers, all we need to do is find out where we are going to store this data. The logical solution here on our new server running Docker (this is a fictitious server and we aren't creating another VM), would be to create a /minecraft directory as we always do, but to create a sub-folder for each customer's Minecraft Server data. If we look back at our Ansible vars we can come up with a naming scheme for something like this per customer server: /minecraft/customerID/server0 for however many Minecraft Servers they happen to have.

So now we have a way to get existing customers into Containers by migrating their data to the Docker server and into the directory dedicated to that Minecraft Server. For new customers who want custom settings for server.properties or anything else, how do we go about getting them setup? With a "vanilla" Docker setup like we are using, BASH scripts are going to be a quick and easy friend for us, but we will  use our original Ansible playbooks and role as its what the business is currently using for everything else.

Obviously from what we've seen and learned from the Ansible deployment of a Minecraft Server to the Container deployment of a Minecraft Server they aren't exactly the same, so we will need to modify our existing Ansible to work with what we have. Since we have been tasked with migrating existing customers as well we will need to create some Ansible to copy their Minecraft Server files and data over to a new place where we will run Containers. I have created a basic script that will do more or less this for us:

---
###
#
# Playbook will be used to convert existing Minecraft Servers to Container Hosts
#
###
- hosts: all
  gather_facts: false
  tasks:
  - name: Container Prerequisites
    include_role:
      name: minecraft
      tasks_from: container-prep.yml

  - name: Stop existing server
    systemd:
      name: minecraft
      state: stopped
    ignore_errors: true

  - name: Container Create
    include_role:
      name: minecraft
      tasks_from: container-create.yml

  - name: Copy existing customer server
    copy:
      src: "{{ minecraft_directory }}"
      dest: "/opt/{{ container_path }}/{{ item.0.key }}/{{ item.1 }}"
      remote_src: true
    run_once: true
    loop: "{{ containers | dict2items | subelements('value') }}"

The above Ansible Playbook should be easy to read and see what is going to happen. You may notice, if you remember all that was done from Part 2, the include_role for minecraft, but the tasks_from is a new line. This allows you to call a specific tasks/file.yml inside of a Role as opposed to letting it default to tasks/main.yml inside the Role. This can get quite powerful, but be warned you can start creating bloated roles if you have just a bunch of tasks that just get called on their own and don't tie into the core Role function. Thankfully these tasks do tie into the Role's core function of deploying Minecraft. On their own we see 2 new ones: container-prep.yml and container-create.yml.

• container-prep.yml - This task installs the required tools and packages to run Containers on a Linux host. For us we are repurposing our existing servers as there was no need to provision new VMs
• container-create.yml - This is nearly identical to our original server-prep.yml adds the logic of start-server.yml. The focus is creating directories for our customers to store their Minecraft Servers since we can now through Containers (more easily anyways) run multiple instances per VM. This could have been done originally with Ansible, but required a little more "engineering" that wasn't required at the time

In theory all we need to do is alert our customers of upcoming maintenance and run playbooks/migrate.yml and voila we are good to go! Now with this, there is one more caveat. Previously with 1 Minecraft Server per server customer ports for Minecraft Server (the default was 25565) it didn't matter if they just used the default port. Now if we have more than one customer, there can only be one port unique port per server (I hope we knew this). This creates a new challenge for us as we automate. We need to try to keep a running list of which servers have which ports open so as not to cause any conflicts or failures when starting Minecraft Servers. For now it will be a manual process for our business, but there are ways around this, just not in the scope of this article. We "may" get to look into ways around this with a new technology (spoiler: we will), but we don't know about it yet.

One last note about our Ansible, we have added a new directory in our role called handlers. A handler in Ansible is a task or set of tasks that run when "notified" by another task and run at the end of a play. To call a handler there are 2 requirements:

1. The task that calls the handler must complete with a changed state (aka it actually did something and not be skipped or ok.
2. A new line needs to be added to the task: notify

The notify keyword must have the name of the handler task you wish to run. Here is ours for example

- name: Create the systemd service from template
  template:
    src: container.service.j2
    dest: "/usr/lib/systemd/system/{{ item.0.key }}-{{ item.1 }}-container.service"
  loop: "{{ containers | dict2items | subelements('value') }}"
  notify: start_containers

---

- name: start_containers
  systemd:
    name: "{{ item.0.key }}-{{ item.1 }}-container.service"
    enabled: yes
    state: started
  loop: "{{ containers | dict2items | subelements('value') }}"

The notify above if it causes a change will trigger the task called start_containers after the play it was called from has finished. This is a very powerful way to handle changes in your Ansible environment and I wanted to make sure you were aware of it prior to moving on.

We have all of this new Ansible code, but what do we do with it? Oh yeah, keep it in source control!

• git add -A . && git commit && git push

Building a Production Ready Image

And back to Containers! (almost). We need to setup something to build our Container Images. We can do this manually, but having Ansible do all of this for us would be a big help in automation AND we get to learn a new tool for Ansible called ansible-vault (no extra installs required, its part of the Ansible package).

Let's look at how to build an Image with Ansible. There are 2 collections in Ansible we can use, one for Docker, the other for Podman. We are going to use the one for Podman as that is what we have chosen to run our Containers with (from an Automation standpoint) as it doesn't have a daemon as mentioned previously. There is no need to worry about differences as the task names are nearly identical and have the same syntax (if you're using Docker). Let's take a look at an example:

- name: Build mineOps - Minecraft Server
  containers.podman.podman_image:
    name: minecraft-server:{{ minecraft_build_tag }}
    path: "/tmp/minecraft-builds"
    build:
      cache: no
      force_rm: yes
      format: oci
      
- name: Build and push an image using username and password
  containers.podman.podman_image:
    name: minecraft-server:{{ minecraft_build_tag }}
    push: yes
    push_args:
      dest: docker.io/mineops/minecraft-server
    tag: "{{ minecraft_build_tag }}"
    username: "{{ docker_hub_username }}"
    password: "{{ docker_hub_password }}"

The above 2 tasks are the core foundation for building and pushing a Container Image to a Container Registry. There is some other logic we will build around this, but the core are those 2 tasks. Let's kick off a test run and see what we get. We will dig into the where docker_hub_password is later along with using ansible-vault to protect these secrets.

Looks like we have ourselves a working Ansible task to build a Container Image and push it to a Registry! The full Ansible task for this can be found in the repo HERE. Now lets look at a few other tricks to make this a little more secure instead of having a seemingly plain text variable somewhere.

Keeping things Secure - Ansible Vault

In the variable {{ docker_hub_password }} it looks like any other variable that we've seen in Ansible. However, its not just a plain text var: string like you would expect. Here is what it actually looks like:

$ cat inventory/group_vars/all/vault/secrets.yml
$ANSIBLE_VAULT;1.1;AES256
32303733353062663135313337303436633138303162393534313637353937646335363133313039
3762333632393531663464376664333234643461646366660a343633353236366164386265353136
30343064346335663830383361326336393937376336616336663934323835623163376131326563
3635353332636235330a623765323966613935393130643839643562633436643763643238333466
62326537323032636533656135306430623363666536373531376636343037393535336134643466
33333537636435376363643261656134623236313131376632666236636138626231336661316438
61663364333135303630323664323061643561393862376339396532333739643665643436386138
63326430396463656464

Now I don't know about you, but that doesn't look anything like a vars file that we've seen so far, but somehow it has data in it. Let's do a quick primer on ansible-vault (note you can use this outside of Ansible for easy storing of secret data.)

When Ansible is installed another "program" is installed along with it called ansible-vault. This tool gives you a few capabilities:

• view - View an encrypted file
• decrypt - Remove encryption from a file
• encrypt - Encrypt a file
• edit - Edit an encrypted file

Now its time for practice! Here is a quick little example of what it looks like to use ansible-vault on its own.

One of the neat things about ansible-vault is that the data inside doesn't have to be a key: value variable style set of data. It can be anything at all. Any time of file, which can come in super handy in case you wanted to encrypt an entire file of lets say, a dockercfg.json containing login data to a Container Registry or just data you don't want visible in Git in general.

Now lets talk about how its used inside of Ansible. If you remember from Part 2, any files placed inside of inventory/group_vars/<GROUP NAME> will be pulled into the tasks of the hosts being targeted if they belong in those Group(s). There is also a special group called all and it is there we are going to create a new directory called vault. NOTE: You do not need to have files encrypted by ansible-vault placed in a directory called vault. This is purely for humans working with the repo to know that all files under vault are encrypted.

So how does Ansible use these files if they are encrypted? When you run ansible-playbook and Ansible pulls in all the files in inventory its going to use, it will detect the files that are encrypted with ansible-vault. If you do not pass in a flag on what to do with encrypted files, you will get a message like this:

$ cat vaulttest.yml
---
- hosts: all
  gather_facts: false
  tasks:
    - name: Check variable through ansible-vault
      debug:
        msg: "echo {{ docker_hub_password }}"
      delegate_to: localhost

$ ansible-playbook -i inventory/hosts vaulttest.yml
PLAY [all] ****************************************************************************************************************************************************
ERROR! Attempting to decrypt but no vault secrets found

There are a few ways for Ansible to use these encrypted files. Each of the methods I will outline are used as parameters when running ansible-playbook:

• --ask-vault-password - When running ansible-playbook you will be prompted with Vault password: for you to enter the vault password(s) needed.
• --vault-password-file=/path/to/password_file - You can store the password in a plain text file for Ansible to read as it runs. This isn't the most secure method, but for testing and rapid usage, it works fine

There are even more secure and advanced methods, but will not be outlined in this guide. Check the Ansible Vault documentation on implementing this on your own.

Let's run our playbook again, but this time giving it the vault password:

$ ansible-playbook -i inventory/hosts vaulttest.yml --ask-vault-password
Vault password:

PLAY [all] ****************************************************************************************************************************************************

TASK [Check variable through ansible-vault] *******************************************************************************************************************
ok: [mineops-0.kywa.io -> localhost] => {
    "msg": "echo PASSWORD_OR_TOKEN"
}

PLAY RECAP ****************************************************************************************************************************************************
mineops-0.kywa.io          : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

$ ansible-vault view inventory/group_vars/all/vault/secrets.yml
docker_hub_password: PASSWORD_OR_TOKEN
docker_hub_username: mineops

As you can see, Ansible was able to get the variable docker_hub_password out of encrypted file and gave us the value and it can be used as any other variable in Ansible. There is more to ansible-vault, but this covers most use cases and can be grown into your actual environment. For us, this is about as far as we will take it. And now its time for one of the final items and that is building a "production grade" Container Image.

Building our Production Image

We are going to create a template for our Dockerfile we will use for our final "production" grade image. There isn't much difference between this and our original Dockerfile, but there are a few things we are adding in. Some labels, a UBI openJDK image from Red Hat (targeted via digest!) and some other interesting tidbits.

FROM registry.access.redhat.com/ubi8/openjdk-17-runtime@sha256:1a40771ffd18a688a4492f319814af5d2e3377eb293c69ca682f29a3a86635ab

USER 0

# Label Image with important information
LABEL builder="{{ mineops_email }}" \
      application="minecraft-server" \
      java_version="{{ minecraft_java_version }}" \
      minecraft_version="{{ minecraft_version }}"

EXPOSE 25565

COPY mc-start.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/mc-start.sh && curl -o /usr/local/bin/jq -sL https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 && chmod +x /usr/local/bin/jq

VOLUME ["/minecraft"]

WORKDIR /minecraft

USER 1001

CMD ["/usr/local/bin/mc-start.sh"]

If we check out Docker Hub we can see our new tags:

We can use our same build Playbook as before except we have just a slightly different command as we will be passing in a variable through the CLI and not in a variable file:

ansible-playbook -i inventory/hosts playbooks/build_image.yml --ask-vault-pass --limit mineops-0.kywa.io -e minecraft_build_tag="prod"

And essentially what we have here is a complete "production ready" build Container Image. We have slightly changed the way in which our image works, but the core is the same, a running Minecraft Server Container Image (again). This new method uses a startup script so we can mount a volume for persistent storage, or in our current case, a local directory.

Before moving to the final section here is the command to run this Container (NOTE: This will start the server in your current directory):

# If the system has SELinux enabled
docker run -d -p 25565:25565 -v "${PWD}":/minecraft:z minecraft-server:latest

# And without SELinux enabled
docker run -d -p 25565:25565 -v "${PWD}":/minecraft minecraft-server:latest

Adding Visibility

After getting all of this going, surely by now we can get a fancy UI to look at all of what we've built and have running? You can run docker ps -a and docker images all day long, but sometimes you just want a "single pane of glass". There are a few options here to do this. Portainer, Rancher and a few others, but we will focus on Portainer as its the "easier" to stand up and get going. Thankfully its also runs as a Container, so we already have our platform. Get it running quickly like this:

docker run -d -p 8000:8000 -p 9443:9443 --name portainer \
    --restart=always \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v portainer_data:/data \
    cr.portainer.io/portainer/portainer-ce:2.9.3

Once you have it up and running you can go to https://localhost:9443, create a password for the admin account and get a pretty UI like this:

From the little gif there you can see a glimpse of some of the features it has. It is more features than most people or businesses need to run a Container Host, but has features that allow it to manage more than a single Container Host.

Here we go again...

As we migrate our final customer over to Containers, we get a knock on our door. We stand up and stretch after a long day's work to go open the door to find no one other than our CEO with a smile on his face. He tells us that after the huge success with Containers, we need to keep modernizing because we are using "old technology" now. We are told an email will be in our inbox soon for us to review since we are rockstars and can "knock this out in a weekend".

The CEO lives and we go sit back at our desk (and cry ever so slightly) we open our email client and see a new email with this subject line:

Subject: RE: Check out kubernetes.io

Series Links:

• Part 1 - Manual Minecraft Server Installation
• Part 2 - Automating with Ansible
• Part 3 - Keeping it Altogether with Git
• Part 4 - Containers Have Joined the Party
• Part 5 - Making Containers Highly Available
• Part 6 - ArgoCD to the Rescue

In Part 2 of this series we began to use Ansible to manage our customer servers from deployment, backup and configuration. We ended that post discussing what to do with all of those files which has led us here, to Git. There are some out there who are not familiar with Git would be turned away from its "stigma" of being complicated. These same people would attempt to use DropBox or Google Drive for their source control needs and would come up lacking down the line at some point. That isn't to say tools like them don't have their place, but when it comes to version control of source code they really do come up lacking.

Just a quick example to make a case for Git over file sync services such as those listed above, suppose you need to make changes to a large amount of files? In a file sync application, you can restore files to a specific point in history, but will have to do this to each file changed and make sure that each of the versions restored are matched to the same point in history. With Git you can rollback the entire project to a point in history.

Another great feature is something called blame which essentially tells you which line of a file was last edited by whom. We will touch on this feature more later, but there are many other reasons on why to use Git for your source control needs.

Installing Git

As with Ansible there are a few ways to install Git, but the package manager of your system will probably be the best bet:

• yum install -y git

You can also download and compile from source if you need a newer version than what your package manager has: https://github.com/git/git/tags

Once you have Git installed, verify its installed and in your path. If you have verified you have Git installed we can proceed to the next section.

$ git --version
git version 2.32.0

Understanding Git

If you are familiar with Git, feel free to move along to the next section. If you wish to have a very in depth tutorial series on Git, I highly recommend checking out this YouTube channel:

Dan Gitschooldude

This channel is dedicated to teaching the distributed version control system git, the best damn source control management tool that has ever existed. Git is quite powerful and can be complicated to learn, but fear not! Watch these tips and tutorials, and you’ll be an expert in no time. In additio…

YouTube

For the Git basics that are recommended for our business, I've created a quick primer for Git and its main commands and phrases used, but we will also go over some of these in detail in the next few sections: https://github.com/KyWa/blogs/blob/master/mineOps/docs/Git/git-primer.md

Working Directory - Local Repository

The core feature of Git that most people will ultimately use can be outlined easily. There is the "working directory", which for our purposes will be the directory where we keep our Ansible code from the last section. This "working directory" is also called the "local repository". This local repository stores your files contained within, but most importantly the history of these files as a whole. To make a directory a Git repository, issue this command in the "top level" of the directory containing the files you wish Git to keep track of. For us in our example, we will be using the "modular_ansible" directory as our "top level":

$ pwd
~/modular_ansible

$ git init
Initialized empty Git repository in ~/modular_ansible/.git/

Once the above has been run our "working directory" where we have all of our files and directories such as playbooks, roles and inventory now exist as part of the local repository, but these files aren't being tracked by Git yet. Let's take a look at a command git status and see what it tells us:

$ git status
On branch master

No commits yet

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        ansible.cfg
        inventory/
        playbooks/
        roles/

nothing added to commit but untracked files present (use "git add" to track)

With the git status command we can get some very important information about our new local repository. One nice thing about Git is that it is very good (sometimes annoyingly so) about telling you what is going on and what you need to do to get a clean and up-to date repository.

• On branch master - This first line tells us the name of the branch we are on (more on this topic later)
• No commits yet - Something you will only ever see on a fresh Git repository via git init
• Untracked files - Now this is where Git starts to show its power. This output shows all the files in the working directory that are not part of the `index`. We will discuss the index further in one of the next sections.
• nothing added to commit but untracked files present (use "git add" to track) - This is Git telling us that there is nothing committed to the `index`, but it is aware of new files that are untracked. Git is also kind enough to tell us what we need to do to track these files.

git status is a command you will get familiar with very quickly as it tells you the "state" of your local repository. Let's move on to the next topic, commits, but to learn about commits we need to understand the index.

Index

The index in Git is a "staging area" for the files in the working directory. In the git status output from earlier, we noticed Git told us this: Untracked files and proceeded to list the files and directories in our local repository. These files are not part of the index, but are part of the working directory. If we issue the git add command from earlier that Git gave us, let's see what our git status looks like and we can see the index change.

$ git add ansible.cfg
$ git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   ansible.cfg

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        inventory/
        playbooks/
        roles/

Notice we have a new part of output,Changes to be committed. This status gives us some very interesting information:

• use "git rm --cached <file>..." to unstage - This shows us that we have a new staged file as part of our index
• new file: ansible.cfg - Git tells us about the file we added via git add and is the file thats part of our index

Let's go ahead and do a git add on the rest of the files in our working directory:

$ git add .
$ git status
On branch master

No commits yet

Changes to be committed:
  (use "git rm --cached <file>..." to unstage)
        new file:   ansible.cfg
        new file:   inventory/group_vars/customer-0/config.yml
        new file:   inventory/group_vars/customer-0/server-properties.yml
        new file:   inventory/group_vars/customer-1/config.yml
        new file:   inventory/group_vars/customer-1/server-properties.yml
        new file:   inventory/group_vars/minecraft/config.yml
        new file:   inventory/group_vars/minecraft/server-properties.yml
        new file:   inventory/host_vars/mineops-0.kywa.io/config.yml
        new file:   inventory/host_vars/mineops-0.kywa.io/server-properites.yml
        new file:   inventory/host_vars/mineops-1.kywa.io/config.yml
        new file:   inventory/host_vars/mineops-1.kywa.io/server-properites.yml
        new file:   inventory/hosts
        new file:   playbooks/minecraft-backup.yml
        new file:   playbooks/minecraft-config-change.yml
        new file:   playbooks/minecraft-install.yml
        new file:   playbooks/minecraft-restore.yml
        new file:   playbooks/minecraft-uninstall.yml
        new file:   roles/minecraft/files/eula.txt
        new file:   roles/minecraft/tasks/backup.yml
        new file:   roles/minecraft/tasks/config-change.yml
        new file:   roles/minecraft/tasks/main.yml
        new file:   roles/minecraft/tasks/restore.yml
        new file:   roles/minecraft/tasks/server-prep.yml
        new file:   roles/minecraft/tasks/start-server.yml
        new file:   roles/minecraft/tasks/uninstall.yml
        new file:   roles/minecraft/templates/minecraft.service.j2
        new file:   roles/minecraft/templates/server.properties.j2

Our index has grown and we no longer have any Untracked files in our working directory and our entire repository. The only item we have left in our git status that Git is telling us about is Changes to be committed. With that, lets move on to learning about commits.

Commits

Commits are "snapshot" of the working directory and the files contained inside that Git is keeping track of. As we saw in the git status above in the last section, Git told us we had Changes to be committed based on the files we added to the index. If we issue the git commit command on its own, we will be brought to your default text editor. This will have you input what is called a commit message which is human slang for "what are you committing to this local repository?". There is another method using git commit where you do not go into an editor, but are able to pass in the commit message in-line via: git commit -m "my message".

For the sake of this guide and getting going, lets use the quicker method:

$ git commit -m "initial commit, adding repo files"
[master (root-commit) 7ba22bd] initial commit, adding repo files
 Committer: Kyle Walker <kylewalker@Kyles-Mac-mini.kywa.io>
Your name and email address were configured automatically based
on your username and hostname. Please check that they are accurate.
You can suppress this message by setting them explicitly. Run the
following command and follow the instructions in your editor to edit
your configuration file:

    git config --global --edit

After doing this, you may fix the identity used for this commit with:

    git commit --amend --reset-author

27 files changed, 437 insertions(+)
 create mode 100644 ansible.cfg
 create mode 100644 inventory/group_vars/customer-0/config.yml
 create mode 100644 inventory/group_vars/customer-0/server-properties.yml
 create mode 100644 inventory/group_vars/customer-1/config.yml
 create mode 100644 inventory/group_vars/customer-1/server-properties.yml
 create mode 100644 inventory/group_vars/minecraft/config.yml
 create mode 100644 inventory/group_vars/minecraft/server-properties.yml
 create mode 100644 inventory/host_vars/mineops-0.kywa.io/config.yml
 create mode 100644 inventory/host_vars/mineops-0.kywa.io/server-properites.yml
 create mode 100644 inventory/host_vars/mineops-1.kywa.io/config.yml
 create mode 100644 inventory/host_vars/mineops-1.kywa.io/server-properites.yml
 create mode 100644 inventory/hosts
 create mode 100644 playbooks/minecraft-backup.yml
 create mode 100644 playbooks/minecraft-config-change.yml
 create mode 100644 playbooks/minecraft-install.yml
 create mode 100644 playbooks/minecraft-restore.yml
 create mode 100644 playbooks/minecraft-uninstall.yml
 create mode 100644 roles/minecraft/files/eula.txt
 create mode 100644 roles/minecraft/tasks/backup.yml
 create mode 100644 roles/minecraft/tasks/config-change.yml
 create mode 100644 roles/minecraft/tasks/main.yml
 create mode 100644 roles/minecraft/tasks/restore.yml
 create mode 100644 roles/minecraft/tasks/server-prep.yml
 create mode 100644 roles/minecraft/tasks/start-server.yml
 create mode 100644 roles/minecraft/tasks/uninstall.yml
 create mode 100644 roles/minecraft/templates/minecraft.service.j2
 create mode 100644 roles/minecraft/templates/server.properties.j2
 
$ git status
On branch master
nothing to commit, working tree clean

If we look at the top of the output, we will see a message stating who the Committer was. When using Git for the first time (and depending on version this output may be different), Git needs to know who is actually committing these changes. This information has important use later on and we will discuss the git config command in a few sections. The git commit command we ran primarily tells us how many files were "changed" in the local repository and how many "insertions". As we will see later in this guide, when we edit files and remove lines, we will see "deletions" as well as "insertions". This is a handy way of having a 30k foot view of the repository and its latest commit changes. Commit messages are very important as they can/should be used to outline what is contained within a commit. Our first commit we said it was the "initial commit, adding repo files". Now, how do we view this commit message? Through git log we can see all commits in a local repository and their commit messages. Here is the git log from our test repo:

$ git log
commit 7ba22bd8923180278deaf938785841b740d9eae4 (HEAD -> master)
Author: Kyle Walker <kywadev@gmail.com>
Date:   Mon Jan 3 14:34:34 2022 -0600

    initial commit, adding repo files

We have some new items here from Git, one of the largest being the long string you see: 7ba22bd8923180278deaf938785841b740d9eae4. This is what is called the hash or SHA of the commit. This is a unique hash for this repository to identify "that point in history". The commit message is for humans to be able to have some form of identifier for a specific commit, although unlike the commit hash, a commit message doesn't have to be unique. We've discussed a lot about commits and mentioned them being a pointer to a specific place in time, so let's move on to branches which are pointers themselves, but to a specific commit history.

Branches

As mentioned above Git branches as essentially a pointer to a specific commit. If all you ever do is use Git locally without ever working with a team or a remote repository, Git branches may not make a lot of sense at first. However, Git branches have some amazing benefits, but can be confusing if you aren't aware the current state of your local repository.

We see in our git status that our current branch is master. This happens to also be the default branch for most if not all Git repositories, but it doesn't have to be. You can actually set the default branch to be whatever you'd like it to be called. Some have chosen to go with "main" or "trunk" or whatever they feel best describes the "core" of their repository. In our git log command in the previous section, we see a single commit hash. Our master branch in this local repository is currently targeting that commit. Remember when we first did git status and it said this:

$ git status
On branch master

No commits yet

This message No commits yet is telling us that currently the master branch isn't targeting anything, but this is no longer the case since our first commit.

$ git status
On branch master
nothing to commit, working tree clean

What if by now we have a few customers on our platform with their Minecraft Servers and we are going to try to update our platform. In theory we would have a development server to test these items on. For the sake of argument, I mean both the virtual server and the newer Minecraft Server we would have a development version of. In our Ansible examples we had just customer servers in place with some global variables for all Minecraft Servers to use. If we had a development environment we could have unique vars for those dev servers, or we could have a dev branch in Git that is used to test these changes before they get merged into the master branch.

Note

In this particular example, Ansible on its own may not be the best example for using a dev branch as we run all of this Ansible by hand for managing these Minecraft Servers. If we were using something like Ansible Tower (now called Red Hat Ansible Automation Platform) or AWX which allows you to automate and schedule your Ansible playbooks, this would make more sense. Later in this series, we will be discussing different applications which may benefit more from unique branches.

Git Config

In the commits section we spoke briefly about the git config, but went into no details about what it is. When you issue your first git config --global command this will create a hidden file in your home directory called .gitconfig.

$ cat ~/.gitconfig
[user]
name = Kyle Walker
email = kywadev@gmail.com
[pull]
    rebase = false
[init]
    defaultBranch = master

The output above is my .gitconfig for my local machine. To setup your user (as mentioned above during the commit section) you can issue these commands to generate do this for you:

• git config --global user.name "Kyle Walker"
• git config --global user.email "kywadev@gmail.com"

There are many other git config options you may end up using, but having your user set to identify you as the author of commits is all most people go with. If you wish to review all configuration options, the official Git docs have some very detailed information: https://git-scm.com/docs/git-config

Using Git

We've discussed the major concepts and commands of working with a local Git repository, but before we move on I'd like to go over what a typical "workflow" is like for someone using Git. In our business we just got a new customer who needs a Minecraft Server. Let's see what this process would be like now that we use Git.

First we would create our Ansible files in the same fashion as was done before, we would update the inventory file to include this new Minecraft Server. We would also create the required variable files in group_vars and host_vars for this new customer. Once we have done that, it's time to run our Ansible playbook and then add these files to Git. For the sake of it being "just us" on this team we will use these files being added to Git as our process of "the work has been done". Let's see what our local repository looks like now:

$ git status
On branch master
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
        modified:   inventory/hosts

Untracked files:
  (use "git add <file>..." to include in what will be committed)
        inventory/group_vars/customer-2/
        inventory/host_vars/mineops-2.kywa.io/

no changes added to commit (use "git add" and/or "git commit -a")

As we can see, we now have a different type of git status, but we've seen all of this before when we did our first git add of ansible.cfg. We have Changes not staged for commit, this being the pre-existing and already tracked inventory hosts file, and we have more Untracked files which we added for our new customer.

Before we add any files to our local repository, let's take a look at a new Git command called git diff and see what the inventory hosts file shows us.

$ git diff inventory/hosts
diff --git a/inventory/hosts b/inventory/hosts
index 510d09b..181a8ea 100644
--- a/inventory/hosts
+++ b/inventory/hosts
@@ -1,9 +1,13 @@
 [minecraft:children]
 customer-0
 customer-1
+customer-2

 [customer-0]
 mineops-0.kywa.io

 [customer-1]
 mineops-1.kywa.io
+
+[customer-2]
+mineops-2.kywa.io

git diff shows us the additions (remember this from earlier during our first commit?) via the + sign on each line of the inventory hosts file. This is quite handy in case you were working on something, had to step away, lost your place and were unsure of where you left off. It has more uses later on as we will see in this guide.

Once you've created the new files to be tracked in Git, you would then add them via git add:

$ git add -A
$ git status
On branch master
Changes to be committed:
  (use "git restore --staged <file>..." to unstage)
        new file:   inventory/group_vars/customer-2/config.yml
        new file:   inventory/host_vars/mineops-2.kywa.io/config.yml
        new file:   inventory/host_vars/mineops-2.kywa.io/server-properites.yml
        modified:   inventory/hosts

We are now ready in theory to create another commit to this repository. This time, lets just use git commit with no -m flag being passed in. For my system I have an environment variable set called EDITOR=vim. If you do not want to alter your systems default editor via this environment variable, you can add something to your Git configuration file through this command git config --global core.editor "vim". The other alternative is an environment variable called GIT_EDITOR which can be set to any editor as with the EDITOR env var. When we run git commit, we are met with a screen that looks somewhat like this (note I am using vim as my EDITOR):

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# On branch master
# Changes to be committed:
#       new file:   inventory/group_vars/customer-2/config.yml
#       new file:   inventory/host_vars/mineops-2.kywa.io/config.yml
#       new file:   inventory/host_vars/mineops-2.kywa.io/server-properites.yml
#       modified:   inventory/hosts
#

Git gives us a clear idea of what it is wanting, a commit message, but with this we are able to more easily create multi-line commit messages for the sake of being explicit. This isn't required, but its something to be aware of if you grow your team and you require more detailed commit messages for some tools in the future such as git-chglog. As for the commit message in this format, you can create them like this:

Adding new customer
- customerID 98765 via ticket #00005
- deployed by Kyle

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# On branch master
# Changes to be committed:
#       new file:   inventory/group_vars/customer-2/config.yml
#       new file:   inventory/host_vars/mineops-2.kywa.io/config.yml
#       new file:   inventory/host_vars/mineops-2.kywa.io/server-properites.yml
#       modified:   inventory/hosts
#

Once you are satisfied with your commit message you can write and quit out of the editor (in vim this is :wq). And the output afterwards is the same as any commit, we can also check the git log and see our multi-line commit message:

$ git commit
[master 552b08d] Adding new customer - customerID 98765 via ticket #00005 - deployed by Kyle
 4 files changed, 9 insertions(+)
 create mode 100644 inventory/group_vars/customer-2/config.yml
 create mode 100644 inventory/host_vars/mineops-2.kywa.io/config.yml
 create mode 100644 inventory/host_vars/mineops-2.kywa.io/server-properites.yml
 
$ git log
commit 552b08d221fc9c6a1875f65d8bcfae7c1bf0aaa4 (HEAD -> master)
Author: Kyle Walker <kywadev@gmail.com>
Date:   Mon Jan 3 15:24:35 2022 -0600

    Adding new customer
    - customerID 98765 via ticket #00005
    - deployed by Kyle

commit 7ba22bd8923180278deaf938785841b740d9eae4
Author: Kyle Walker <kywadev@gmail.com>
Date:   Mon Jan 3 14:34:34 2022 -0600

    initial commit, adding repo files

As for using Git, there is really only one other item to address as it relates to a local repository and that is reverting commits with git revert. Depending on the scope and size of changes in the commit you wish to revert, you could either edit the files (if it was a small change) and create another commit, or you could use git revert if it was a large amount of files and changes. Please note that git revert can only be used to revert commits, but by doing so creates a new commit marking the revert. You will need to get the commit hash via git log or some other method and then you can run git revert 552b08d221fc9c6a1875f65d8bcfae7c1bf0aaa4. Let's see what that looks like and what happens when we run this:

$ git revert 552b08d221fc9c6a1875f65d8bcfae7c1bf0aaa4
Revert "Adding new customer"

This reverts commit 552b08d221fc9c6a1875f65d8bcfae7c1bf0aaa4.

# Please enter the commit message for your changes. Lines starting
# with '#' will be ignored, and an empty message aborts the commit.
#
# On branch master
# Changes to be committed:
#       deleted:    inventory/group_vars/customer-2/config.yml
#       deleted:    inventory/host_vars/mineops-2.kywa.io/config.yml
#       deleted:    inventory/host_vars/mineops-2.kywa.io/server-properites.yml
#       modified:   inventory/hosts
#

$ git revert 552b08d221fc9c6a1875f65d8bcfae7c1bf0aaa4
Removing inventory/host_vars/mineops-2.kywa.io/server-properites.yml
Removing inventory/host_vars/mineops-2.kywa.io/config.yml
Removing inventory/group_vars/customer-2/config.yml
[master c6f7b7b] Revert "Adding new customer"
 4 files changed, 9 deletions(-)
 delete mode 100644 inventory/group_vars/customer-2/config.yml
 delete mode 100644 inventory/host_vars/mineops-2.kywa.io/config.yml
 delete mode 100644 inventory/host_vars/mineops-2.kywa.io/server-properites.yml

$ git status
On branch master
nothing to commit, working tree clean

$ git log
commit c6f7b7b392a52edbbde4be26e8ed00c8487a9403 (HEAD -> master)
Author: Kyle Walker <kywadev@gmail.com>
Date:   Mon Jan 3 15:45:28 2022 -0600

    Revert "Adding new customer"

    This reverts commit 552b08d221fc9c6a1875f65d8bcfae7c1bf0aaa4.

commit 552b08d221fc9c6a1875f65d8bcfae7c1bf0aaa4
Author: Kyle Walker <kywadev@gmail.com>
Date:   Mon Jan 3 15:24:35 2022 -0600

    Adding new customer
    - customerID 98765 via ticket #00005
    - deployed by Kyle

commit 7ba22bd8923180278deaf938785841b740d9eae4
Author: Kyle Walker <kywadev@gmail.com>
Date:   Mon Jan 3 14:34:34 2022 -0600

    initial commit, adding repo files

Note we have a new commit showing the revert. If you do not wish to have revert commits in your git log (I would recommend that you do keep them there for the sake of history especially as you move to remote repositories and working with teams), you can use the git reset command which we are going to go over quickly below.

If you need to revert changes in your index to "rollback" changes you made to existing files you can use git reset --hard hash. This can potentially be destructive so you should probably use git checkout . to reset your index to HEAD (latest commit for your branch). Please note this only works if you haven't commited any changes locally yet.

That wraps up just about everything you would typically do with Git in a local repository. It is time to learn about remote repositories and then working with a team once we have a remote repository working.

Remote Repositories

So we've talked about Git a lot, but not once have I mentioned GitHub, GitLab, Bitbucket or any other of the big names in the Git repository world and that was on purpose. People often confuse GitHub and Git as being just different versions of the same thing and this couldn't be further from the truth.

Setting up a Remote Repository

Each of the big 3 source-code repository hosting services GitHub, GitLab and Bitbucket ultimately function as a remote repository. Setting up a new repository is roughly the same in each one, but we will not outline how to do that for each one. All that you really need is an account and then a name for the new repository. Here are links to each of the docs to create a remote repository:

• GitHub - https://docs.github.com/en/get-started/quickstart/create-a-repo
• GitLab - https://docs.gitlab.com/ee/user/project/working_with_projects.html#create-a-project
• BitBucket - https://support.atlassian.com/bitbucket-cloud/docs/create-a-git-repository/

When a new repository is set up, each of the big 3 will give you instructions on how to use the new remote repository. For this guide I will be using GitHub, but any repository hosting will work. The setup instructions it gives you outlines how to create a new repository from the command line to use this new remote repository. We do not need to do this as we have our own local repository already. Thankfully there are instructions for this use case as well. We will dive into this in the next section.

Adding a Remote Location to your Local Repository

Once you have created a new remote repository in GitHub/GitLab/BitBucket, you will need to let your local repository know about this remote location. Thankfully you get these instructions once you setup a remote repository:

$ git remote add origin git@github.com:KyWa/mineOps.git

There are other instructions given but these will be outlined in the following section. The only item to note about the above command is the name origin. This is the name used for the remote repository that your local repository can work with.

Working with a Remote Repository

In your local repo once you have run the above command to add a remote target we can now begin to work with it. First let's use git push to push our local repository to the remote.

$ git push
fatal: The current branch master has no upstream branch.
To push the current branch and set the remote as upstream, use

    git push --set-upstream origin master

$ git push --set-upstream origin master
Warning: Permanently added 'github.com' (ED25519) to the list of known hosts.
Enumerating objects: 54, done.
Counting objects: 100% (54/54), done.
Delta compression using up to 12 threads
Compressing objects: 100% (45/45), done.
Writing objects: 100% (54/54), 7.43 KiB | 1.86 MiB/s, done.
Total 54 (delta 6), reused 0 (delta 0), pack-reused 0
remote: Resolving deltas: 100% (6/6), done.
To github.com:KyWa/mineops-modular_ansible.git
 * [new branch]      master -> master
Branch 'master' set up to track remote branch 'master' from 'origin'.

Running git push on its own at the moment throws an error that the current branch master has no upstream branch. We can remedy this with the command that Git gives us (see how helpful Git can be?). Let's break down what we this command is doing.

• git push - pushing local changes to an upstream branch
• --set-upstream - starts the setup of targeting the upstream branch
• origin - the remote target used by the --set-upstream flag
• master - the name of the remote branch to push your local branch to

So we now have content in our remote repository, this is a huge step forward for our company. We no longer have our files being stored on our local machine and have Git tracking our files locally, with the big plus of being stored in GitHub (our remote repository).

Some of the other tasks you will do when working with a remote repository is getting changes from that remote repository. Now at the moment its just us in this company so there won't be much to get from our remote repository as we are the only ones doing any work. In the next section when we start "working with a team" we will dig more into pulling code from a remote repository after our "teammates" have made changes we need.

Working With a Team

Our company has grown and we need another team member. Once this new team member has gotten set up they will start working and adding code to our remote repository. We now need to get this code so we can have these changes locally to ensure we don't have any issues with our own changes. There are a few methods to do this with the more common one being git pull. This command will technically do 2 other commands a git fetch and a git merge. Fetch will "fetch" all the objects and references in "remote" if there are any, but will not be part of your local Git history just yet. If you were to run a git merge once you have this (and there were changes in the upstream branch matching your local branch) it will update your local history by "merging" each history together (local and remote). The more simpler way is git pull which will handle both of these for you and is the most common path used to do this.

A note about working with a team and pulling code down or pushing new code. There can be things known as a "Merge Conflict". These are not fun, but we will not being dealing with these and you may not either if you ensure you have a clean "working directory" prior to doing a push or pull. For more information around "Merge Conflicts" check out this article: https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/addressing-merge-conflicts/about-merge-conflicts

Now this is where more fun begins, but also where Git truly can shine if using a remote repository as outlined above. Why did I say its fun? Enter git blame. Despite its name, this doesn't send an email to your co-worker to tell them they did something wrong. What it does do however is show you all the lines of a file that Git tracks and who edited which line based on the current commit.

$ git blame ansible.cfg
^7ba22bd (Kyle Walker 2022-01-04 14:34:34 -0600 1) [defaults]
^7ba22bd (Kyle Walker 2022-01-04 14:34:34 -0600 2) roles_path = roles
^7ba22bd (Kyle Walker 2022-01-04 14:34:34 -0600 3) retry_files_save_path = /tmp

Now obviously there is more to working with a team through Git and git blame and git pull, but for our little company we have covered all we need for using Git. There isn't much more to discuss, but there will be more topics come up in later parts of this series.

Git Extra Credit

If you really want to get the most Git, there are tools you can use to make life easier and expand on the functionality of Git. I will list these below, but none are required to use Git.

• tig - A text-based repository browser that really helps "view" a repository
• git-chglog - CHANGELOG creator for your repository based on commits
• GitHub Desktop - GUI repository manager from GitHub

Moving On

In the next installment in the series we are going to move away from Ansible and traditional "servers" and start our journey to containers because our CEO heard the buzzword "containers" so we need to migrate to "containers". And if history has taught us anything, they are going to hear about another buzzword and we will probably have to migrate to that afterwards as well. See you in the next installment!

Series Links:

• Part 1 - Manual Minecraft Server Installation
• Part 2 - Automating with Ansible
• Part 3 - Keeping it Altogether with Git
• Part 4 - Containers Have Joined the Party
• Part 5 - Making Containers Highly Available
• Part 6 - ArgoCD to the Rescue

In Part 1 of this series we went over manually deploying a Minecraft Server as part of a fictitious business to show their journey of growth through DevOps. In this post we will go over automating all that was done in Part 1 through Ansible. We will also touch on some nice to haves through automation and see where we can make our lives easier for this company.

Installing Ansible

There are a few ways to install Ansible and all of those methods can be found in the Ansible docs: https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html

I tend to use pip to install it Ansible, but the package manager of your system will work as well. Here are some quick examples on installing Ansible

• python -m pip install --user ansible
• yum install -y ansible

Setting up our inventory

If you are familiar with Ansible or have read the primer I wrote in the supporting mineOps GitHub repo, you should be aware of what an Ansible Inventory is. A quick overview is just a file containing the servers you wish to manage with Ansible. For the moment, we are going to create this inventory file wherever we are on our "control node" most likely our laptop or workstation for this business.

[minecraft]
mineops-0.kywa.io

So in this inventory file we have our single Minecraft Server called mineops-0.kywa.io. For now our inventory is quite simple, but we will be expounding on this later in this article.

Automating the install

Let's look over what all we did during the manual install process and see about converting it to Ansible.

In order we:

• Installed Java 17 (or later)
• Downloaded the Minecraft Server jar file
• Accepted the EULA for Minecraft Server
• Ran the server

After that we did a few other things such as creating a dedicated directory, a dedicated user, creating a systemd unit to start/stop the server without using up a terminal and creating backups of the Minecraft Server. We will begin by just automating the install going forward from there. Each of the Ansible Modules used in the beginning here will have a link to their documentation in a comment above the task.

The "basic" Ansible playbook can be found in the supporting repo: https://github.com/KyWa/blogs/blob/master/mineOps/files/Ansible/basic_ansible/minecraft-install.yml

As you can see this is fairly straightforward and essentially what we did manually, but all in one Ansible Playbook. We could technically do all of this in a BASH script and it would work just fine, but the issue with BASH scripts comes scalability and extensibility. There are some sysadmins I know who would argue with me to the end of time that BASH scripts can automate with the best of Ansible, but we will see here in a little bit why we would want to avoid that and stick with something like Ansible.

We can give our Ansible playbook a run and see what we get. Again note I'm passing an argument to ansible-playbook to specify my ssh user. This can be done in a file called ansible.cfg and we will get to that later in this post:

ansible-playbook -i hosts minecraft-install.yml -e ansible_ssh_user=root

If all went according to plan (which it should have) Ansible will have completed all the tasks we gave it and a running Minecraft Server should exist at mineops-0.kywa.io:25565. Please note this server isn't publicly accessible and only reachable from inside my home lab so don't bother going to test it out because it will not be there.

We have a working Minecraft Server (again)! Even though we didn't create the playbook in the first part of this blog post, the effort to write this Ansible Playbook to do essentially what we already had through a BASH script was a little on the high side (as is most automation), but the power isn't in automating something you could do manually, the power is in how you can use and re-use that automation. In the next section we will expound upon our Ansible Playbook and break it out into more modular Ansible items such as a roles and inventory vars.

There are a few more things we would need to do for a customer, but we can move along to the more "modular" approach to Ansible and tackle these other needs. To summarize the "basic" Ansible approach we have a playbook, an inventory file and 2 Minecraft specific files we are copying over. All of this sits in a single directory and to make any lasting changes we will have to edit this file or copy it and run it the new playbook. In the next section we are going to break this all apart and spread out most of this into an Ansible Role.

Upgrading our Automation

Using the more "basic" Ansible approach we are able to do quite a bit and get us where we need to be in terms of deploying a Minecraft Server. Now let's say the customer needs to make some changes. Maybe they want the Message of the Day changed for the server, or want to disable/enable PVP. First let's convert our single playbook into a role. This isn't a going to initially grant us any new automation ability, however it does enable us to be able to move things around and fine tune our deployments (especially if growing to more than one customer).

Compare these two directories:

$ tree basic_ansible/
basic_ansible/
├── eula.txt
├── hosts
├── minecraft-install.yml
└── minecraft.service

0 directories, 4 files


$ tree modular_ansible/
modular_ansible/
├── ansible.cfg
├── inventory
│   ├── group_vars
│   │   ├── customer-0
│   │   │   ├── config.yml
│   │   │   └── server-properties.yml
│   │   └── minecraft
│   │       ├── config.yml
│   │       └── server-properties.yml
│   ├── host_vars
│   │   └── mineops-0.kywa.io
│   │       ├── config.yml
│   │       └── server-properites.yml
│   └── hosts
├── playbooks
│   ├── minecraft-backup.yml
│   ├── minecraft-config-change.yml
│   ├── minecraft-install.yml
│   ├── minecraft-restore.yml
│   └── minecraft-uninstall.yml
└── roles
    └── minecraft
        ├── files
        │   └── eula.txt
        ├── tasks
        │   ├── backup.yml
        │   ├── config-change.yml
        │   ├── main.yml
        │   ├── restore.yml
        │   ├── server-prep.yml
        │   ├── start-server.yml
        │   └── uninstall.yml
        └── templates
            ├── minecraft.service.j2
            └── server.properties.j2

12 directories, 23 files

As you can see there is much more going on in the "modular" section. There is one item to point out and that the files under host_vars/servername and group_vars/groupname do not matter. Their names and layout are purely for those who use Ansible to be able to identify which files contain which variables. Ultimately it is the same data as in the basic section, but more spread out and "modular". One of the largest changes is in the inventory section. Our previous example had a single inventory file with nothing special about it and if we attempted to add variables to it, we would quickly lose our ability to easily see what we had. Enter group and host vars. Breaking up variables per group and even per host can help us tackle multiple instances and multiple customers with ease.

$ cat inventory/hosts
[minecraft:children]
customer-0

[customer-0]
mineops-0.kywa.io


$ tree inventory/
inventory/
├── group_vars
│   ├── customer-0
│   │   └── config.yml # Contains Customer specific variables
│   └── minecraft
│       └── config.yml # Contains variables for ALL Minecraft instances
├── host_vars
│   └── mineops-0.kywa.io
│       └── config.yml # Contains Server specific variables

With our example inventory file and the comments of which config.yml file contains what, it should be apparent the benefits of using this model. Going forward with Ansible and having multiple customers, it may happen quickly where you end up with multiple variable files in your group/host var directories. This is expected and placing your variables in files related to what they are will help greatly as scale occurs. Obviously you can throw every variable into a config.yml, but that isn't very descriptive. Minecraft Server may not have many variable files, but in other projects where you would use Ansible, this can grow to some very large files and separating them out will greatly increase your sanity when hunting for them.

One such inventory variable file will be server-properites.yml and as you can probably guess, this will mirror the server.properties file that Minecraft Server creates. Remember when I mentioned things we would need to change for a customer? Well this is where we can do that (as this is really the only area they would want things changed in most circumstances). In the "modular" Ansible directory we can see I have converted the server.properites into a new file called server.properites.j2. Ansible utilizes the Jinja2 templating language and we can make use of it here. As with any variable you can see all of the properties values now equal an Ansible variable:

$ cat roles/minecraft/templates/server.properties.j2
enable-jmx-monitoring={{ enable_jmx_monitoring }}
rcon.port={{ rcon_port }}
gamemode={{ gamemode }}
enable-command-block={{ enable_command_block }}
enable-query={{ enable_query }}
level-name={{ level_name }}
motd={{ motd }}
query.port={{ minecraft_query_port }}
pvp={{ pvp }}
difficulty={{ difficulty }}
~~~~~OMITTED~~~~~

Each of these variables (outlined by their {{ name }} syntax) has a corresponding variable set in modular_ansible/inventory/group_vars/minecraft/server-properties.yml.

$ cat inventory/group_vars/minecraft/server-properties.yml
enable_jmx_monitoring: "false"
rcon_port: 25575
gamemode: survival
enable_command_block: "false"
enable_query: "false"
level_name: world
motd: A Minecraft Server
minecraft_query_port: 25565
pvp: "true"
difficulty: easy
~~~~~OMITTED~~~~~

Handling our variables in this way allows us to push out changes per customer as we will see in just a moment. As we saw in our breakdown earlier of the inventory file hosts up above we see that we have a group called: customer-0. This is a unique group that is part of the minecraft group and will inherit all variables in the inventory/group_vars/minecraft/ directory. However with the way that variable precedence works in Ansible, if we place customer specific variables in their groups_vars or host_vars they will be what is ultimately pushed out to the server. Here is an example with our "modular" Ansible:

$ grep -r motd inventory/
modular_ansible/inventory/group_vars/customer-0/server-properties.yml:motd: "Welcome to customer-0 Minecraft Servers!"
modular_ansible/inventory/group_vars/minecraft/server-properties.yml:motd: "A Minecraft Server"

$ cat modular_ansible/inventory/hosts
[minecraft:children]
customer-0

[customer-0]
mineops-0.kywa.io

In the above output we have 2 motd variables set, only one of them will be used. From what we see in our inventory file, the child group customer-0 will take precedence over the "parent" minecraft group and the server.properties file will have the motd of: "Welcome to customer-0 Minecraft Servers!". This will remain the same for all Ansible you utilize so there is not much more need for examples around this. If you wish to know more about Ansible and variables always check the docs: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html. Another item of note around variables is their names, with some of them being special and reserved inside of Ansible. There is a doc outlining each of them and their use: https://docs.ansible.com/ansible/latest/reference_appendices/special_variables.html

Operating a Minecraft Server

I've created a few tasks in the role minecraft to handle a few nice things for us and our customers. We have a backup, restore and config-change task as well as an unfortunate uninstall if a customer decides to leave us. We can demo these out, but they are fairly straightforward. Let's at least look at the backup task to get an idea of what we are trying to accomplish.

Backing up a Minecraft Server

- name: Mount the Minecraft Server Backup Share
  mount:
    path: /mnt
    src: "{{ minecraft_backup_share }}"
    fstype: nfs
    state: mounted

- name: Backup the systemd servie
  copy:
    remote_src: yes
    src: /usr/lib/systemd/system/minecraft.service
    dest: /mnt/minecraft.service

- name: Stop the Minecraft Server Service
  systemd:
    name: minecraft
    state: stopped

- name: Archive and Backup the Minecraft Server Directory
  archive:
    path: "{{ minecraft_directory }}/*"
    dest: "/mnt/minecraft-backup.tgz"
    format: gz
    mode: 0644

- name: Start the Minecraft Server Service
  systemd:
    name: minecraft
    state: started

- name: Unmount the Minecraft Server Backup Share
  mount:
    path: /mnt
    state: unmounted

In this particular instance we are using an external NFS server to hold our backup data, but this could technically also be Google Drive, a Windows SMB server or anywhere where files are accepted. I will be using NFS as its the easiest to setup and get working in a Linux environment. The task above mounts a share (NFS) stops Minecraft Server, copies the data over to the share along with its systemd service, then restarts Minecraft Server and finally removes the backup share to create an "air gap" of sorts. This last step is important to not leave your backup shares mounted to any system where you hold your backups dear to you. Most Ransomware goes through all mounted directories and disks in Linux and Windows. Keeping your backups disconnected from your production servers will save you heartache, time and money down the road.

Reconfiguring a Minecraft Server

Now that we have a customer with a running server and we've automated its install, they want to make a change. What do we need to change and where do we keep that data? As mentioned previously, most changes will take place in the server.properties file which we have templatized and put on the each server during install time. If a customer wants to make changes away from the default configuration we provide, the customers server should have their own server-properties.yml updated in the host_vars section of the inventory. Let's take a look at this example, the customer wishes to remove PVP from their world because they want to invite a bunch of people, but don't want to have to worry about their friends losing their stuff. Here is what we would add:

$ cat inventory/host_vars/mineops-0.kywa.io/server-properites.yml
pvp: "false"

In the "basic" Ansible to push this change out, we would have to do quite a bit to go and edit the server.properties file on the Minecraft Server and restart the server. Here is what it would look to complete this in the "basic" Ansible fashion:

• Add a task to run the lineinfile module against the server.properties file to change the requested property
• Add another task to the playbook to stop the server and another to start it back

If we add this to our existing playbook we won't disrupt anything, but a lot of steps will have to be gone through again (even if they are already completed) just to make this one change. By using the "modular" Ansible approach via a role we can have a task file for a start of the server such as start-server.yml. This task file can be called from other tasks in a role via the include_tasks: mytask.yml parameter making it far easier to extend your roles with other tasks you have created. This makes it far easier to add in tasks without having your primary playbook grow to unnecessary lengths.

For the sake of reconfiguring a Minecraft Server, we can use the task in the "modular" Ansible role called config-change.yml. Let's take a look at this task:

---
- name: Create ResourcePack directory
  file:
    path: "{{ minecraft_directory }}/resourcepack"
    owner: "{{ minecraft_user }}"
    group: "{{ minecraft_group }}"
    state: directory
    mode: 0755
  when: resource_pack is defined

- name: Create server.properties from template
  template:
    src: server.properties.j2
    dest: "/tmp/server.properties"

- name: Create server.properties from template
  copy:
    remote_src: true
    src: /tmp/server.properties
    dest: "{{ minecraft_directory }}/server.properties"

- name: Stop the Minecraft Server
  systemd:
    name: minecraft
    state: stopped

- name: Starting up Minecraft
  include_tasks: start-server.yml

This task will create a directory for resource packs if this config has been defined, create a new server.properties file from our template, stop the Minecraft Server, put the new server.properties file in place and issue the task to start the Minecraft Server up. The last section as you can see is calling the include_tasks we discussed earlier. This particular task is the same task we use during our main install in the "modular" Ansible configuration.

Adding a new Customer

As a business we've grown and through word of mouth, we have another customer at last! Using our "modular" Ansible we can add in their new group/host vars and update our inventory file and we can provision their new instance. Let's look at the updated inventory file and inventory vars:

$ cat inventory/hosts
[minecraft:children]
customer-0
customer-1

[customer-0]
mineops-0.kywa.io

[customer-1]
mineops-1.kywa.io

$ tree inventory/
inventory/
├── group_vars
│   ├── customer-0
│   │   ├── config.yml
│   │   └── server-properties.yml
│   ├── customer-1
│   │   ├── config.yml
│   │   └── server-properties.yml
│   └── minecraft
│       ├── config.yml
│       └── server-properties.yml
├── host_vars
│   ├── mineops-0.kywa.io
│   │   ├── config.yml
│   │   └── server-properites.yml
│   └── mineops-1.kywa.io
│       ├── config.yml
│       └── server-properites.yml
└── hosts

7 directories, 11 files

We've created our new variables and updated our inventory file, how do we deploy this new customer server? There are a few ways with the first of which being to just run the playbooks/minecraft-install.yml again, but this will attempt to do this for all servers in the inventory file. This isn't going to cause any issues because we have built our Ansible to not do any damaging or disruptive tasks. Ansible itself only "makes changes" to items that require changes. For instance the Install Java task which uses the dnf module will check to see if java-17-openjdk is installed and only install it IF it isn't there already. Now there is a caveat to this, the shell and cmd modules just run whatever commands you pass them so this is just something to be aware of.

Due to the caveat above regarding the shell and cmd modules in our server-prep.yml task file we have a stat task which will verify the existence of the server.jar. This will prevent the task downloading a new server.jar and ultimately ensure we do not replace it with a newer or different version.

To deploy just the new customers server we can run our install playbook with a flag at runtime called --limit. This limits the playbook to just the group or host you specify. So to deploy the new customers server, all we would have to do is run ansible-playbook like so:

ansible-playbook -i inventory/hosts playbooks/minecraft-install.yml --limit mineops-1.kywa.io

You can use the host or the group name inside of --limit, but for making changes to an entire customer's group of servers, you would most likely (depending on the change) run the --limit against their group as opposed to a single host.

Storing our Ansible files

By this point we have our business running with our customers happy and all of these Ansible files (hopefully the "modular" kind), but what happens to these files if your computer crashes? Where are these files being stored? What if the team grows beyond you and you have other people updating these config files? Do you use DropBox or Google Drive and share out these files? You could do that and many businesses get away with this, but there is a better way and that way is through a Version Control System.

There are a few VCS or Source Control systems out there, but there really is only one that people use across the board and that is called git. Git has pretty much taken over every workflow out there and is even used for greater deeds than just versioning source code. This however will be tackled in the next blog post. See you for the next one!

Series Links:

• Part 1 - Manual Minecraft Server Installation
• Part 2 - Automating with Ansible
• Part 3 - Keeping it Altogether with Git
• Part 4 - Containers Have Joined the Party
• Part 5 - Making Containers Highly Available
• Part 6 - ArgoCD to the Rescue

As was discussed in the Introduction to this series, we will be focusing on deploying Minecraft Servers as part of a fictitious business. This business needs to get up and going with their first customer, so we need to get at least one Minecraft Server up and running. Let's get going!

NOTE:

Throughout this series we will be focusing our efforts on RPM based Operating Systems (RHEL, CentOS, Fedora etc...). Almost all of the examples will carry over with a few exceptions to Debian based Operating Systems (Debian, Ubuntu etc...) and we will call out those differences where they come up.

Another item to note is this guide will not outline the creation of any Virtual Machines or the use of any cloud providers. This series will use oVirt for the virtual machines used, but will not come into play. Note that Ansible can be used to provision Virtual Machines in whichever platform you happen to utilize, but will not be discussed in this series.

One final item of note is that "sudo" will not be used throughout this guide. All commands will assume you have root privileges on the servers.

Minecraft Server Prerequisites

CPU and Memory resources will be low for this series, but in a real world deployment you would want to ensure you have ample resources for each of the servers being used. 2 vCPU and 4GB of Ram per server will be enough for the purposes of this series.

Outside of resources one of the primary and nearly only requirement for running a Minecraft Server is java. There are a few ways to do this, but the easiest of which is to use the package manager of your system. For RPM based operating systems this can be done with a simple command:

dnf install -y java-17-openjdk

Java 17 is installed as it is the "oldest" version compatible with Minecraft Server.

Acquiring Minecraft Server

To run a Minecraft Server you will need to obtain the server.jar from Minecraft.net. You can obtain the URL for the file by copying the download link here. You can either obtain the link and download it directly from the "to be" Minecraft Server (using curl or wget) or you can download it to your local machine and copy it over via scp, rsync or some other method.

In the spirit of automating and making life easier overall, we have a much quicker path. The server will need jq installed and can be handled in a simple one-liner (if you already have jq installed on your server you can skip this step):

curl -o /usr/bin/jq -sL https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 && chmod +x /usr/bin/jq

This will place jq in your /usr/bin/ which should be in the path, but this now enables us to run and grab the latest Minecraft Server .jar file. Here is a handy script to grab the latest release of Minecraft Server:

curl -o server.jar $(curl `curl -sL https://launchermeta.mojang.com/mc/game/version_manifest.json | jq -r '.latest.release as $release | .versions[] | select (.id == $release) | .url'` | jq -r '.downloads.server.url')

The above script will reach out to Mojang servers and look through the latest releases and grab whichever happens to be the latest.

If you wish to obtain a specific version of Minecraft Server (which would need to for some customers) you can run the same script, but with a slightly different lookup. Notice we have removed early parts of the first jq query and turned $release into "1.18.1" where "1.18.1" is the specific version you are requiring:

curl -o server.jar $(curl `curl -sL https://launchermeta.mojang.com/mc/game/version_manifest.json | jq -r '.versions[] | select (.id == "1.18.1") | .url'` | jq -r '.downloads.server.url')

Starting Minecraft Server

We are just about to spin up our first Minecraft Server, but there is only one item that will block us from starting. If you attempt to run Minecraft Server without accepting the EULA, you will get a failure similar to what you see below:

java -jar server.jar
<OMITTED OUTPUT>
[22:59:42] [ServerMain/ERROR]: Failed to load properties from file: server.properties
[22:59:42] [ServerMain/WARN]: Failed to load eula.txt
[22:59:42] [ServerMain/INFO]: You need to agree to the EULA in order to run the server. Go to eula.txt for more info.

However you can prevent this message by accepting the EULA with another simple one-liner:

echo "eula=true" > eula.txt

Once you have "accepted" the EULA, Minecraft Server will be able to start, thus getting us one step closer to having customers paying for this service. Let's give it a go. Fair warning depending on the server's resources, the initial start-up may take a minute or more.

$ java -jar server.jar
Starting net.minecraft.server.Main
[23:03:08] [ServerMain/INFO]: Environment: authHost='https://authserver.mojang.com', accountsHost='https://api.mojang.com', sessionHost='https://sessionserver.mojang.com', servicesHost='https://api.minecraftservices.com', name='PROD'
[23:03:09] [ServerMain/INFO]: Reloading ResourceManager: Default
[23:03:10] [Worker-Main-2/INFO]: Loaded 7 recipes
[23:03:11] [Worker-Main-2/INFO]: Loaded 1141 advancements
[23:03:13] [Server thread/INFO]: Starting minecraft server version 1.18.1
[23:03:13] [Server thread/INFO]: Loading properties
[23:03:13] [Server thread/INFO]: Default game type: SURVIVAL
[23:03:13] [Server thread/INFO]: Generating keypair
[23:03:14] [Server thread/INFO]: Starting Minecraft server on *:25565
[23:03:14] [Server thread/INFO]: Using epoll channel type
[23:03:14] [Server thread/INFO]: Preparing level "world"
[23:03:22] [Server thread/INFO]: Preparing start region for dimension minecraft:overworld
[23:03:24] [Worker-Main-2/INFO]: Preparing spawn area: 0%
<OMITTED OUTPUT>
[23:04:27] [Worker-Main-2/INFO]: Preparing spawn area: 84%
[23:04:27] [Worker-Main-2/INFO]: Preparing spawn area: 84%
[23:04:31] [Server thread/INFO]: Time elapsed: 68584 ms
[23:04:31] [Server thread/INFO]: Done (77.021s)! For help, type "help"

Verifying Connectivity to Minecraft Server

At this point we can try to connect to our newly deployed Minecraft Server. If you are familiar with Minecraft, you will know to click "Multiplayer" and then for our tests click "Direct Connect". You will need to input the IP address of your server and the default port for Minecraft Server which is 25565. This will attempt a connection to your Minecraft Server.

If you are unsure of the IP address of your Minecraft Server you can run ip address on your system and look for the primary NIC:

$ ip address
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    link/ether 56:6f:0d:9a:00:00 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.50/24 brd 10.0.0.255 scope global noprefixroute enp1s0
       valid_lft forever preferred_lft forever
    inet6 fe80::c0ea:7070:3ce9:7c0f/64 scope link noprefixroute
       valid_lft forever preferred_lft forever

We may run into one more blocker before the Minecraft Server is operational and that is a firewall blocking the port.

If you attempt to connect to your server by putting into the "Server Address:" field the ip-address:port of the server you deployed Minecraft Server to, you may run into something like you see in the video "Connection Refused". To prevent this from being an issue, we will need to open up a port in the firewall for Minecraft.

firewall-cmd --add-port=25565/tcp

Please note the above command is not persistent and will not survive a server reboot. To have this persist your command will look slightly different:

firewall-cmd -add-port=25565/tcp --permanent && firewall-cmd --reload

The default port for Minecraft is 25565 and can be changed in a file we are about to look at (before runtime). With the firewall opened and the server running, we should be able to connect to our newly deployed Minecraft Server.

Tuning Minecraft Server

At this point we have a running Minecraft Server, but what is granting configuration to it? If we hit CTRL+C the server will exit, but we can take a look at the files that now exist in our directory. There are a few files of note, but the rest can be reviewed at the Minecraft Wiki. For the purposes of configuring this Minecraft Server for our customer we will focus on server.properties, but as you can see there are many configuration files that now exist in our directory:

$ ls -lhF
total 45M
-rw-r--r--.  1 root root    2 Dec 28 23:08 banned-ips.json
-rw-r--r--.  1 root root    2 Dec 28 23:08 banned-players.json
-rw-r--r--.  1 root root   10 Dec 28 23:03 eula.txt
drwxr-xr-x.  8 root root   77 Dec 28 22:59 libraries/
drwxr-xr-x.  2 root root   78 Dec 28 23:08 logs/
-rw-r--r--.  1 root root    2 Dec 28 23:08 ops.json
-rw-r--r--.  1 root root  45M Dec 28 22:59 server.jar
-rw-r--r--.  1 root root 1.1K Dec 28 23:08 server.properties
-rw-r--r--.  1 root root  109 Dec 28 23:12 usercache.json
drwxr-xr-x.  3 root root   20 Dec 28 22:59 versions/
-rw-r--r--.  1 root root    2 Dec 28 23:03 whitelist.json
drwxr-xr-x. 12 root root 4.0K Dec 28 23:15 world/

Inside server.properties we have quite a few tunable variables to look into. Some of these include the port in which Minecraft Server runs on, or whether or not to enable PVP (Player Versus Player actions). Each of these settings would be up to the customer to decide which to use, but we will need to tune these to meet customer demands.

Operating a Minecraft Server

Now that we have a running server and are able to make changes to its configuration, how do we keep it running? What about backing up the data for the customer in case there is an issue either from their side or yours? We will quickly focus on these items here.

For keeping a server running and having it run without taking up a shell/terminal, we will need to have a system service run it for us. There are a few things to consider to ensure we have a clean running environment. Let's outline those below:

• A directory dedicated to this server
• A user to be the "owner" of the server
• A systemd unit file to manage the state of the server

Some of these can be handled quite easily. For the dedicated directory we can create it via mkdir -p /opt/minecraft as its a logical place and out of the normal paths. If we are going to deploy multiple servers for customers, they will each need their own unique location. A unique directory should be created for each server using their customer ID or email address. If a customer has multiple servers, you could create a directory with a unique world name (customer choice) like so: mkdir -p /opt/myfunworld-00012345.

For the user to be the "manager" of the Minecraft Server we can just run useradd minecraft. However if we are going to have multiple customer servers running, it may be more logical to have a username that is inline with the customer. This would be most likely their customer ID or email address of the customer. For now we will just use minecraft.

The final piece would be the systemd service unit to keep the server running for us without having a terminal dedicated to running java -jar server.jar. Let's create a systemd unit file.

cat << EOF >> minecraft.service
[Unit]
Description=start and stop the minecraft-server

[Service]
WorkingDirectory=/opt/minecraft/

User=minecraft
Group=minecraft
Restart=on-failure
RestartSec=20 5

ExecStart=/usr/bin/java -Xms512M -Xmx3048M -jar server.jar nogui

[Install]
WantedBy=multi-user.target
EOF

If you notice the line ExecStart you will see a similar command to what we used to run the server the first time, however this command has a few new flags included. -Xms and -Xmx are the minimum and maximum amounts of memory the JVM can use while running Minecraft. The nogui flag tells Java to not run a Java GUI for the application. You may have noticed (if your server has a GUI) when you initially ran the commands for java -jar server.jar a window pop-up showing logs and a possibly resource utilization. This isn't something we want running as its wasting resources for what needs to be used for our Minecraft Server.

Now lets get this systemd unit file into place and enable it.

cp minecraft.service /usr/lib/systemd/system/ && systemctl enable --now minecraft

So something that will need to be done from time to time is to back-up this server and its configurations you will need to copy the contents of /opt/minecraft to some other location (ideally not connected to the server). We will not be going over a script to accomplish this because it will be very different for each environment. However the principal is the same, copy the data somewhere else except to cleanly grab the data, you will need to stop the Minecraft Server first.

Outcome

After all this effort to get one server going, what about the next customer? Quite a bit was done just to get a single server running and if we had to do this for multiple customers or multiple customers with multiple servers, it would become quite chaotic very quickly. What if all of these servers had multiple configuration changes? How would we manage and maintain all of them?

The next two items we are going to look at in the next few blog posts are Ansible for automating all of the deploys and Git to keep it all in line.

Series Links:

• Part 1 - Manual Minecraft Server Installation
• Part 2 - Automating with Ansible
• Part 3 - Keeping it Altogether with Git
• Part 4 - Containers Have Joined the Party
• Part 5 - Making Containers Highly Available
• Part 6 - ArgoCD to the Rescue

Hardware

Currently my homelab is on revision 5428 which consists of a single "server". This server is an AMD Ryzen 2700X with 64GB of RAM backed by a 1TB NVMe SSD. The storage for my environment is handled by a Synology DS1618+ filled with 16TB Seagate Exos drives. This nets me ~43TB usable and currently consuming 20TB of that (I run a large media server). I also have a single 2TB RAID1 SSD array for VM storage over NFS which is currently served over 1Gbps, but will be upgrading to 10Gbps soon. I already have the 10GB card for the Synology, I just need to acquire a 10GB card for the server (or servers if I upgrade anytime soon).

For networking, I have a dumb 24-port NetGear switch and a Synology RT2600AC router. This router is pretty dope and allows for all the functionality I need which is really just a DNS server I don't have to install myself. It is also managing DHCP for the network and gives pretty decent data visualization.

Storage

As mentioned above, I have a Synology DS1618+ unit. Prior to purchasing an actual NAS, I was using MDADM to manage a software raid and manually configured all of the shares I needed. Later on when I started storing our family photos and video, it became a pain to deal with Samba to share out so my Wife's Windows laptop could access it. For me in the long run, it was just easier to get a unit that did all of this for me (and had the ability to do a little bit more if needed). So far it has worked out amazingly and I wouldn't go back to managing my own storage without having a Synology.

This unit has 4 RJ45 Ports that can all be configured if required and has expandability for either a 10Gbe NIC or I believe an SSD NVMe expansion drive (I chose the 10Gbe route).

Virtualization

For virtualization the single server has oVirt installed (Red Hat Virtualization) and runs as a single host (with the hosted-engine running locally). The configuration is a single node cluster in a single datacenter. This presents some challenges I will outline later on.

Ovirt has a pretty awesome UI and functions quite well as far as virtualization platforms go, especially with it being open source. Currently here are the VMs that live on this node:

• hosted-engine (Virtualization Manager)
• bastion (Proxy and only VM exposed to the internet)
• Plex
• Minecraft
• OKD Nodes (3 nodes in total)
• Services (just a VM for testing things)

Containerization

Continuing on with the usage of open source projects, I run my own OpenShift Origin cluster (Red Hat OpenShift 3.11). For those that don't know or have been living under a rock, Kubernetes is a container orchestration system and has more or less gone from buzzword to the standard for application platforms. OpenShift is a distribution of Kubernetes and has quite a few handy add-ons built in. Some of these may not make sense to small groups or solo developers, but the features that OpenShift provides for an Enterprise (or any business looking to have some real features) cannot be matched.

OpenShift

First and foremost, OpenShift puts container security at the top of everything it does. Containers are somewhat secure by just existing, but can easily be exploited if just running Docker or a more vanilla Kubernetes platform. OpenShift enforces by default that containers/pods cannot run as root. This is already a leap ahead of every Kubernetes deployment in the wild. It also brings some challenges to newcomers to the platform as when most people get started, they find a tutorial on Kubernetes and just apply the code. They will be shocked to find that most tutorials out there assume that your container will run as root (although this is getting better as time goes on). There are other nice things in security, but I will leave those for another post.

Another neat item built in is around image registries and a build system. For most developers, you wouldn't think of having a build system as being an important item in Kubernetes since you will most likely have Docker installed locally. What if you work in an environment where you can't install Docker locally? Or don't have the option of a VM running Docker to do a build for you? That is where the build system comes in for OpenShift. It exists and allows you to build container images from source, Dockerfile or other means and then push them into the built in container registry built into OpenShift. You do not need to worry about having to spin up your own image registry when OpenShift just gives you one. Obviously for multiple clusters this becomes its own interesting problem and a central registry (Quay.io maybe?) would be more suitable. But for the sake of a homelab, this is more than sufficient.

Applications

In my lab, OpenShift Origin is used for me to test various tools on a live Kubernetes cluster. I have attempted using Docker Desktop on both Mac and Windows, but prefer to have a real multi-node cluster as it just sits a little closer to what is being done in the wild. Some of the projects I have running here are outlined below:

• ArgoCD
• Jenkins
• MongoDB
• Ghost blog (not this one you are reading)
• MySQL
• VS CodeServer (super cool project https://github.com/cdr/code-server)

Not all of these are in "production" use, but I have them around for testing and experimenting.

Challenges

One of the biggest issues I have is the current state of the single oVirt node. If there is an issue with the node, which rarely happens but can, the hosted-engine goes down with it. Getting this started back up due to it running on "local" storage, requires some finesse with Linux services starting up. I have a script to do this for me, but the beginning was quite troublesome.

The other issue faced is just lack of resources. I understand that there are devs and others out there who would kill to have 64GB on their main machine, let alone on an entirely independent server. However for me, I hit resource limits quite often (mostly memory related) and am unable to deploy the latest version of OKD the way I would like. Hopefully soon when chips and things become more available, this can be rectified in homelab v2.0.

Conclusion

This is the current iteration of my homelab with more to come in the near future as nature heals and chips become available. The next iteration will be 10Gbe with x2 servers for the oVirt cluster so I can utilize failover and not jump through hoops when the hosted engine fails. Please feel free to ask questions or links for any of the items/software listed here.

This blog will be focusing on a few different things. I will try to keep this blog as tech focused as possible. As I have grown in tech I have found the value that good blogs and articles provide for those learning or looking to grow.

Technology

The first tech related article will be over my homelab and how I use it. From then on out it will focus on most things around Kubernetes or OpenShift and tooling as it relates to those platforms.

Automotive

I will also have some articles posted around my primary hobby, cars. I will be documenting my build and usage of my main car and the side project I have. I no longer have my Camaro, but here is a video of it at Houston Raceway Park back in 2020:

Photography

I am no photographer, but I do have an interest in it and a great interest in drones. Recently I purchased a DJI Mini 2 and have grown to love the freedom they provide and the ease in which you can get some truly amazing shots. This video is a quick flight from my house to the local lake in my neighborhood.

I will do my best to post weekly, and look forward to hearing feedback as we move forward.