Building and deploying PlayPen packages with GitLab CI

Starting on December, I decided that I was tired of manually deploying PlayPen (a server management and load balancing framework) packages to a testing environment and later to production all by hand, and as so, I dove into the world of automation. Luckily, we were using GitLab, which is known for its amazing Continuous Integration, Delivery and Deployment capabilities integrating all the automation technologies you can think of. Our project is based on Java plugins which integrate together with the help of a coordination system running under one PlayPen instance. All our projects are built in Java and use Maven, so the first step was to setup the automated compilation of packages:

This is fairly easy, as there are lots of prebuilt Docker images which contain Maven. Create a .gitlab-ci.yml file on your project directory and add the following:

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
  MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"

cache:
  paths:
  - .m2/repository

maven-build:
  image: maven:3.3.9-jdk-8
  stage: build
  script:
  - "mvn $MAVEN_CLI_OPTS package"
  artifacts:
    paths:
    - target/*.jar
    expire_in: 1 day

What this does is create a maven-build job which runs on the build stage (GitLab CI has three predefined stages by default: build, test and deploy), will package our Java project and will keep the produced artifacts for one day on GitLab (which you can download from the CI/CD panel). Now that we’ve built our project we need to package our artifacts and other assets which are required for the project to work with playpen-p3. Now, here’s where the magic part happens, we’re going to create our own Docker image which will package and deploy the code for us!

Docker image creation

PlayPen didn’t have a Docker image so I decided to create one for you, you can check it out on the Docker Hub registry. Now that I’ve shamefully promoted my own creation, let’s create a Dockerfile in a new directory that will contain our build scripts. You should also create a separate GitLab repo (which optionally needs Git LFS support as we will be storing huge files in it) that will contain all your assets that need to be packaged with the Maven artifacts produced in the previous phase.

FROM hugmanrique/playpen:latest
LABEL version="1.0"\b
LABEL author="Your Name <name@organization.tld>"

ENV LFS_VERSION 1.0.2
ENV GITLAB_HOST gitlab.example.com

# Install git
RUN apk add --update bash git openssh
RUN apk add --update --virtual build-dependencies curl && \
  curl -sLO https://github.com/github/git-lfs/releases/download/v${LFS_VERSION}/git-lfs-linux-amd64-${LFS_VERSION}.tar.gz && \
  tar xzf /git-lfs-linux-amd64-${LFS_VERSION}.tar.gz -C / && \
  mv /git-lfs-${LFS_VERSION}/git-lfs /usr/local/bin/ && \
  git-lfs init && \
  apk del build-dependencies && \
  rm -rf /git-lfs-${LFS_VERSION} && \
  rm -rf /git-lfs-linux-amd64-${LFS_VERSION}.tar.gz && \
  rm -rf /var/cache/apk/*

What these first lines do is install bash, git, openssh and git-lfs as they don’t come installed by default on the Alpine distro which is the distro the PlayPen image is based on as you can check on Microbadger (playpen -> openjdk -> alpine). Next, we will create a SSH key pair that will be used as a read-only deploy key to clone and update the GitLab project that contains all our assets. First, we will need to create the SSH key pair (I’m assuming you’re using Linux, macOS or Git Bash on Windows):

ssh-keygen -t rsa -C "playpenDeployer@example.com" -b 4096

This command will ask you where you want to save your key and if you want to set a password, which we aren’t going to use as the SSH key usage is going to be automated. Once you have it, move them to your Dockerfile directory and create a .dockerignore file with the id_rsa.pub key as we only need the private key on the Docker image. Note: Your Docker image will contain a private SSH key, which means you shouldn’t share it, and that’s the main reason why we are giving it limited read-only access to a specific project by using the deploy keys GitLab feature. Talking about deploy keys, let’s set them up!

First, access your assets project, hover over settings and click on “Repository”:

GitLab project sidebar

Now, expand the “Deploy Keys” panel and paste the SSH public key we generated on the previous step. We don’t need to enable write access as the Docker image is just going to clone and pull contents from this repo. Finally click on “Add key” and we’re done! We’re now going to return to our Dockerfile, enable the SSH agent, clone the repo and add a git alias (because git pull can be harmful in some cases as it can modify our working directory in unpredictable ways + we only want to fast-forward as the Docker image will become outdated with each commit we make to our assets repo):

# Setup ssh to get access to the packages repo on GitLab
ADD the_generated_private_key /root/.ssh/id_rsa
RUN \\
  chmod 600 /root/.ssh/id_rsa && \
  eval $(ssh-agent) && \
  echo -e "Host ${GITLAB_HOST}\\n\\tStrictHostKeyChecking no\\n" >> /etc/ssh/ssh_config && \
  ssh-add /root/.ssh/id_rsa

# Create safe git update alias
RUN git config --global alias.up '!git remote update -p; git merge -q --ff-only @{u}'

# Clone assets project (change master by your branch)
RUN git clone -b master git@${GITLAB_HOST}:/playpen-packages.git /home/packages

First, we will add the generated SSH private key to the image, initialize the ssh-agent and add a no StrictHostKeyChecking rule which we need because it’s the first time we will connect to GITLAB_HOST and there’s no entry for it in known_hosts. Finally, we add the git alias and clone the repo which means our Docker image will now contain all the files in the last commit. We’re almost done with the Dockerfile, but we still need to add our own PlayPen build script, a config file for PlayPen logging and set some permissions because GitLab CI doesn’t run as root by default and, as so, doesn’t have access to some directories we’ve created:

# Add main script
COPY start.sh /home/start.sh
COPY build-log.xml /home/playpen/

# GitLab CI doesn't have access to /home
RUN chmod -R 700 /home/

Where build-log.xml is a log4j config file. Here’s mine, but you can also copy the default logging-local.xml file created when you install PlayPen on your machine:

<?xml version="1.0" encoding="UTF-8" ?>
<Configuration>
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout>
                <Pattern>%-5p %c{1}: %m%n</Pattern>
            </PatternLayout>
        </Console>
        <File name="file" filename="playpen.log">
            <PatternLayout>
                <Pattern>%d{yyyy-MM-dd HH:mm:ss} %-5p %c: %m%n</Pattern>
            </PatternLayout>
        </File>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="console" />
            <AppenderRef ref="file" />
        </Root>
        <Logger name="org.zeroturnaround.zip.ZipUtil" level="warn">
        </Logger>
    </Loggers>
</Configuration>

Notice how we copied a start.sh file that we haven’t created yet? This is the code that the GitLab CI runner will execute everytime a commit is pushed to your project’s repo:

#!/bin/bash

# Exit automatically if an error occurs
set -e

if ! { [ "$PP_UUID" ] && [ "$PP_KEY" ] && [ "$PP_IP" ] && [ "$PP_PORT" ] ; }; then
  echo "Some PlayPen variables are missing. Check the Secret variables Settings section on GitLab"
  exit 1
fi

# Update the assets repo from GitLab
cd /home/packages

# Checkout the needed branch (not really needed, but just in case), remove local changes created by previous builds and "pull" the changes from the repo
git checkout master && git reset --hard && git up
echo "Updated assets"

package=$PLAYPEN_NAME
package_dir=/home/packages/$package

# The PlayPen Docker image stores the JAR in /home/playpen
playpen=/home/playpen/PlayPen-1.1.2.jar
playpen_args="-Dlog4j.configurationFile=build-log.xml -jar $playpen"
output_dir=/home/output
package_output_dir=$package_dir/$PACKAGE_PATH

# Go back to the produced artifacts dir
cd -

# Move GitLab CI artifact to PlayPen package dir
# Create output dir (if missing)
mkdir -p $package_output_dir

# Grab all the JAR artifacts
for file in `ls -a target/*.jar` ; do
  # Maven produces "shaded" and "original" junk jars we don't want to move
  if [[ $file != *"shaded"* && $file != *"original"* ]] ; then
    echo "Moving $file artifact"
    mv $file $package_output_dir
  fi
done

# Create default PlayPen config files
cp /home/playpen/build-log.xml .
java $playpen_args ignore

echo "Packaging $package..."
mkdir -p $output_dir

# Package all the Maven artifacts + assets into a .p3 file
java $playpen_args p3 pack $package_dir $output_dir

# Get the output name, PlayPen always writes to "<name>_<version>.p3"
p3_name=$(ls $output_dir | sort -n | head -1)

echo "Creating PlayPen cli config for $CI_JOB_NAME"
# PlayPen uses JSON for config files and there's no fancy way to create them other than this
echo "{\"name\": \"$CI_JOB_NAME\", \"uuid\": \"$PP_UUID\", \"key\": \"$PP_KEY\", \"coord-ip\": \"$PP_IP\", \"coord-port\": $PP_PORT, \"resources\": {}, \"attributes\": [], \"strings\": {}, \"use-name-for-logs\": true}" > /home/playpen/local.json

echo "Uploading package $p3_name to PlayPen network"
java $playpen_args cli upload "$output_dir/$p3_name"

echo "Done!"

Phew! That was a long one. Let me try to explain what each section does:

I’ve simplified my script a bit, but here’s what I removed not to make this article longer than a book:

Now that you’ve modified that script to suit your needs, we will need to set all the environment variables such as PP_UUID, PP_KEY

Setting up our secret environment variables

Add variable GitLab dialog

GitLab has an awesome option that allows us to hide some environment variables by not having to paste them directly into our .gitlab-ci.yml build config file. Go to your project page, hover over Settings and click on “CI / CD”. Now, expand the “Secret variables” panel and add the following:

Variable Explanation
PP_UUID Your user’s UUID
PP_KEY Your user’s key
PP_IP The address where the PlayPen network coordinator is running on
PP_PORT The port where the PlayPen network coordinator is listening to
PLAYPEN_NAME The subdirectory inside your GitLab assets repo which contains all the assets for this specific project
PACKAGE_PATH Where the Maven artifacts should be placed (relative to <asset_repo>/$PLAYPEN_NAME)

That’s it, you can now build your image with docker build -t playpen-runner . and tell GitLab to use it by adding a new deploy phase to your project’s .gitlab-ci.yml file:

playpen-deploy:
  image: playpen-runner
  stage: deploy
  when: manual
  script: "/home/start.sh"
  only:
  - master

Once you deploy this config to your repo and the build phase completes successfully you will see a “Play” button on the CI pipeline which you can click to package and deploy the artifacts to your PlayPen network:

GitLab project pipeline panel with Play button

If you push a commit to master you will realize the runner can’t find the playpen-runner image (I know what you’re thinking: “But Hugo, didn’t we build it earlier?”). The problem is that GitLab defaults to only pull from the Docker Hub registry and we want it to find our local playpen-runner image we just built. The fix is really easy, edit your /etc/gitlab-runner/config.toml file and add the following line:

[[runners]]
  ...
  [runners.docker]
    ...
    pull_policy = "if-not-present"
  ...

Now restart the runner by executing gitlab-runner restart. The if-not-present policy tells GitLab to only check on the Docker Hub registry if the image isn’t present on the local registry (where our playpen-runner image is). Remember to not deploy this image to any public registry as it contains your deploy key which grants access to your assets repository.

That’s it! You now have a fully automated build, test and deploy pipeline integrated with your GitLab repos. Instead of manually compiling your project on your machine, realizing you have missing dependencies or you are running a different outdated version of a dependency we now let GitLab handle all of this for you by clicking that little “Play” button. I’ve succesfully integrated this into one of the organizations I work at and it’s saving all the team members lots of time and headaches who now only need to run git push to get an updated version on a development environment in just 1 minute.