Deploying PlayPen packages to production using GitLab CI

Starting on December, I decided that I was tired of manually deploying PlayPen (a server management and load balancing framework) packages to a testing environment and later to production all by hand, and as so, I dove into the world of automation. Luckily, we were using GitLab, which is known for its amazing Continuous Integration, Delivery and Deployment capabilities integrating all the automation technologies you can think of. Our project is based on Java plugins which integrate together with the help of a coordination system running under one PlayPen instance. All our projects are built in Java and use Maven, so the first step was to setup the automated compilation of packages:

This is fairly easy, as there are lots of prebuilt Docker images which contain Maven. Create a .gitlab-ci.yml file on your project directory and add the following:

variables:
  MAVEN_OPTS: "-Dmaven.repo.local=.m2/repository -Dorg.slf4j.simpleLogger.log.org.apache.maven.cli.transfer.Slf4jMavenTransferListener=WARN -Dorg.slf4j.simpleLogger.showDateTime=true -Djava.awt.headless=true"
  MAVEN_CLI_OPTS: "-s .m2/settings.xml --batch-mode --errors --fail-at-end --show-version -DinstallAtEnd=true -DdeployAtEnd=true"

cache:
  paths:
  - .m2/repository

maven-build:
  image: maven:3.3.9-jdk-8
  stage: build
  script:
  - "mvn $MAVEN_CLI_OPTS package"
  artifacts:
    paths:
    - target/*.jar
    expire_in: 1 day

What this does is create a maven-build job which runs on the build stage (GitLab CI has three predefined stages by default: build, test and deploy), will package our Java project and will keep the produced artifacts for one day on GitLab (which you can download from the CI/CD panel). Now that we've built our project we need to package our artifacts and other assets which are required for the project to work with playpen-p3. Now, here's where the magic part happens, we're going to create our own Docker image which will package and deploy the code for us!

Docker image creation

PlayPen didn't have a Docker image so I decided to create one for you, you can check it out on the Docker Hub registry. Now that I've shamefully promoted my own creation, let's create a Dockerfile in a new directory that will contain our build scripts. You should also create a separate GitLab repo (which optionally needs Git LFS support as we will be storing huge files in it) that will contain all your assets that need to be packaged with the Maven artifacts produced in the previous phase.

FROM hugmanrique/playpen:latest
LABEL version="1.0"
LABEL author="Your Name <name@organization.tld>"

ENV LFS_VERSION 1.0.2
ENV GITLAB_HOST gitlab.example.com

# Install git
RUN apk add --update bash git openssh
RUN apk add --update --virtual build-dependencies curl && \
  curl -sLO https://github.com/github/git-lfs/releases/download/v${LFS_VERSION}/git-lfs-linux-amd64-${LFS_VERSION}.tar.gz && \
  tar xzf /git-lfs-linux-amd64-${LFS_VERSION}.tar.gz -C / && \
  mv /git-lfs-${LFS_VERSION}/git-lfs /usr/local/bin/ && \
  git-lfs init && \
  apk del build-dependencies && \
  rm -rf /git-lfs-${LFS_VERSION} && \
  rm -rf /git-lfs-linux-amd64-${LFS_VERSION}.tar.gz && \
  rm -rf /var/cache/apk/*

What these first lines do is install bash, git, openssh and git-lfs as they don't come installed by default on the Alpine distro which is the distro the PlayPen image is based on as you can check on Microbadger (playpen -> openjdk -> alpine). Next, we will create a SSH key pair that will be used as a read-only deploy key to clone and update the GitLab project that contains all our assets. First, we will need to create the SSH key pair (I'm assuming you're using Linux, macOS or Git Bash on Windows):

ssh-keygen -t rsa -C "playpenDeployer@example.com" -b 4096

This command will ask you where you want to save your key and if you want to set a password, which we aren't going to use as the SSH key usage is going to be automated. Once you have it, move them to your Dockerfile directory and create a .dockerignore file with the id_rsa.pub key as we only need the private key on the Docker image. Note: Your Docker image will contain a private SSH key, which means you shouldn't share it, and that's the main reason why we are giving it limited read-only access to a specific project by using the deploy keys GitLab feature. Talking about deploy keys, let's set them up!

First, access your assets project, hover over settings and click on "Repository":

GitLab project sidebar

Now, expand the "Deploy Keys" panel and paste the SSH public key we generated on the previous step. We don't need to enable write access as the Docker image is just going to clone and pull contents from this repo. Finally click on "Add key" and we're done! We're now going to return to our Dockerfile, enable the SSH agent, clone the repo and add a git alias (because git pull can be harmful in some cases as it can modify our working directory in unpredictable ways + we only want to fast-forward as the Docker image will become outdated with each commit we make to our assets repo):

# Setup ssh to get access to the packages repo on GitLab
ADD the_generated_private_key /root/.ssh/id_rsa
RUN \
  chmod 600 /root/.ssh/id_rsa && \
  eval $(ssh-agent) && \
  echo -e "Host ${GITLAB_HOST}\n\tStrictHostKeyChecking no\n" >> /etc/ssh/ssh_config && \
  ssh-add /root/.ssh/id_rsa

# Create safe git update alias
RUN git config --global alias.up '!git remote update -p; git merge -q --ff-only @{u}'

# Clone assets project (change master by your branch)
RUN git clone -b master git@${GITLAB_HOST}:/playpen-packages.git /home/packages

First, we will add the generated SSH private key to the image, initialize the ssh-agent and add a no StrictHostKeyChecking rule which we need because it's the first time we will connect to GITLAB_HOST and there's no entry for it in known_hosts. Finally, we add the git alias and clone the repo which means our Docker image will now contain all the files in the last commit. We're almost done with the Dockerfile, but we still need to add our own PlayPen build script, a config file for PlayPen logging and set some permissions because GitLab CI doesn't run as root by default and, as so, doesn't have access to some directories we've created:

# Add main script
COPY start.sh /home/start.sh
COPY build-log.xml /home/playpen/

# GitLab CI doesn't have access to /home
RUN chmod -R 700 /home/

Where build-log.xml is a log4j config file. Here's mine, but you can also copy the default logging-local.xml file created when you install PlayPen on your machine:

<?xml version="1.0" encoding="UTF-8" ?>
<Configuration>
    <Appenders>
        <Console name="console" target="SYSTEM_OUT">
            <PatternLayout>
                <Pattern>%-5p %c{1}: %m%n</Pattern>
            </PatternLayout>
        </Console>
        <File name="file" filename="playpen.log">
            <PatternLayout>
                <Pattern>%d{yyyy-MM-dd HH:mm:ss} %-5p %c: %m%n</Pattern>
            </PatternLayout>
        </File>
    </Appenders>
    <Loggers>
        <Root level="info">
            <AppenderRef ref="console" />
            <AppenderRef ref="file" />
        </Root>
        <Logger name="org.zeroturnaround.zip.ZipUtil" level="warn">
        </Logger>
    </Loggers>
</Configuration>

Notice how we copied a start.sh file that we haven't created yet? This is the code that the GitLab CI runner will execute everytime a commit is pushed to your project's repo:

#!/bin/bash

# Exit automatically if an error occurs
set -e

if ! { [ "$PP_UUID" ] && [ "$PP_KEY" ] && [ "$PP_IP" ] && [ "$PP_PORT" ] ; }; then
  echo "Some PlayPen variables are missing. Check the Secret variables Settings section on GitLab"
  exit 1
fi

# Update the assets repo from GitLab
cd /home/packages

# Checkout the needed branch (not really needed, but just in case), remove local changes created by previous builds and "pull" the changes from the repo
git checkout master && git reset --hard && git up
echo "Updated assets"

package=$PLAYPEN_NAME
package_dir=/home/packages/$package

# The PlayPen Docker image stores the JAR in /home/playpen
playpen=/home/playpen/PlayPen-1.1.2.jar
playpen_args="-Dlog4j.configurationFile=build-log.xml -jar $playpen"
output_dir=/home/output
package_output_dir=$package_dir/$PACKAGE_PATH

# Go back to the produced artifacts dir
cd -

# Move GitLab CI artifact to PlayPen package dir
# Create output dir (if missing)
mkdir -p $package_output_dir

# Grab all the JAR artifacts
for file in `ls -a target/*.jar` ; do
  # Maven produces "shaded" and "original" junk jars we don't want to move
  if [[ $file != *"shaded"* && $file != *"original"* ]] ; then
    echo "Moving $file artifact"
    mv $file $package_output_dir
  fi
done

# Create default PlayPen config files
cp /home/playpen/build-log.xml .
java $playpen_args ignore

echo "Packaging $package..."
mkdir -p $output_dir

# Package all the Maven artifacts + assets into a .p3 file
java $playpen_args p3 pack $package_dir $output_dir

# Get the output name, PlayPen always writes to "<name>_<version>.p3"
p3_name=$(ls $output_dir | sort -n | head -1)

echo "Creating PlayPen cli config for $CI_JOB_NAME"
# PlayPen uses JSON for config files and there's no fancy way to create them other than this
echo "{\"name\": \"$CI_JOB_NAME\", \"uuid\": \"$PP_UUID\", \"key\": \"$PP_KEY\", \"coord-ip\": \"$PP_IP\", \"coord-port\": $PP_PORT, \"resources\": {}, \"attributes\": [], \"strings\": {}, \"use-name-for-logs\": true}" > /home/playpen/local.json

echo "Uploading package $p3_name to PlayPen network"
java $playpen_args cli upload "$output_dir/$p3_name"

echo "Done!"

Phew! That was a long one. Let me try to explain what each section does:

  • Check if the environment variables (which we will later define through the Secret variables GitLab feature) for connecting to PlayPen are set.
  • Update the assets repo from the GitLab remote using the SSH deploy key we copied before
  • Define some useful variables to cleanup this messy script a bit. Notice we also use the $PLAYPEN_NAME and $PACKAGE_PATH variables which are the subdirectory in the assets repo which will be packaged with playpen-p3 and the directory inside <assets_repo>/$PLAYPEN_NAME where the Maven artifacts will be copied to respectively.
  • Create the package output dir inside the assets repo directory if not present
  • Move all the files which don't contain "shaded" or "original" on their name to the assets repo subdirectory.
  • Let PlayPen create its default config files for us.
  • Package /home/packages/$PLAYPEN_NAME with playpen-p3 and place the .p3 file on /home/output
  • Get the P3 file name for logging and deployment reasons
  • Create a default PlayPen CLI config that contains all the environment variables we will set later.
  • Finally upload the package to the network

I've simplified my script a bit, but here's what I removed not to make this article longer than a book:

  • Different PlayPen environments by using indirect expansions (${!uuid_key})
  • We need to create a SSH tunnel to connect to our dedis, and my script creates a UNIX socket which uses another SSH private key.
  • Logging the time it took to deploy the packages (it's normally on the milliseconds magnitude for us, so it's not that useful if you're running the GitLab CI runner on the same machine your PlayPen network is running)

Now that you've modified that script to suit your needs, we will need to set all the environment variables such as PP_UUID, PP_KEY...

Setting up our secret environment variables

Add variable GitLab dialog

GitLab has an awesome option that allows us to hide some environment variables by not having to paste them directly into our .gitlab-ci.yml build config file. Go to your project page, hover over Settings and click on "CI / CD". Now, expand the "Secret variables" panel and add the following:

VariableExplanation
PP_UUIDYour user's UUID
PP_KEYYour user's key
PP_IPThe address where the PlayPen network coordinator is running on
PP_PORTThe port where the PlayPen network coordinator is listening to
PLAYPEN_NAMEThe subdirectory inside your GitLab assets repo which contains all the assets for this specific project
PACKAGE_PATHWhere the Maven artifacts should be placed (relative to <asset_repo>/$PLAYPEN_NAME )

That's it, you can now build your image with docker build -t playpen-runner . and tell GitLab to use it by adding a new deploy phase to your project's .gitlab-ci.yml file:

playpen-deploy:
  image: playpen-runner
  stage: deploy
  when: manual
  script: "/home/start.sh"
  only:
  - master

Once you deploy this config to your repo and the build phase completes successfully you will see a "Play" button on the CI pipeline which you can click to package and deploy the artifacts to your PlayPen network:

GitLab project pipeline panel with Play button

If you push a commit to master you will realize the runner can't find the playpen-runner image (I know what you're thinking: "But Hugo, didn't we build it earlier?"). The problem is that GitLab defaults to only pull from the Docker Hub registry and we want it to find our local playpen-runner image we just built. The fix is really easy, edit your /etc/gitlab-runner/config.toml file and add the following line:

[[runners]]
  ...
  [runners.docker]
    ...
    pull_policy = "if-not-present"
  ...

Now restart the runner by executing gitlab-runner restart. The if-not-present policy tells GitLab to only check on the Docker Hub registry if the image isn't present on the local registry (where our playpen-runner image is). Remember to not deploy this image to any public registry as it contains your deploy key which grants access to your assets repository.

That's it! You now have a fully automated build, test and deploy pipeline integrated with your GitLab repos. Instead of manually compiling your project on your machine, realizing you have missing dependencies or you are running a different outdated version of a dependency we now let GitLab handle all of this for you by clicking that little "Play" button. I've succesfully integrated this into one of the organizations I work at and it's saving all the team members lots of time and headaches who now only need to run git push to get an updated staged version on a separate development environment in just 1 minute. As always, if you have any doubts, problems or concerns, you can contact me by clicking on the Contact button below.

© Hugmanrique. Made with