Normalization of Deviance is "the process in which deviance from correct or proper behavior or rule becomes normalized in a government or corporate culture."

Over time, as you deviate from a process further and further, your safety margins shrink. Your actions become less resilient to failure, and more accident prone. As pictured above, a normalization of deviance, or a culture of deviation often results in a spiral. The deviation becomes the new normal, and further deviance continues.

At some point, that Deviance becomes codified further shrinking your failure, safety, and accident margins. As was the case with the Aleutian Trade Act of 1990; Where boats whose primary purpose it is engage in fishing commerce; were allowed to bypass inspections related to cargo for hire boats. Those fishing boats were then allowed to haul cargo. In the case of the FV Northern Belle, this led to overloading of a fishing ship aswell as incorrect load and balance calculations. Those two items, combine with a poorly maintained ship, with rusty watertight bulkheads; resulted in it's rapid sinking, unfortunate death of it's captain, and complete hull-loss.

And yet - Deviation from normal process is in some cases necessary. How do we design systems and processes that allow for deviation when necessary, but do not fuel a culture of deviance? A key component of resilience is flexibility; How do we design systems of delivery that are safe and secure, while remaining resilient and flexible?

Deployment of UN's FPIC Framework

The United Nations has a policy for working work Indigineous Peoples. This framework is called FPIC, or Free Prior Informed Consent. The purpose of this is to respect the decision and input of Indigenous Communities. By implementing this framework, we can work to ensure that Indigenous Communities are able to make informed decisions, without fear or pressure of coercion. However, as a side-effect of this framework, we ensure that the decision being made is fully weighed and understood. That it's benefits, costs, detriments, weaknesses, and downsides are fully understood. That the outcome of said action is understood to the best of everyone's ability, and any unknown outcomes that occur, are in good faith unknown to all.

With this, any actions or agreements we undertake are performed with a full understanding of the potential outcomes.

Which is exactly what a deviantion of process requires to insulate and prevent against creating a cultue or normalization of deviation. More importantly, because a deviation is a risk event, having informed consent is required to guard against adverse outcomes. Every deviation should include FPIC, so that the deviation and it's outcomes are fully understood and weighed, every time that it occurs.

An example of this would be skipping security tests for a software deployment, because the security scanners are offline. A resilient process is flexible, and it may be necessary to ship software for legal, finance, or other reasons. However the inability to skip security scans is a blocking function, because it's critical to the process. So in order to deliver, we need to deviate.

In this scenario, we would involve the Security Team, Management Team, QA Team, and Development Teams. We would sit down and collect all the evidence we have of previous security scans, any changes that are made, and all information we have on hand at that moment.

Then we allow said teams to process this information, become informed, and provide their consent. With all consent, the security scan is skipped, but the remainder of the process remains in effect. Without all consent, the delivery is scrapped, without backlash or political fallout.

A key component of Informed Consent is the lack of pressure or force. Consent is automatically invalidated the moment coercion is introduced. If we look at many Aviation or Boating incidents, we can find that respecting the single voice of a single dissenter, it could have (and has!) resulted in the avoidance of diaster.

Rejection of Process, and Setting things on fire

Now, if you've gotten this far, I know some of you like me are going "But Chris, when do we set the entire thing on fire."

When you reach a point where requests for deviation become the normal. That is a key indicator your process is broken and not working. That it's no longer meeting it's current purpose and requires reconsideration. Without changing the process, you risk making deviations normal, and inadvertently create a culture where your folks accept deviation as normal. This is a unwanted outcome even if FPIC is utilized, every deviation from process increases your risk. Over time, small dangerous events, can result in a high risk level.

So, instead of burning the process down. You take it, study it, identify what works and does not work, finally you slowly iterate on it to improve it. Process exists for a reason; just because a process no longer benefits you, it does not mean that reason ceases to exist. So, respect your prior process and the lessons that process was born from.

Collect metrics on the deviations from your current process, categorize them, identify them, and understand why they occurred. Take those lessons and build them into your next process so that you have less deviation events. In our security scanner example above; Maybe we build a testing environment for our security tools, and maybe we only allow maintenance on the production tools on Mondays... Unless a deviation is required.

In Closing

My goal of editorials on my blog are to get you thinking. To get you to wonder about things, ask questions, and challenge yourself. I aim to be as factual as possible, and I aim to never give you information that can not be actionable.

Normalization of Deviation is behind a great many accidents, mistakes, and it has robbed many of their lives. However, Deviation is also required, and without deviation, we also risk removing the tools necessary to prevent such things. I hope after reading this it has you wondering where deviation occurs in your process, and how you can improve it.

This is an article I've been meaning to write for a while. The purpose of this article is to cover the methodology and purpose behind "Purpose Built Containers". This is a concept you've likely seen expressed elsewhere and in other canontations. As good example of this is the 12 Factor Application or OpenShift's S2I. To be clear, these are not new concepts when it comes to containers, but it's a methodology tied to GitLab and one that stands on the shoulders of giants.

A "Purpose-Built Container" is, if you've ever seen Rick and Morty, esentially a Meeseeks Box of the IT World. That means it's a container that exist only to serve a specific purpose and then it quickly and quietly dies or is shut down. It is a container that is not designed to be long standing or host an application. It starts, it does it's purpose, then it stops and exits.

Note: When we refer to container, we mean the compiled image that is runable. When we refer to the image, we're refering to the dockerfile definition, prior to being built.

Our Purpose Built Container
Using Our Purpose Built Container

An example of this process can be seen here

GitLab

Every CI/CD Job inside of a GitLab Pipeline observes this process. Every Job spins up, pulls & starts a docker image, and then conducts a build job. Finally the container is stopped and shutdown. Every job spins up a new container from a docker image, and does the same process. No container that is created is ran twice; Even if a job is retried, a new container is spun up.

GitLab CI/CD can utilize any docker image, but not every docker image is built with this process in mind. For example, some docker images spin up an entire virtual machine, others will spin up a database alongside your application in the container. These are not purpose-built containers. They serve multiple purposes and their purposes change over time, Where-as a purpose built container serves one purpose and then dies.

Anatomy of a Purpose-Built Container

A Purpose-Built Container is comprised of the following components:

1. They're Small: A purpose-built container should be comprised of only the minimum number of components needed to run. This may include a NodeJS Runtine for example. However, you'd want to avoid including any application specific dependencies. You only want to include the NodeJS Runtime. After the container is spun up and running, then you inject your dependancies into it. There are exceptions to this rule, For example internal certificates may be included into the image. The rule is, if the image doesn't need it to run, it shouldn't be included.

2. They're Multiuse: "Purpose-built" applies to how a container runs, and not how it's built. You can build a container for multiple purposes and uses aslong as it doesn't violate rule #1. You want your purpose-built containers to be built with multiple consumers and usages in mind. Not only should this reduce the specific configurations you include in it, it should keep the image small in size. Therefore following Rule #1 and giving it speed. I.e. Maven and NodeJS should be separate containers, but each can include global requirements like internal certificates.

3. Their entrypoint is empty: A "Purpose-Built" container is a zombie, it has no purpose other than what is given to it. Because of this, it should contain a dummy entrypoint or empty entrypoint. A perfect example of this is an entrypoint with a shell-script that echos the instructions on how to use the container and not run it. "Purpose-Built" containers are not designed to be long standing or ran on a customer facing/production environment.

4. Avoid needless layers: Every command inside of a Dockerfile produces a new "layer" inside of the Docker Image, which makes the Image larger and also can make it more difficult to pull down. Both of these things result in a slower image to pull down, start, run, and destroy. You should reduce those as much as possible, opting for RUN command && command as opposed to RUN command \n RUN command - The first will produce one layer, the second two layers.

5. Run with least-amount of privilege:" Most purpose-built containers will not need any form of elevated permissions. They need the bare minimum of user permissions to function. With these you should have them run with USER 1001 so that they run with a random user. On OpenShift this is a requirement, on Docker/Kubernetes this is a great practice. On OpenShift, all random user's run under the ROOT user group, but without ROOT permissions. So just set any files or folders to being owned by the root group and they'll be usable by this root user.

How do we build the image?

To build the image, we're going to start with a basics folder structure. Go ahead and make a new GitLab Project, then initialize it with a readme and clone it down to your local machine. The folder structure for this should follow the following.

• Dockerfile
• .gitlab-ci.yaml

1. For our Dockerfile we're going to do the following..

Remember, we want out Dockerfile to be as light and generic as possible. Notice on the RUN line how I'm merging two commands into one to reduce layers. You will also note that I have added USER 1001 to the bottom of the Dockerfile, this is to ensure that the container runs as a random user without any permissions. While some workloads may require more, for this purpose, we don't.

At the end of the RUN line, we have a chmod command. This command is used to give the /.npm folder root group permissions. The purpose of this is that any random user that spins up has the root group, but not root user privilege.

FROM alpine:3.12.0
RUN apk update && apk add --no-cache nodejs npm && mkdir /.npm && chmod -R g=u /.npm

USER 1001

CMD ["echo", "This is a 'Purpose-Built Container', It is not meant to be ran this way. Please visit www.lackastack.net to see how to use it!"]

2. Our GitLab CI File to automating create and maintain it.

This GitLab CI file will build our image and save it in the local GitLab Container Registry of this project. It will save it as latest when built. When git is tagged, it will release an image with that GitLab CI Tag as the Docker Image tag. This will allow you to properly version your Docker Images for release and maintenance.

docker-build-master:
  image: docker:latest
  stage: build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
    - docker push "$CI_REGISTRY_IMAGE"
  only:
    - master

docker-build-release:
  image: docker:latest
  stage: build
  when: manual
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
  only:
    - tags

How do we use the image?

If you've gotten here, we have an image. It's light-weight, it follows the rules, and it has a NodeJS Runtime. So, now below we have a GitLab CI Pipeline. This pipeline will utilize our image. You'll also notice that this has a before_script block with an array of parameters. We're using only one parameter, to install expressjs. Now, Why are we installing this here?

Above, we spoke about the fact our image needs to be multiuse. This means we can't bake things like expressjs into our NodeJS image. So we use before_script to run the commands to install it. This is a careful dance, if you have a couple dependancies, this wont take long. But if you have a large number of dependancies that many people use. You should bake them into the docker image and allow them to be reused.

The goal here is about speed, and less waste. If you have a bunch of unique images where everyone makes one, you have a bunch of waste. But if you have a small number of images with everything but the kitchen sink, you end up with very slow pipelines. There's a careful and delicate balance to be had.

build:
  stage: build
  image: registry.gitlab.com/lackastack/article-repos/purpose-built-containers:latest
  before_script:
    - npm install expressjs
  script:
    - echo "Installed"

Conclusion

I hope you got some new insight out of this. The goal here again, is to build a container for use inside of GitLab CI in addition to being used inside of OpenShift. Many public containers are not usable inside of OpenShift due to security rules and thus you may be forced to make your own. If you follow this process you images should be quick and effective on GitLab but also run properly on OpenShift.

May your containers be small, and your CI Pipelines be fast.

By default GitLab & it's Runners require anyuid and root access to work. They also require Helm charts to install. If you're an OpenShift Administrator, these two things are likely not compatible with your way of doing things. You likely also dont want GitLab Runners to run CI Jobs using AnyUID and Root access. The purpose of this guide is to walk you through how to build your own GitLab Runner container that does not use Root or AnyUID. So that it's secure and also communicated properly with GitLab.

Requirements

• One OpenShift Cluster (With Storage)
• One GitLab Installation
• Cluster Admin access to OpenShift Cluster

The very first thing you're going to want to do is clone the repo with our OpenShift Resources. Then we'll go through them.

cd <Your_Projects_Directory>
git clone https://gitlab.com/lackastack/article-repos/openshift-gitlab-runner.git

How the runner is assembled.

I've gone through the process of setting up OpenShift Templates to make this process easier. Without it, this guide may be super long. I encourage you to read through the templates and understand how they work and what they do. I'm also not going to deeply dive into the dockerfile, but call out some specific things.

Go ahead and open the Dockerfile. This is the file we're going to be using for building our container. From lines 6 - 27, You'll see that we're inserting the GitLab Certificates for YUM Repos. Then we're going through on line 29-31 and installing the gitlab-runner. Everytime you build this container, it will install the newest gitlab runner into the container.

But it's lines 34-40 I want to point out.

RUN mkdir -p /etc/gitlab-runner/certs && \
    mkdir /.gitlab-runner && \
    mkdir /tmp/gitlab-home && \
    chmod -R g=u /.gitlab-runner && \
    chmod -R g=u /etc/gitlab-runner && \
    chmod -R g=u /tmp/gitlab-home && \
    chmod +x /usr/bin/dumb-init

These lines do the magic. When the gitlab-runner installs via yum, it creates folders itself with permissions for root and the gitlab user. Because OpenShift runs as random uid, we can't/shouldn't use these. So we're going ahead and making new folders and also reassigning those folders to the root group. You see, when OpenShift assigns a random uid to a container, that random uid is always in the root group. So because we use chmod -r g=u this command removes the user permissions to those folders and assigns them to a group.

Now, let's go ahead and open the entrypoint file. Scroll down to lines 19-31, Ignore that certificate stuff.

if [ ! -f "/.gitlab-runner/config.toml" ]; then
  gitlab-runner register --non-interactive \
                        --template-config /scripts/config.toml \
                        --url $GITLAB_URL \
                        --executor kubernetes \
                        --locked=false \
                        --access-level="not_protected" \
                        --docker-privileged=false \
                        --kubernetes-privileged=false 
fi
# Start the runner
gitlab-runner run --working-directory=/tmp/gitlab-home --listen-address=0.0.0.0

Everytime the container starts/restarts, etc will run this script. We start by checking to see if the config.toml is in one of the directories we created. If it's not, then we need to register the runner. So we start that process, we register the runner with a one-line registration method. The secret sauce is here --template-config /scripts/config.toml. Later in our openshift templates, we'll create a configmap with our basic config.toml, and then input our default values into it. The GitLab Runner will take this file, use it as a template and fill out the rest for us.

--executor kubernetes

This is so the runner operates as if it was in Kubernetes, and it sort-of is.

--access-level="not_protected" \
--docker-privileged=false \
--kubernetes-privileged=false

These lines tell the runner to run with the fewest permissions as possible. This prevents us from doing Docker in Docker - But because you're on OpenShift we don't need that.. You'll see why in an upcoming GitLabOps on OpenShift tutorial.

Importing and Building the Runner.

This is where you're Cluster Admin access will come in handy! Before we do this, Please take a look at the build-template.yml file. This is what we'll be using and importing. You shouldn't import anything into your cluster without looking into it and what it does. Without further-a-do lets import the template, run the following.

oc login <stuff>
oc project openshift
oc process -f ./build-template.yml | oc apply -f-

This will import the build object and image stream into your OpenShift cluster in the OpenShift Namespace. Why there? Because we want this image and template to be accessible by everyone, but not editable by anyone but a cluster admin. Now that this is done, let's go ahead and start the image build.

cd <Your_GitCloned_Directory>
oc start-build gitlab-runner --from-dir=. --follow

At this point, the build is going to start and stream it's output to the terminal. Let it run it can take up to 5 minutes. When it's done you should see a success or pushing image message. This means the image is in the repo and ready to be used.

The GitLab Runner Template.

This is the template that we want to import for all users of GitLab to provision and deploy their own runners. The idea is that users of OpenShift and GitLab can have a namespace on OpenShift and a project on GitLab, Then install a runner on OpenShift in their namespace pointed to their GitLab project. Then their runners don't affect anyone elses namespaces, you can monitor metrics per namespace, and limit resources on the namespace also. Plus, it's self-service, so win-win.

The template included is big, Please read it. However, i'm not going to quote it here because it's huge. This template when installed in the OpenShift namespace adds the GitLab-Runner to the self-service portal for namespaces. When executed this template installs:

• 2 Persistent Volume Claims
• 1 Deployment Config
• 1 ConfigMap
• 2 RoleBindings
• 1 Service Account

All of these elements are required to power the runner. Let's go ahead and install it now.

oc login <stuff>
oc project openshift
oc apply -f ./runner-template.yml -n openshift

If this succeeds in a few moments users will be able to see the GitLab-Runner object in their self-service portal.

The Final Touches, Deploying the Runner.

Running Jenkins Files inside GitLab CI

First, I want to set some ground rules for this. For starters, this process is not meant for long term use. There are many downsides to this - Such as it only runs in one GitLab Stage and isn't asyncronous. However this process can be used to run your Jenkins builds in GitLab CI, While you're migrating your Jenkinsfile to GitLab CI Syntax. Make no mistake - This doesn't solve your migration woes, But it does allow you to run your Jenkinsfile inside GitLab for the time being. It's a stop-gap measure.

Step 1: Setting up Jenkins Locally.

This process requires files from Jenkins. In order to do that we need to spin up a Jenkins instance locally, configure it, then extract the files. You can do that using the commands below. Note: The Jenkins version here is hard-set for a reason. The Jenkins version has to match the Jenkins Version listed in the JenkinsFile-Runner.jar file below.

docker run -d -p 49001:8080 jenkins/jenkins:2.176.2
fac214d16d9f302cbdcc7d950314cfb9e9da8b06efd7c8104c52034c82328a6a
docker exec -it ${THE_DOCKERID_RETURNED_ABOVE} /bin/bash

From this point, You should have a terminal that can access the bash prompt on Jenkins. In a few moments you can access http://localhost:49001 and be able to see the Jenkins initial setup. You should see something like below. We're going to need to use that Jenkins bash prompt to get the password.

jenkins@b4d2a5891b7f:/$ cat /var/jenkins_home/secrets/initialAdminPassword
c10985e17f7f44038fff0fa776e343e4

Next, we're going to be prompted with which plugins we want to install. Go ahead and pick your option. I chose just the typical plugins, But if you have unique needs, install those. We'll need them later. Once this is done you should see a screen with the plugins installing.

After this you'll be prompted to make a user account, please do so. Also select a password thats not used elsewhere, as we may be using it in plain text elsewhere.

Couple clicks here, couple taps there, and we're done!

Step 2: Extracting the Files from Jenkins

Now that we have a running Jenkins instance, we need to extract all of the files from Jenkins. We're going to exfiltrate the entire Jenkins Home directory from the docker container. So go ahead and spin up another bash prompt. You're going to want to make a local directory for this to save the files. You're going to need your containers ID for this. If you dont have it, use docker ps

mkdir -p ~/Projects/jenkins_home
docker cp ${DOCKER_CONTAINER_ID}:/var/jenkins_home ~/Projects/jenkins_home
docker cp ${DOCKER_CONTAINER_ID}:/usr/share/jenkins/jenkins.war ~/Projects/jenkins.war

Step 3: Making the JenkinsFile-Runner Bin File.

We're going to make a new file, Call it jenkinsfile-runner. Inside this file we're going to put the following. This is used to execute Jenkinsfile-Runner.

java -jar /app/bin/jenkinsfile-runner.jar ${@}

Step 4: Creating the GitLab CI Container.

So, I trust/assume you know how to make a GitLab CI Project. Go ahead and make one, name it whatever you want. For this project, i've named it gitlab-jenkinsfile-runner. When you create this repo, Make sure you initialize it with a README, Now clone it, we're going to do some things to it.

git clone ${YOUR_REPO_URL}
cd ${YOUR_REPO_DIR}
# Now move the Jenkins Home
mv ~/Projects/jenkins_home/ ./
# Now save the Jenkins Home
git add jenkins_home

Now that we have Jenkins Home setup to be in our Git Repo, We should make the .gitlab-ci.yml file to make the runner image and push it. In the interest of time, I've just copy/pasted the .gitlab-ci.yml file I've used below. It basically builds a docker image and uploads it to the local GitLab Registry.

docker-build:
  image: docker:latest
  stage: build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
  except:
    - master

Now add this file using git, so that it's tracked. git add .gitlab-ci.yml From here we need to go about the longer process of making the dockerfile. I'm going to break up the dockerfile into chunks and explain it step by step. It's important that this is done right. If the jenkins version or plugins don't match/conflict you're going to have a bad time. So go ahead and create a Dockerfile.

The from should be from a JRE. All of Jenkins is using Java, so we need Java.

FROM openjdk:11-jdk

With these commands we're making the folder structure. We need these folders to put data into. The last step - We're downloading the jenkinsfile-runner.jar file. I have pegged the version of both this file and the Jenkins docker container above. The JenkinsFile-Runner has a pom dependancy on Jenkins. If they're out of sync. Bad things occur.

RUN mkdir -p /app/bin && \
    mkdir -p /app/jenkins_home && \
    mkdir -p /app/jenkins && \
    wget https://repo.jenkins-ci.org/releases/io/jenkins/jenkinsfile-runner/jenkinsfile-runner/1.0-beta-10/jenkinsfile-runner-1.0-beta-10.jar -O /app/bin/jenkinsfile-runner.jar

With this, We're moving all of our files into the container in their proper places.

COPY jenkins_home/ /app/jenkins_home
COPY jenkins.war /app/bin/jenkins.war
COPY jenkinsfile-runner /app/bin/jenkinsfile-runner
COPY Test-Jenkinsfile /tmp/Jenkinsfile

We're setting the Home directory for Jenkins so Jenkins will not try to reinitalize itself.

ENV JENKINS_HOME /app/jenkins_home

Ok! Couple things going on here. To start with, we're unzipping the jenkins war so it can be used. We're moving and setting permissions of the jenkinsfile-runner execution script, and we're setting /bin/bash as the command. GitLab will override the entry point.

RUN unzip /app/bin/jenkins.war -d /app/jenkins && \
    ln -s /app/bin/jenkinsfile-runner /usr/bin/jenkinsfile-runner && \
    chmod +X /usr/bin/jenkinsfile-runner && \
    chmod 777 /usr/bin/jenkinsfile-runner
CMD ["/bin/bash"]

Now, From here, you need to save git add * all of your files. Then you need to push them up git commit -am message && git push.

Step 4: How do we use this monstrosity?

Go ahead and make a new repo, This one will be the one we put your Jenkinsfile application into. We're going to create a simple .gitlab-ci.yml file to do the building. Below you'll see mine. In the script block, it calls the runner and specifies our Jenkins instance and our Plugins directory. Finally it accepts a Jenkinsfile. From here it executes the Jenkins file.

stages:
    - build-jenkins

"JenkinsFile Build":
    stage: build-jenkins
    image: registry.gitlab.com/lackastack/article-repos/gitlab-jenkinsfile-runner:master
    script:
      - jenkinsfile-runner -w /app/jenkins -p /app/jenkins_home/plugins/ -f ./Jenkinsfile

Now, go ahead and add a Jenkinsfile. The simplier the better...

pipeline {
    agent any
    parameters {
        string(name: 'param1', defaultValue: '', description: 'GitLab')
        string(name: 'param2', defaultValue: '', description: 'Really Whips The Llamas Waterbowl.')
    }
    stages {
        stage('Build') {
            steps {
                echo 'Hello Field!'
                echo "message: ${params.param1}"
                echo "param2: ${params.param2}"
                sh 'ls -la'
            }
        }
    }
}

Step 5: Conclusion.

Let's talk about this a bit. The way this works is that it starts a Docker container with a Jenkins instance inside it and uses the JenkinsFile-Runner project to parasite Jenkins and run our file. In an ideal production environment, You would take your Jenkins home directory and War files and put them into this container to run your Jenkinsfiles from within GitLab. The catch with this is versions. Jenkinsfile-Runner has a dependancy on Jenkins. So the versions have to match otherwise the plugins fail because they see the Jenkins version that JenkinsFile-Runner is built against.

Another thing not covered is credentials. With this method - You have to modify the credentials files in the Jenkins Home Directory to store credentials. Once you store them there, you can use them in your Jenkinsfile. You can even edit those files as part of the GitLab CI Job.

Lastly is Plugins. Many Jenkins jobs have numerous plugins in their pipeline. But Jenkins jobs also have plugins that do things outside of their pipeline. Currently, there is no solution for those as they're not in the Jenkinsfile. However, if you move those commands to your Jenkinsfile, You should be good to go.

This isn't a bulletproof solution, it's close. But you'll need to manage it and customize it. It's entirely possible you just copy your existing Jenkins War and Home directory into the container, and run it there. That should work. But if not, you'll need to make changes to Jenkins in the container to make it work 100%.

In a previous article we wrote about how to use an external bot to obtain approvals. This method is one take on that problem. It has it's advantages, such as having the bot being able to be tied into third party services. It can handle logging, it can collect information at approval time to pass to the pipeline. There's alot of flexibility there. The downside being that it's an external bot, and everything that comes with that.

Today, we're going to show how to do Multiple Approvals in a GitLab CI pipeline, with just GitLab and it's tools. We're going to be using Protected Environments, Manual CI Jobs, and Approvers. The way this works is we have multiple stages in a CI Pipeline that require approvers for each. Without an approval - that stage does not progress.

Make an Environment.

The first step in this is to go to Operations -> Environments and create a new environment. I named mine ctimberlake-approval with my username being first, then approval as second. This makes it easy to know what it's for at a glance. Leave the URL Blank. You'll want to do this for every person who needs to approve the job.

Once that's done, and you have the environments. You need to protect them. From here you go to Settings -> CI/CD -> Protected Environments. You click the environment of the user, then select the user.

After this is done - You're process should look like this. If it does, The setup is done for environments.

Now that we have the setup done, we need to set this up to work properly. Below is our pipeline, I'm going to explain it simply. First, We define the stages. We have a build stage, This could be anything you want. But then we have an approvals stage. Everything before this approvals stage will run without delay. However, then it gets to approvals. The pipeline stops solid. The final step is the release step - Which only fires if we have approvals.

The magic here works because we've specified the approval job with (3) three things.

1. when: manual - This will prevent the pipeline from proceeding without manual intervention.
2. allow_failure: false - This will prevent the release stage from running by itself. Without it the pipeline wont stop at the approvals.
3. environment: name: ctimberlake-approval - Because this environment is protected, The only person who can trigger this job are those that have access to the environment.

stages:
  - build
  - approvals
  - release

build-1:
  stage: build
  script:
    - echo "We're building here"

ctimberlake-approval:
  stage: approvals
  when: manual
  allow_failure: false
  environment:
    name: ctimberlake-approval
  script:
    - echo "We're just doing a simple echo statement here. But a curl request could be made to mark the approval to a cental compliance tool."

versable-approval:
  stage: approvals
  when: manual
  allow_failure: false
  environment:
    name: versable-approval
  script:
    - echo "We're just doing a simple echo statement here. But a curl request could be made to mark the approval to a cental compliance tool."

release-1:
  stage: release
  script:
    - echo "We're releasing stuff here"

If you did everything right, You'll be met with a pipeline like below. Where it shows that I can approve one job (and I did). However, the release stage has not triggered yet because Michael hasn't hit the button. When he does, Release will then fire.

How to protect the pipeline.

If you take this approach. Then it means that the .gitlab-ci.yml is limiting your pipeline from running through and requiring approval for the end steps. What is to prevent someone from just editing that file, removing the approval steps, and then rogue releasing?

Simple answer, We lock the .gitlab-ci.yml file.

Create in the root of your gitlab directory a file named CODEOWNERS. In this file, add a line .gitlab-ci.yml @your-username-here. Now, commit it. Anytime someone wants to edit the pipeline they'll need specific approval from someone in the codeowners file that's listed there. Now Michael can't edit the gitlab-ci.yml file to remove my approvals! Muhaha.

If you succeeded, you should now see the following when you look at .gitlab-ci.yml

Many companies and organizations that deal with IT have some form of Self-Service. Whether this be using something like ServiceNow, Microsoft Sharepoint, or something random like a Rube Goldberg Machine initiated by an e-mail. In either case, they have a system where users can go to a form, initiate a request and have a process ran. This functionality is a key requirement of process control theory. Today we're going to discuss how we can use GitLab to handle Self-Service.

Before we begin, This is not meant to replace ServiceNow or Sharepoint. It's to show an alternate using GitLab

For this article, you're going to need the following:

1. A GitLab Instance that supports GitLab Pages.
2. A GitLab Admin account, or Owner Access to (Two) GitLab Repositories.
3. A WYSIWYG Editor, For this article we're using Adobe Dreamweaver.
4. Some Knowledge of HTML

For the next Article in this series you'll need:

1. Some Knowledge of Ansible
2. A cloud provider account. Vultr, DigitalOcean, Amazon AWS, Etc.

The Forms

The most important part of this is the form. After-all, without it, the user can't make a request. For this we're going to use GitLab Pages to serve our self-service form. So, at this point, go ahead and login to GitLab and make a new repository. Call it something like "Self-Service Portal". This should be based off of the Pages/HTML Template. See the picture below for this.

Once this is done, you're given a repository with a public/index.html and public/style.css structure. Now we're going to need to pull down that repo and modify the index.html file.

git clone https://repourl/repo

With the files downloaded, we'll need to open that public/index.html file with your HTML WYSIWYG Editor. If you remember i'm using Dreamweaver. Once open you're going to need to do the following..

1. Open the file in the WYSIWYG Editor.
2. Delete all text in the body such as This is a simple plain-HTML website on GitLab Pages, without any fancy static site generator.
3. You're going to want to add a form to this page.
4. Within that form, you're going to want to add fields and labels. One label and field for each value.

Below is a screenshot from my Dreamweaver editor showing what I've created so far. Don't worry about the details of things now, just try to get a form laid out.

The CI/CD Portion

Now that we have that setup, we're going to need to create another GitLab Repo. This repo is going to house all of our infrastructure code. The goal for this repo is to contain no secrets, no specific data, it's meant to be a complete dumb pipeline. It should only know what we pass from the form, This means all secrets, keys, data, etc should be from the form.

Below is an example of how it should be setup.

At this point, you have a completely blank repository. It should have a readme, and that's all.

Do you see that button that says "Set Up CI/CD"? Go ahead and click that, this will take you to the GitLab Code Editor. We're going to walk through creating a GitLab CI File. You can read tons more docs on this here.

For starters, we're going to want to define the stages our jobs will run in. For this example, we're going to have only two. The first is a sanity check, the second is the infrastructure portion. The top of your CI File should look like this..

stages:
    - check-vars
    - do-infrastructure

Pretty descriptive right? Next we're going to define the first stage, the one that checks the variables. For this article, it's just echoing them.

check-variables:
    stage: check-vars
    script:
        - echo "VM Name is $VM_Name"
        - echo "Amazon AWS Region is $AWS_Region"
        - echo "E-Mail is $EMAIL_NOTIFICATION"
        - echo "Reason is $REASON"

The first line above is defining the job name `check-variables`. Then we indent and define the stage, it needs to match one of our previously defined stages. `stage: check-vars`.The script block defines what we need to do in this job. Imagine each line in a bash file is listed out with a - preceding it to designate it as an array item. For the infrastructure block, we're just going to echo some garbage for now.

build-infrastructure:
    stage: do-infrastructure
    script:
        - echo "Do Infrastructure in Article #2 W00T!"

Go ahead and commit those changes. With that, our pipeline should be good to go.

Connecting The Form to the CI/CD.

Now, the big question arises, How the heck do we connect these two things? It's a secret, it'll cost you $3.50... No, it's simple. We'll simply use a pipeline trigger.

In the left navigation of the Pipeline Repo, go to Settings -> CI/CD. Then click expand on Pipeline Triggers

From here you'll want to create a trigger. In the Trigger Description field, enter something in. In our case, we're going to call it "Self-Service Trigger". Once you create it, you'll be granted a pipeline trigger token. Save this token somewhere. Then below that you'll see a number of use cases for the pipeline trigger. Find the section "Use Webhook", copy the URL thats in that field. Entirely.

Now that we have that information, We need to update our forms. Go ahead and find your HTML file (should be named index.html). Open it with notepad or your favorite error.

The first thing we need to setup the form to POST to the Trigger pipeline. You'll need to find your <form block and add action="webhookurl" with the REF_NAME being master, and TOKEN being replaced with your token. Then you need to add method="POST" to the same line, this marks it as a POST request. My example is below.

<form id="feedbackForm" class="text-center" action="https://gitlab.com/api/v4/projects/14883084/ref/master/trigger/pipeline?token=943227eec64c3ac6ab4e8436a45bfa" method="POST">

Now, when we submit the form it will trigger the pipeline! AWESOME! Except... We also want to pass variables. Remember above how we referenced the $VM_Name, $AWS_Region, $EMAIL_NOTIFICATION, $REASON above? Now, we need to assign those in our form. To link these values from the HTML Form to the pipeline, we need to set the name of the form field. This name must be named like this: variables[VM_Name] or variables[AWS_Region].

Go ahead and make those changes, add name="variables[VM_Name]" to your input field, or select the field in the editor and set the name. An example of this can be seen below.

Now save this file, and commit it to the self-service repository. The pipeline should kick off and your form should then become visible. You can find the URL for your self-service at Settings -> Pages. It may take up to five minutes for this to happen.

In Closing

At this point, if you've followed everything. You should be able to submit the form and see the pipeline kick off. Let us know if you have any concerns!

You can see the example of the form i've created here.

You can see your pipeline being ran and it's result here.