Running Jenkins Files inside GitLab CI
First, I want to set some ground rules for this. For starters, this process is not meant for long term use. There are many downsides to this - Such as it only runs in one GitLab Stage and isn't asyncronous. However this process can be used to run your Jenkins builds in GitLab CI, While you're migrating your Jenkinsfile to GitLab CI Syntax. Make no mistake - This doesn't solve your migration woes, But it does allow you to run your Jenkinsfile inside GitLab for the time being. It's a stop-gap measure.
Step 1: Setting up Jenkins Locally.
This process requires files from Jenkins. In order to do that we need to spin up a Jenkins instance locally, configure it, then extract the files. You can do that using the commands below. Note: The Jenkins version here is hard-set for a reason. The Jenkins version has to match the Jenkins Version listed in the JenkinsFile-Runner.jar file below.
docker run -d -p 49001:8080 jenkins/jenkins:2.176.2
fac214d16d9f302cbdcc7d950314cfb9e9da8b06efd7c8104c52034c82328a6a
docker exec -it ${THE_DOCKERID_RETURNED_ABOVE} /bin/bash
From this point, You should have a terminal that can access the bash prompt on Jenkins. In a few moments you can access http://localhost:49001 and be able to see the Jenkins initial setup. You should see something like below. We're going to need to use that Jenkins bash prompt to get the password.
jenkins@b4d2a5891b7f:/$ cat /var/jenkins_home/secrets/initialAdminPassword
c10985e17f7f44038fff0fa776e343e4
Next, we're going to be prompted with which plugins we want to install. Go ahead and pick your option. I chose just the typical plugins, But if you have unique needs, install those. We'll need them later. Once this is done you should see a screen with the plugins installing.
After this you'll be prompted to make a user account, please do so. Also select a password thats not used elsewhere, as we may be using it in plain text elsewhere.
Couple clicks here, couple taps there, and we're done!
Step 2: Extracting the Files from Jenkins
Now that we have a running Jenkins instance, we need to extract all of the files from Jenkins. We're going to exfiltrate the entire Jenkins Home directory from the docker container. So go ahead and spin up another bash prompt. You're going to want to make a local directory for this to save the files. You're going to need your containers ID for this. If you dont have it, use docker ps
mkdir -p ~/Projects/jenkins_home
docker cp ${DOCKER_CONTAINER_ID}:/var/jenkins_home ~/Projects/jenkins_home
docker cp ${DOCKER_CONTAINER_ID}:/usr/share/jenkins/jenkins.war ~/Projects/jenkins.war
Step 3: Making the JenkinsFile-Runner Bin File.
We're going to make a new file, Call it jenkinsfile-runner. Inside this file we're going to put the following. This is used to execute Jenkinsfile-Runner.
java -jar /app/bin/jenkinsfile-runner.jar ${@}
Step 4: Creating the GitLab CI Container.
So, I trust/assume you know how to make a GitLab CI Project. Go ahead and make one, name it whatever you want. For this project, i've named it gitlab-jenkinsfile-runner. When you create this repo, Make sure you initialize it with a README, Now clone it, we're going to do some things to it.
git clone ${YOUR_REPO_URL}
cd ${YOUR_REPO_DIR}
# Now move the Jenkins Home
mv ~/Projects/jenkins_home/ ./
# Now save the Jenkins Home
git add jenkins_home
Now that we have Jenkins Home setup to be in our Git Repo, We should make the .gitlab-ci.yml file to make the runner image and push it. In the interest of time, I've just copy/pasted the .gitlab-ci.yml file I've used below. It basically builds a docker image and uploads it to the local GitLab Registry.
docker-build:
image: docker:latest
stage: build
services:
- docker:dind
before_script:
- docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
script:
- docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
- docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
except:
- master
Now add this file using git, so that it's tracked. git add .gitlab-ci.yml From here we need to go about the longer process of making the dockerfile. I'm going to break up the dockerfile into chunks and explain it step by step. It's important that this is done right. If the jenkins version or plugins don't match/conflict you're going to have a bad time. So go ahead and create a Dockerfile.
The from should be from a JRE. All of Jenkins is using Java, so we need Java.
FROM openjdk:11-jdk
With these commands we're making the folder structure. We need these folders to put data into. The last step - We're downloading the jenkinsfile-runner.jar file. I have pegged the version of both this file and the Jenkins docker container above. The JenkinsFile-Runner has a pom dependancy on Jenkins. If they're out of sync. Bad things occur.
RUN mkdir -p /app/bin && \
mkdir -p /app/jenkins_home && \
mkdir -p /app/jenkins && \
wget https://repo.jenkins-ci.org/releases/io/jenkins/jenkinsfile-runner/jenkinsfile-runner/1.0-beta-10/jenkinsfile-runner-1.0-beta-10.jar -O /app/bin/jenkinsfile-runner.jar
With this, We're moving all of our files into the container in their proper places.
COPY jenkins_home/ /app/jenkins_home
COPY jenkins.war /app/bin/jenkins.war
COPY jenkinsfile-runner /app/bin/jenkinsfile-runner
COPY Test-Jenkinsfile /tmp/Jenkinsfile
We're setting the Home directory for Jenkins so Jenkins will not try to reinitalize itself.
ENV JENKINS_HOME /app/jenkins_home
Ok! Couple things going on here. To start with, we're unzipping the jenkins war so it can be used. We're moving and setting permissions of the jenkinsfile-runner execution script, and we're setting /bin/bash as the command. GitLab will override the entry point.
RUN unzip /app/bin/jenkins.war -d /app/jenkins && \
ln -s /app/bin/jenkinsfile-runner /usr/bin/jenkinsfile-runner && \
chmod +X /usr/bin/jenkinsfile-runner && \
chmod 777 /usr/bin/jenkinsfile-runner
CMD ["/bin/bash"]
Now, From here, you need to save git add * all of your files. Then you need to push them up git commit -am message && git push.
Step 4: How do we use this monstrosity?
Go ahead and make a new repo, This one will be the one we put your Jenkinsfile application into. We're going to create a simple .gitlab-ci.yml file to do the building. Below you'll see mine. In the script block, it calls the runner and specifies our Jenkins instance and our Plugins directory. Finally it accepts a Jenkinsfile. From here it executes the Jenkins file.
stages:
- build-jenkins
"JenkinsFile Build":
stage: build-jenkins
image: registry.gitlab.com/lackastack/article-repos/gitlab-jenkinsfile-runner:master
script:
- jenkinsfile-runner -w /app/jenkins -p /app/jenkins_home/plugins/ -f ./Jenkinsfile
Now, go ahead and add a Jenkinsfile. The simplier the better...
pipeline {
agent any
parameters {
string(name: 'param1', defaultValue: '', description: 'GitLab')
string(name: 'param2', defaultValue: '', description: 'Really Whips The Llamas Waterbowl.')
}
stages {
stage('Build') {
steps {
echo 'Hello Field!'
echo "message: ${params.param1}"
echo "param2: ${params.param2}"
sh 'ls -la'
}
}
}
}
Step 5: Conclusion.
Let's talk about this a bit. The way this works is that it starts a Docker container with a Jenkins instance inside it and uses the JenkinsFile-Runner project to parasite Jenkins and run our file. In an ideal production environment, You would take your Jenkins home directory and War files and put them into this container to run your Jenkinsfiles from within GitLab. The catch with this is versions. Jenkinsfile-Runner has a dependancy on Jenkins. So the versions have to match otherwise the plugins fail because they see the Jenkins version that JenkinsFile-Runner is built against.
Another thing not covered is credentials. With this method - You have to modify the credentials files in the Jenkins Home Directory to store credentials. Once you store them there, you can use them in your Jenkinsfile. You can even edit those files as part of the GitLab CI Job.
Lastly is Plugins. Many Jenkins jobs have numerous plugins in their pipeline. But Jenkins jobs also have plugins that do things outside of their pipeline. Currently, there is no solution for those as they're not in the Jenkinsfile. However, if you move those commands to your Jenkinsfile, You should be good to go.
This isn't a bulletproof solution, it's close. But you'll need to manage it and customize it. It's entirely possible you just copy your existing Jenkins War and Home directory into the container, and run it there. That should work. But if not, you'll need to make changes to Jenkins in the container to make it work 100%.