Note: This is an updated version of a previously published blog post, now including Directed Acyclic Graphs and minor code example corrections. Thanks Ivan Nemytchenko for authoring the original post!
Let's assume that you don't know anything about continuous integration (CI) and why it's needed.
Imagine that you work on a project, where all the code consists of two text files. Moreover, it is super critical that the concatenation of these two files contains the phrase "Hello world."
If it's not there, the whole development team won't get paid that month. Yeah, it is that serious!
The most responsible developer wrote a small script to run every time we are about to send our code to customers.
The code is pretty sophisticated:
cat file1.txt file2.txt | grep -q "Hello world"
The problem is that there are 10 developers on the team, and, you know, human factors can hit hard.
A week ago, a new guy forgot to run the script and three clients got broken builds. So you decided to solve the problem once and for all. Luckily, your code is already on GitLab, and you remember that there is built-in CI. Moreover, you heard at a conference that people use CI to run tests...
Let's run our first test inside CI
After taking a couple of minutes to find and read the docs, it seems like all we need is these two lines of code in a file called .gitlab-ci.yml:
test:
script: cat file1.txt file2.txt | grep -q 'Hello world'
We commit it, and hooray! Our build is successful:
Build succeeded!
Let's change "world" to "Africa" in the second file and check what happens:
Build failed
The build fails as expected!
OK, we now have automated tests here! GitLab CI will run our test script every time we push new code to the repository.
Note: In the above example, we assume that file1.txt and file2.txt exist in the runner host.
To run this example in GitLab, use the below code that first will create the files and than run the script.
test:
before_script:
- echo "Hello " > | tr -d "\n" | > file1.txt
- echo "world" > file2.txt
script: cat file1.txt file2.txt | grep -q 'Hello world'
For the sake of compactness, we will assume that these files exist in the host, and will not create them in the following examples.
Make results of builds downloadable
The next business requirement is to package the code before sending it to our customers. Let's automate that as well!
All we need to do is define another job for CI. Let's name the job "package":
test:
script: cat file1.txt file2.txt | grep -q 'Hello world'
package:
script: cat file1.txt file2.txt | gzip > package.gz
We have two tabs now:
Two tabs generated from two jobs
However, we forgot to specify that the new file is a build artifact, so that it could be downloaded. We can fix it by adding an artifacts section:
test:
script: cat file1.txt file2.txt | grep -q 'Hello world'
package:
script: cat file1.txt file2.txt | gzip > packaged.gz
artifacts:
paths:
- packaged.gz
Checking... it is there:
Checking the download buttons
Perfect!
However, we have a problem to fix: the jobs are running in parallel, but we do not want to package our application if our tests fail.
Run jobs sequentially
We only want to run the 'package' job if the tests are successful. Let's define the order by specifying stages:
stages:
- test
- package
test:
stage: test
script: cat file1.txt file2.txt | grep -q 'Hello world'
package:
stage: package
script: cat file1.txt file2.txt | gzip > packaged.gz
artifacts:
paths:
- packaged.gz
That should be good!
Also, we forgot to mention, that compilation (which is represented by concatenation in our case) takes a while, so we don't want to run it twice. Let's define a separate step for it:
stages:
- compile
- test
- package
compile:
stage: compile
script: cat file1.txt file2.txt > compiled.txt
artifacts:
paths:
- compiled.txt
test:
stage: test
script: cat compiled.txt | grep -q 'Hello world'
package:
stage: package
script: cat compiled.txt | gzip > packaged.gz
artifacts:
paths:
- packaged.gz
Let's take a look at our artifacts:
Unnecessary artifact
Hmm, we do not need that "compile" file to be downloadable. Let's make our temporary artifacts expire by setting expire_in to '20 minutes':
compile:
stage: compile
script: cat file1.txt file2.txt > compiled.txt
artifacts:
paths:
- compiled.txt
expire_in: 20 minutes
Now our config looks pretty impressive:
- We have three sequential stages to compile, test, and package our application.
- We pass the compiled app to the next stages so that there's no need to run compilation twice (so it will run faster).
- We store a packaged version of our app in build artifacts for further usage.
Learning which Docker image to use
So far, so good. However, it appears our builds are still slow. Let's take a look at the logs.
Wait, what is this? Ruby 2.5?
Why do we need Ruby at all? Oh, GitLab.com uses Docker images to run our builds, and by default it uses the ruby:2.1 image. For sure, this image contains many packages we don't need. After a minute of Googling, we figure out that there's an image called alpine which is an almost blank Linux image.
OK, let's explicitly specify that we want to use this image by adding image: alpine to .gitlab-ci.yml.
Now we're talking! We shaved nearly three minutes off:
Build speed improved
It looks like there's a lot of public images around. So we can just grab one for our technology stack. It makes sense to specify an image which contains no extra software because it minimizes download time.
Dealing with complex scenarios
So far, so good. However, let's suppose we have a new client who wants us to package our app into .iso image instead of .gz.
Since CI does all the work, we can just add one more job to it.
ISO images can be created using the mkisofs command. Here's how our config should look:
image: alpine
stages:
- compile
- test
- package
# ... "compile" and "test" jobs are skipped here for the sake of compactness
pack-gz:
stage: package
script: cat compiled.txt | gzip > packaged.gz
artifacts:
paths:
- packaged.gz
pack-iso:
stage: package
script:
- mkisofs -o ./packaged.iso ./compiled.txt
artifacts:
paths:
- packaged.iso
Note that job names shouldn't necessarily be the same. In fact if they were the same, it wouldn't be possible to make the jobs run in parallel inside the same stage. Hence, think of same names of jobs and stages as coincidence.
Anyhow, the build is failing:
Build failed due to missing mkisofs
The problem is that mkisofs is not included in the alpine image, so we need to install it first.
Dealing with missing software/packages
According to the Alpine Linux website mkisofs is a part of the xorriso and cdrkit packages. These are the magic commands that we need to run to install a package:
echo "ipv6" >> /etc/modules # enable networking
apk update # update packages list
apk add xorriso # install package
For CI, these are just like any other commands. The full list of commands we need to pass to script section should look like this:
script:
- echo "ipv6" >> /etc/modules
- apk update
- apk add xorriso
- mkisofs -o ./packaged.iso ./compiled.txt
However, to make it semantically correct, let's put commands related to package installation in before_script. Note that if you use before_script at the top level of a configuration, then the commands will run before all jobs. In our case, we just want it to run before one specific job.
Directed Acyclic Graphs: Get faster and more flexible pipelines
We defined stages so that the package jobs will run only if the tests passed. What if we want to break the stage sequencing a bit, and run a few jobs earlier, even if they are defined in a later stage? In some cases, the traditional stage sequencing might slow down the overall pipeline execution time. Imagine that our test stage includes a few more heavy tests that take a lot of time to execute, and that those tests are not necessarily related to the package jobs. In this case, it would be more efficient if the package jobs don't have to wait for those tests to complete before they can start. This is where Directed Acyclic Graphs (DAG) come in: to break the stage order for specific jobs, you can define job dependencies which will skip the regular stage order.
GitLab has a special keyword needs which creates dependencies between jobs, and allow jobs to run earlier, as soon as their dependent jobs complete.
In the below example, the pack jobs will start running as soon as the test job completes, so if in future someone adds more tests in the test stage, the package jobs will start to run before the new test jobs complete:
pack-gz:
stage: package
script: cat compiled.txt | gzip > packaged.gz
needs: ["test"]
artifacts:
paths:
- packaged.gz
pack-iso:
stage: package
before_script:
- echo "ipv6" >> /etc/modules
- apk update
- apk add xorriso
script:
- mkisofs -o ./packaged.iso ./compiled.txt
needs: ["test"]
artifacts:
paths:
- packaged.iso
Our final version of .gitlab-ci.yml:
image: alpine
stages:
- compile
- test
- package
compile:
stage: compile
before_script:
- echo "Hello " | tr -d "\n" > file1.txt
- echo "world" > file2.txt
script: cat file1.txt file2.txt > compiled.txt
artifacts:
paths:
- compiled.txt
expire_in: 20 minutes
test:
stage: test
script: cat compiled.txt | grep -q 'Hello world'
pack-gz:
stage: package
script: cat compiled.txt | gzip > packaged.gz
needs: ["test"]
artifacts:
paths:
- packaged.gz
pack-iso:
stage: package
before_script:
- echo "ipv6" >> /etc/modules
- apk update
- apk add xorriso
script:
- mkisofs -o ./packaged.iso ./compiled.txt
needs: ["test"]
artifacts:
paths:
- packaged.iso
Wow, it looks like we have just created a pipeline! We have three sequential stages, the jobs pack-gz and pack-iso, inside the package stage, are running in parallel:
Summary
There's much more to cover but let's stop here for now. I hope you liked this short story. All examples were made intentionally trivial so that you could learn the concepts of GitLab CI without being distracted by an unfamiliar technology stack. Let's wrap up what we have learned:
- To delegate some work to GitLab CI you should define one or more jobs in
.gitlab-ci.yml. - Jobs should have names and it's your responsibility to come up with good ones.
- Every job contains a set of rules and instructions for GitLab CI, defined by special keywords.
- Jobs can run sequentially, in parallel, or out of order using DAG.
- You can pass files between jobs and store them in build artifacts so that they can be downloaded from the interface.
Below are more formal descriptions of the terms and keywords we used, as well as links to the relevant documentation.
Keyword descriptions and documentation
| Keyword/term | Description |
|---|---|
| .gitlab-ci.yml | File containing all definitions of how your project should be built |
| script | Defines a shell script to be executed |
| before_script | Used to define the command that should be run before (all) jobs |
| image | Defines what Docker image to use |
| stages | Defines a pipeline stage (default: test) |
| artifacts | Defines a list of build artifacts |
| artifacts:expire_in | Used to delete uploaded artifacts after the specified time |
| needs | Used to define dependencies between jobs and allows to run jobs out of order |
| pipelines | A pipeline is a group of builds that get executed in stages (batches) |
More on CI/CD
- GitLab’s guide to CI/CD for beginners
- Get faster and more flexible pipelines with a Directed Acyclic Graph
- Decrease build time with custom Docker image
Photo by Christophe Dion on Unsplash
