Speeding up GitLab CI Laravel Tests with a Custom Docker Image
This is a follow up to the previous post Continuous Integration for your Laravel Application with GitLab CI.
In this article we’ll run through what the costs are of using the template, reviewing the template contents and how we can use our own docker image to speed up the job run time by over 200%.
Reliability at the Cost of Speed
The template that ships in GitLab is a great place to start because it just works (after I fixed it). Considering our test job took 6 minutes and 7 seconds to run it really leaves a lot to be desired, especially when you consider that the PHPUnit tests only ran for 395 ms.
Why does it take so long?
First a brief introduction into how GitLabCi works and then we’ll dive into the actual contents of the template and see what it is doing.
GitLab CI & Docker Containers
When a GitLab CI job is run, it spins up a docker image with your git repository mounted inside of it. Inside of this container you can do anything you can think of with your code.
The docker image used for the job is defined by image:.
Here we are checking out the latest php image which will run our tests using the latest available php version. See https://hub.docker.com/_/php/
The services: section adds multiple containers that you can connect to via the network.
Here we are asking to spin up a mysql container (used for database seeding / database reliant tests).
services: - mysql:5.7
The variables: section adds environment variables that are injected globally and per job.
Here we provide special variables to help setup our mysql container.
variables: MYSQL_DATABASE: project_name MYSQL_ROOT_PASSWORD: secret
The cache: section defines where to maintain data between subsequent builds.
Here we cache both vendor and node_modules. This should speed up composer / npm installs as there won’t be any code to download.
cache: paths: - vendor/ - node_modules/
The before_script: section defines cli commands that are run before each job script. In this template we are doing a bunch of work to prepare the container to run our unit tests.
before_script: - apt-get update -yqq - apt-get install gnupg -yqq - curl -sL https://deb.nodesource.com/setup_8.x | bash - - apt-get install git nodejs libcurl4-gnutls-dev libicu-dev libmcrypt-dev libvpx-dev libjpeg-dev libpng-dev libxpm-dev zlib1g-dev libfreetype6-dev libxml2-dev libexpat1-dev libbz2-dev libgmp3-dev libldap2-dev unixodbc-dev libpq-dev libsqlite3-dev libaspell-dev libsnmp-dev libpcre3-dev libtidy-dev -yqq - docker-php-ext-install mbstring pdo_mysql curl json intl gd xml zip bz2 opcache - pecl install xdebug - docker-php-ext-enable xdebug - curl -sS https://getcomposer.org/installer | php - php composer.phar install - npm install - cp .env.testing .env - npm run dev - php artisan key:generate - php artisan config:cache - php artisan migrate - php artisan db:seed
The test: section defines our job, names it test, and then runs our phpunit and node tests.
test: script: - php vendor/bin/phpunit --coverage-text --colors=never - npm test
The most time consuming portion of this script is all of the prep and setup in before_script. To test the impact of this portion of the template I ran this manually in a fresh php:latest container and it took over 9 minutes on my laptop for me to run through everything. The most time consuming part of this was installing all of the php modules.
Before we can actually speed up the testing process we need to understand briefly how Docker containers work.
Consider Docker containers as pizzas. Each pizza starts with a crust and then you pile on toppings to create your pizza. With a container you start with a parent image as your crust and then apply additions to that crust to get your pizza. Each time you run a GitLab job you are building another pizza and have to start all over from the crust.
So how can we decrease the time it takes to make the pizza? Following the pizza analogy - we’re going to switch from fresh to frozen pizzas. A frozen pizza is pre-made elsewhere and all you have to do is open it up and pop it in the oven. We can do the same with our docker container by building off our base php:latest container, installing the time consuming php modules, and then freezing the container using GitLab’s Container Registry.
Creating a Docker Image
#We will start from a base image of the latest php version. #If your project has a specific php version requirement you can just specify that rather than latest (FROM php:7.2) FROM php:latest #Run all commands from before_script that we only want to run once RUN apt-get update -yqq RUN apt-get install gnupg -yqq RUN curl -sL https://deb.nodesource.com/setup_8.x | bash - RUN apt-get install git nodejs libcurl4-gnutls-dev libicu-dev libmcrypt-dev libvpx-dev libjpeg-dev libpng-dev libxpm-dev zlib1g-dev libfreetype6-dev libxml2-dev libexpat1-dev libbz2-dev libgmp3-dev libldap2-dev unixodbc-dev libpq-dev libsqlite3-dev libaspell-dev libsnmp-dev libpcre3-dev libtidy-dev -yqq RUN docker-php-ext-install mbstring pdo_mysql curl json intl gd xml zip bz2 opcache RUN pecl install xdebug RUN docker-php-ext-enable xdebug RUN curl -sS https://getcomposer.org/installer | php RUN mv composer.phar /usr/local/bin/composer
In a typical docker container it is advised to group RUN instructions together, however I broke each command into it’s own for legibility.
Within your GitLab project you will see instructions on building your docker image and how to upload it to your registry.
Below are the steps I used to create my docker image and push it to GitLab.
# create Dockerfile cd .gitlab docker build -t registry.gitlab.com/aknosis/laravel-ci-example . #images builds docker push registry.gitlab.com/aknosis/laravel-ci-example
Now that I have an image ready to reference I just need to now use that container in my CI jobs.
#.gitlab-ci.yml image: registry.gitlab.com/aknosis/laravel-ci-example:latest
And remove the commands we executed inside the images.
#.gitlab-ci.yml before_script: - composer install - npm install - cp .env.testing .env - npm run dev - php artisan key:generate - php artisan config:cache - php artisan migrate - php artisan db:seed
Now our tests will run with this new docker image and only run the commands needed to run our tests. Comparing our two test job runs, with the original docker image, and the new docker image you can see we went from a run time of 6 minutes and 7 seconds to 1 minute and 52 seconds. So we’re saving roughy 4 minutes saved per job run.
Here’s the final .gitlab-ci.yml:
image: registry.gitlab.com/aknosis/laravel-ci-example:latest services: - mysql:5.7 variables: MYSQL_DATABASE: project_name MYSQL_ROOT_PASSWORD: secret cache: paths: - vendor/ - node_modules/ before_script: - composer install - npm install - cp .env.testing .env - npm run dev - php artisan key:generate - php artisan config:cache - php artisan migrate - php artisan db:seed test: script: - php vendor/bin/phpunit --coverage-text --colors=never - npm test
Thanks for reading.