Speeding up GitLab CI Laravel Tests with a Custom Docker Image

Nov 10, 2018

This is a follow up to the previous post Continuous Integration for your Laravel Application with GitLab CI.

In this article we’ll run through what the costs are of using the template, reviewing the template contents and how we can use our own docker image to speed up the job run time by over 200%.

Reliability at the Cost of Speed

The template that ships in GitLab is a great place to start because it just works (after I fixed it). Considering our test job took 6 minutes and 7 seconds to run it really leaves a lot to be desired, especially when you consider that the PHPUnit tests only ran for 395 ms.

Why does it take so long?

First a brief introduction into how GitLabCi works and then we’ll dive into the actual contents of the template and see what it is doing.

GitLab CI & Docker Containers

When a GitLab CI job is run, it spins up a docker image with your git repository mounted inside of it. Inside of this container you can do anything you can think of with your code.

The docker image used for the job is defined by image:.

Here we are checking out the latest php image which will run our tests using the latest available php version. See https://hub.docker.com/_/php/

image: php:latest

The services: section adds multiple containers that you can connect to via the network.

Here we are asking to spin up a mysql container (used for database seeding / database reliant tests).

services:
  - mysql:5.7

The variables: section adds environment variables that are injected globally and per job.

Here we provide special variables to help setup our mysql container.

variables:
  MYSQL_DATABASE: project_name
  MYSQL_ROOT_PASSWORD: secret

The cache: section defines where to maintain data between subsequent builds.

Here we cache both vendor and node_modules. This should speed up composer / npm installs as there won’t be any code to download.

cache:
  paths:
  - vendor/
  - node_modules/

The before_script: section defines cli commands that are run before each job script. In this template we are doing a bunch of work to prepare the container to run our unit tests.

before_script:
  - apt-get update -yqq
  - apt-get install gnupg -yqq
  - curl -sL https://deb.nodesource.com/setup_8.x | bash -
  - apt-get install git nodejs libcurl4-gnutls-dev libicu-dev libmcrypt-dev libvpx-dev libjpeg-dev libpng-dev libxpm-dev zlib1g-dev libfreetype6-dev libxml2-dev libexpat1-dev libbz2-dev libgmp3-dev libldap2-dev unixodbc-dev libpq-dev libsqlite3-dev libaspell-dev libsnmp-dev libpcre3-dev libtidy-dev -yqq
  - docker-php-ext-install mbstring pdo_mysql curl json intl gd xml zip bz2 opcache
  - pecl install xdebug
  - docker-php-ext-enable xdebug
  - curl -sS https://getcomposer.org/installer | php
  - php composer.phar install 
  - npm install
  - cp .env.testing .env
  - npm run dev
  - php artisan key:generate
  - php artisan config:cache
  - php artisan migrate
  - php artisan db:seed

The test: section defines our job, names it test, and then runs our phpunit and node tests.

test:
  script:
  - php vendor/bin/phpunit --coverage-text --colors=never 
  - npm test

The most time consuming portion of this script is all of the prep and setup in before_script. To test the impact of this portion of the template I ran this manually in a fresh php:latest container and it took over 9 minutes on my laptop for me to run through everything. The most time consuming part of this was installing all of the php modules.

Understanding Docker

Before we can actually speed up the testing process we need to understand briefly how Docker containers work.

Consider Docker containers as pizzas. Each pizza starts with a crust and then you pile on toppings to create your pizza. With a container you start with a parent image as your crust and then apply additions to that crust to get your pizza. Each time you run a GitLab job you are building another pizza and have to start all over from the crust.

So how can we decrease the time it takes to make the pizza? Following the pizza analogy - we’re going to switch from fresh to frozen pizzas. A frozen pizza is pre-made elsewhere and all you have to do is open it up and pop it in the oven. We can do the same with our docker container by building off our base php:latest container, installing the time consuming php modules, and then freezing the container using GitLab’s Container Registry.

Creating a Docker Image

Docker images are defined by files called Dockerfile. Here is my Dockerfile used in this example, also committed to the repository.

#We will start from a base image of the latest php version. 
#If your project has a specific php version requirement you can just specify that rather than latest (FROM php:7.2)
FROM php:latest

#Run all commands from before_script that we only want to run once
RUN apt-get update -yqq

RUN apt-get install gnupg -yqq

RUN curl -sL https://deb.nodesource.com/setup_8.x | bash -

RUN apt-get install git nodejs libcurl4-gnutls-dev libicu-dev libmcrypt-dev libvpx-dev libjpeg-dev libpng-dev libxpm-dev zlib1g-dev libfreetype6-dev libxml2-dev libexpat1-dev libbz2-dev libgmp3-dev libldap2-dev unixodbc-dev libpq-dev libsqlite3-dev libaspell-dev libsnmp-dev libpcre3-dev libtidy-dev -yqq

RUN docker-php-ext-install mbstring pdo_mysql curl json intl gd xml zip bz2 opcache
RUN pecl install xdebug

RUN docker-php-ext-enable xdebug

RUN curl -sS https://getcomposer.org/installer | php

RUN mv composer.phar /usr/local/bin/composer

In a typical docker container it is advised to group RUN instructions together, however I broke each command into it’s own for legibility.

Within your GitLab project you will see instructions on building your docker image and how to upload it to your registry.

Below are the steps I used to create my docker image and push it to GitLab.

# create Dockerfile
cd .gitlab
docker build -t registry.gitlab.com/aknosis/laravel-ci-example .
#images builds
docker push registry.gitlab.com/aknosis/laravel-ci-example

Now that I have an image ready to reference I just need to now use that container in my CI jobs.

#.gitlab-ci.yml
image: registry.gitlab.com/aknosis/laravel-ci-example:latest

And remove the commands we executed inside the images.

#.gitlab-ci.yml
before_script:
  - composer install 
  - npm install
  - cp .env.testing .env
  - npm run dev  - php artisan key:generate
  - php artisan config:cache
  - php artisan migrate
  - php artisan db:seed

Analyzing Results

Now our tests will run with this new docker image and only run the commands needed to run our tests. Comparing our two test job runs, with the original docker image, and the new docker image you can see we went from a run time of 6 minutes and 7 seconds to 1 minute and 52 seconds. So we’re saving roughy 4 minutes saved per job run.

Here’s the final .gitlab-ci.yml:

image: registry.gitlab.com/aknosis/laravel-ci-example:latest

services:
  - mysql:5.7

variables:
  MYSQL_DATABASE: project_name
  MYSQL_ROOT_PASSWORD: secret

cache:
  paths:
  - vendor/
  - node_modules/

before_script:
  - composer install 
  - npm install
  - cp .env.testing .env
  - npm run dev
  - php artisan key:generate
  - php artisan config:cache
  - php artisan migrate
  - php artisan db:seed

test:
  script:
  - php vendor/bin/phpunit --coverage-text --colors=never
  - npm test

Thanks for reading.

If you are looking to make a change in your career consider checking out our job listings, subscribing to our newsletter, and following us on Twitter @__phpJobs__.