CI/CD pipeline is a piece of automation that resided outside of usual developers’ duties. On the other side, it’s an essential component of any system. Today, we’ll see how to simplify life with a correctly set up pipeline.
Outcome
By the end of this part, we build two pipelines. The first one will build binaries for every pull request (PR) whenever it is created or updated. The second one will make Docker images with those binaries inside and publish them into the Github container registry when we push a tag into our git repo.
For those who are eager to see the code right now, it’s here. And here is a commit that adds CI/CD stuff.
Pipeline for PR
Every time we create a PR, we must validate that the new feature doesn’t break anything else. It’s crucial, primarily when working with monorepo, where we can have tens of loosely coupled or not coupled at all microservices.
For such verification, it’s nice to run a bunch of unit tests; the issue is that not all projects, including the current one at the time of writing this post, have unit tests.
As a backup verification method, we will build everything. This way, we verify that if all microservices are buildable, we can deploy them. It doesn’t verify all the code work as expected, only tests can prove it.
Workflow
To run anything automatically, we have to create a workflow.
Workflow
is a series of jobs that run in parallel by default. It’s also possible to run
them sequentially. Workflow is defined as a YAML file and should be resided in
the .github/workflows
directory.
Here is a build.yml workflow:
name: Build
on: [ pull_request ]
jobs:
build:
runs-on: ubuntu-latest
steps:
- name: Cache bazel
uses: actions/cache@v2
env:
cache-name: bazel-cache
with:
path: |
~/.cache/bazelisk
~/.cache/bazel
key: ${{ runner.os }}-${{ env.cache-name }}
- name: Setup Bazelisk
uses: bazelbuild/setup-bazelisk@v1.0.1
- name: Checkout sources
uses: actions/checkout@v2
- name: Build everything
run: bazel build //...
# - name: Test everything
# run: bazel test //...
let’s look a bit closer:
name: Build
- workflow name, which appears on Actions page in the workflow list.on: [ pull_request ]
- event that triggers our workflow, pull_request in our case. All events are listed in docs.jobs:
- is a list of jobs for the workflow.build:
- is a job name that appears on action page details.runs-on: ubuntu-latest
- we specify that we will run build on Ubuntu machine, a virtual machine provided by Github. All available environments are listed here.name: Cache bazel
- is a job step that restores Bazel cache if one exists. This cache is updated after each successful workflow run. Also, it saves a lot of time, for the current project build from scratch takes ~6m30s, but with cache, it takes only ~15s.name: Setup Bazelisk
- Bazel already pre-installed onto Ubuntu machine, but Bazelisk isn’t. So, here we install it.name: Checkout sources
- check out our source code.name: Build everything
- runsbazel build //...
.name: Test everything
- run tests, but this project doesn’t have it, so, just skipping.
That’s it. Now, if something goes wrong, we will know it after creating or updating a PR.
Pipeline for Tag
After we finish all of the work on the PR, we probably, want to publish Docker images somewhere. For this example, I use the Github container registry. To publish images, we have to create a push git tag. When the tag is pushed, release.yml will be run:
name: Release
on:
push:
tags:
- "v*"
jobs:
publish_images:
runs-on: ubuntu-latest
steps:
- name: Cache bazel
uses: actions/cache@v2
env:
cache-name: bazel-cache
with:
path: |
~/.cache/bazelisk
~/.cache/bazel
key: ${{ runner.os }}-${{ env.cache-name }}
- name: Setup Bazelisk
uses: bazelbuild/setup-bazelisk@v1.0.1
- name: Checkout sources
uses: actions/checkout@v2
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: Push service-one image to ghcr.io
run: bazel run --define=IMAGE_TAG=${{ github.ref_name }} --stamp //service-one:push_image
- name: Push authz image to ghcr.io
run: bazel run --define=IMAGE_TAG=${{ github.ref_name }} --stamp //authz:push_image
- name: Push envoy image to ghcr.io
run: bazel run --define=IMAGE_TAG=${{ github.ref_name }} --stamp //k8s/envoy:push_image
This one is similar to the Build
workflow. So, I’ll list the differences only:
on:
push:
tags:
- "v*"
- as a trigger, we specify a
push
event, which will be started whenever we pushtags
started withv
.
- name: Login to GitHub Container Registry
uses: docker/login-action@v1
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- then, we need to log in to the registry itself. Here:
registry: ghcr.io
points to GitHub container registry.${{ github.actor }}
provided by the context and returns a user who run the workflow.${{ secrets.GITHUB_TOKEN }}
is pre-defined.
- name: Push service-one image to ghcr.io
run: bazel run --define=IMAGE_TAG=${{ github.ref_name }} --stamp //service-one:push_image
- after we logged in, we can run
push_image
target for every image we want to push.
the target itself
container_push(
name = "push_image",
format = "Docker",
image = ":image",
registry = "ghcr.io",
repository = "ekhabarov/service-one",
tag = "$(IMAGE_TAG)",
)
defines what to push with image = ":image"
and where to push it with
registry
/repository
params. The image will have a tag, the value of which
comes from outside, --define=IMAGE_TAG=${{ github.ref_name }}
. Without
--stamp
flag, the image will have a default value from
.bazelrc.
More about stamping
here.
Now, when we create and push git tag, like v0.0.1
, the workflow will run Bazel,
which in turn push all images to the registry.