CI/CD Optimization
with a Case Study Involving Ansible and GitLab

Presented by Zackary Lowery
on October 24th, 2023 at Leading EDJE.

Available online at https://presentations.xcjs.com/

Press your space bar or swipe to navigate down and then forward. »

...or How to Write a Bad CI/CD Pipeline and Improve it Iteratively

In Review

This presentation is a spiritual follow-up to Ansible: Putting IaC in Your CI/CD.

What is Ansible?

Ansible is a state-management based IaC solution for host configuration.

https://www.ansible.com/

A Note on Ansible Concepts

Inventory

A list of network-accessible hosts.

Role

Unit of software or configuration state.

⬊

⬋

Playbook

Assign reusable roles to selected hosts.

How am I using Ansible?

I...have a lot of hosts.

(Still.)

351MP
AERO15
K8S_[0-4]
Lakka
MDEV
MDEV2
NAS
PlexBox
Runner
UDEV3
USERV
XCJS
Z390

Integration Testing Options

ansible-test (Containers)

✅	🚫
First-party Native performance	Limited OS and distribution support

ansible-test (Cloud VMs)

✅	🚫
First-party Native performance	Monetary cost Limited provider support

Docker

✅	🚫
Native performance	CI/CD runner was shared with production Docker host Complexity of Docker in Docker for Docker-based roles Limited OS support (Linux only on Linux hosts)

Vagrant

✅	🚫
Most encapsulated/isolated Nearly unlimited OS support	Slow Performance-intensive

Iteration 1

The Naive Approach

All roles are tested against every supported OS on every feature branch push.
A single VM is used for every role per supported OS.

153 roles as of writing multiplied by

18.04

20.04

22.04

Abbreviated Iteration 1 Pipeline

							flowchart LR
							A[Start] --> B{Next OS?};
							B -- Yes --> C[Create VM];
							B -- No --> D[End];
							C --> E[Provision VM];
							E --> F{Next Role?};
							F -- Yes --> G[VM Applies Role];
							F -- No --> H[Destroy VM];
							G --> F;
							H --> B;

The Problem

...or just over 27 hours.

...to fail.

Iteration 1 Pipeline Highlights

Every role is installed to the same VM.
Vagrant VMs are limited to 50 GB by default.
Disk I/O is insane, leading to...

☠️ The Other Problem ☠️

😭

Yet Another Reminder

⚠️ If it's important, back it up! ⚠️

Iteration 2

CI/CD Goals

Reduce Redundant Testing

Resolutions

Perform diff-detection on Ansible roles.
Dynamically iterate over roles (We'll come back to this later.)
Test each role in a separate VM.

Implement GitLab CI/CD Rules:Changes Directive

Not to be confused with the obsoleted changes directive
Requires a long-running branch to compare against

Rules:Changes Example


							${role} Test:
								stage: Test
								rules:
								- if: '$CI_COMMIT_BRANCH == "testing"'
								  changes:
								    - roles/.common/tasks/*.yml
								    - roles/${role}/tasks/*.yml
								    - test/generate-tests.sh
								    - .gitlab-ci.yml
								    - Vagrantfile

Abbreviated Iteration 2 Pipeline

							flowchart LR
							A[Start] --> B{Next Role?};
							B -- Yes --> C{Next VM?};
							B -- No --> D[End];
							C -- Yes --> E[Create VM];
							C -- No --> B;
							E --> G[Provision VM];
							G --> H[VM Applies Role];
							H --> I[Destroy VM];
							I --> B{Next Role?};

Iteration 2 Pipeline Highlights

Only roles altered since the last release are tested.
VMs have to be created and destroyed per-role.
Disk I/O is down significantly, but still higher than it needs to be.

Iteration 3

CI/CD Goals

Utilize separation of concerns to further reduce repeat testing.

Resolutions

Remove repeat testing of supported operating
systems that have already passed testing for a specific role.

Rules:Changes Updates


							${role}/Ubuntu 20.04:
								stage: Test
								rules:
								  - if: '$CI_COMMIT_BRANCH == "testing"'
								    changes:
								      - roles/.common/tasks/*.yml
								      - roles/${role}/tasks/main.yml
								      - roles/${role}/tasks/linux.yml
								      - roles/${role}/tasks/ubuntu_20.04.yml
								      - test/templates/ubuntu_20.04/.gitlab-ci.part.yml
								      - test/templates/ubuntu_20.04/blacklist.yml
								      - test/templates/ubuntu_20.04/site-test.yml
								      - test/generate-tests.sh
								      - .gitlab-ci.yml
								      - Vagrantfile

Abbreviated Iteration 3 Flow

							flowchart LR
							A[Start] --> B{Role Changed?};
							B -- Yes --> C{Next OS?};
							B -- No --> J[End];
							C -- Yes --> D[Prepare VM Disk];
							C -- No --> B;
							D --> E[Provision VM];
							E --> F[Start VM];
							F --> G[Apply Role];
							G --> H[Destroy VM];
							H --> C;

Iteration 3 Highlights

Each task list in a role is for a specific operating system.
Diff detection now works on a per-role and per-operating-system basis.

Iteration 4

CI/CD Goals

Know your tooling!

Resolutions

Prepare virtual machine images once and revert after each role is applied.

Abbreviated Iteration 4 Flow

Setup Loop

							flowchart LR
							A[Start] --> B{Next OS?};
							B -- Yes --> C[Create VM];
							B -- No --> D[Start Testing];
							C --> E[Provision VM];
							E --> F[VM Disk Snapshot];
							F --> G[Halt VM];
							G --> B;

Abbreviated Iteration 4 Flow

Testing Loop

							flowchart LR
							A[Start] --> B{Role Changed?};
							B -- Yes --> C[Restore Disk Snapshot];
							B -- No --> G[End]
							C --> E[Apply Role];
							E --> F[Halt VM];
							D -- Yes --> C;
							D -- No --> B;
							F --> D{Next OS?};

Iteration 4 Highlights

VMs are now reused for each role with a snapshot that restores the VM to its state prior to having the previous role installed.

Results!

Wait!

How am I Iterating over Roles in the First Place?

Abusing GitLab CI/CD by Bash-ing it to Death

...and Taking Advantage of GitLab's Child Pipeline Feature

bash Script File Icon ./test/generate-tests.sh

Supported Operating Systems are Stored in a bash Array


							supportedOperatingSystems=(
								"ubuntu_18.04"
								"ubuntu_20.04"
								"ubuntu_22.04"
								"windows_10"
								"windows_11"
							)

Supported Roles are Iterated from the Filesystem


							for path in roles/*/; do # Whitespace-safe
								role=$(basename "${path}")
							done

Supported Operating Systems are Iterated per Role


							for os in "${supportedOperatingSystems[@]}"
							do
								templateDir="test/templates/${os}/"
								...
							done

I could directly iterate the template directories, but I wanted the option to easily sunset formerly supported operating systems.

OS-Specific Task List Files are Verified Per Role


							if [ -f "${path}/tasks/${os}.yml" ]
						      && ! containsElement "${role}" "${roleBlacklist[@]}"; then
							  ...
							endif

Append to the Child Pipeline Using a Template

(Using `envsubst`)


							echo "Role ${role} supports ${os} and will be added to CI/CD."

							gitlabRoleCi=$(envsubst '$role'
							  < "test/templates/${os}/.gitlab-ci.part.yml")

							echo "${gitlabRoleCi}" >> ".gitlab-ci-roles.yml"

GitLab Provided Diff Detection

.gitlab-ci.part.yml templates include rules:changes directives:


						  - if: '$CI_COMMIT_BRANCH == "testing"'
						    changes:
						      - roles/.common/tasks/*.yml
						      - roles/${role}/tasks/main.yml
						      - roles/${role}/tasks/linux.yml
						      - roles/${role}/tasks/ubuntu_18.04.yml
						      - test/templates/ubuntu_18.04/.gitlab-ci.part.yml
						      - test/templates/ubuntu_18.04/blacklist.yml
						      - test/templates/ubuntu_18.04/playbook-test.yml
						      - test/generate-tests.sh
						      - .gitlab-ci.yml
						      - Vagrantfile

Registering the Child Pipeline (Parent)


						  Generate Tests:
						    stage: Test Setup
						    artifacts:
						      name: gitlab-ci-roles
						      expire_in: 1 week
						      paths:
						        - .gitlab-ci-roles-ubuntu_18.04.yml
						        - .gitlab-ci-roles-ubuntu_20.04.yml
						        - .gitlab-ci-roles-ubuntu_22.04.yml
						        - .gitlab-ci-roles-windows_10.yml
						        - .gitlab-ci-roles-windows_11.yml
						        - .vagrant/

Registering the Child Pipeline (Child)


						  {$role}/{$os} Test:
						    Stage: Test
						    needs:
						      - pipeline: $PARENT_PIPELINE_ID
						        job: Generate Tests

Additional Enhancements

Use Language Tooling for CI/CD Resources

shellcheck allows us to fail early!


						  Lint Test Generation:
						    stage: Lint
						    tags:
						      - linux
						      - shell
						      - shellcheck
						    script:
						      - shellcheck test/generate-tests.sh

Trends in DevOps

Alternatives to shell scripting are becoming more common for more complex pipelines.

Python
Powershell
C# Scripting
Node.js

Throw More Hardware at the Problem!

Future Iterations?

Persist VM snapshots across pipelines and update them on box updates?
```
vagrant box update
```
Maintain my own base VM images?
Support multiple testing strategies?
- Containers for simple Linux-supported roles?
- Virtual machines for tests requiring isolation and non-Linux hosts?

End (of the Pipeline)

Questions? Suggestions? Rude Insults?

Return to the rest of the presentations.

CI/CD Optimization with a Case Study Involving Ansible and GitLab

In Review

What is Ansible?

A Note on Ansible Concepts

Inventory

Role

Playbook

How am I using Ansible?

Integration Testing Options

ansible-test (Containers)

ansible-test (Cloud VMs)

Docker

Vagrant

Iteration 1

The Naive Approach

Abbreviated Iteration 1 Pipeline

The Problem

Iteration 1 Pipeline Highlights

☠️ The Other Problem ☠️

Yet Another Reminder

Iteration 2

CI/CD Goals

Resolutions

Implement GitLab CI/CD Rules:Changes Directive

Rules:Changes Example

Abbreviated Iteration 2 Pipeline

Iteration 2 Pipeline Highlights

Iteration 3

CI/CD Goals

Resolutions

Rules:Changes Updates

Abbreviated Iteration 3 Flow

Iteration 3 Highlights

Iteration 4

CI/CD Goals

Resolutions

Abbreviated Iteration 4 Flow

Setup Loop

Abbreviated Iteration 4 Flow

Testing Loop

Iteration 4 Highlights

Results!

Wait!

How am I Iterating over Roles in the First Place?

Abusing GitLab CI/CD by Bash-ing it to Death

...and Taking Advantage of GitLab's Child Pipeline Feature

Supported Operating Systems are Stored in a bash Array

Supported Roles are Iterated from the Filesystem

Supported Operating Systems are Iterated per Role

OS-Specific Task List Files are Verified Per Role

Append to the Child Pipeline Using a Template

(Using envsubst)

GitLab Provided Diff Detection

Registering the Child Pipeline (Parent)

Registering the Child Pipeline (Child)

Additional Enhancements

Use Language Tooling for CI/CD Resources

Trends in DevOps

Throw More Hardware at the Problem!

Future Iterations?

End (of the Pipeline)

CI/CD Optimization
with a Case Study Involving Ansible and GitLab

(Using `envsubst`)