Setup the CI using the shared template

../_images/Shared-CI-Infrastructure.png

The Shared CI Infrastructure is project agnostic. It is tuned for open-source projects hosted on GitHub and willing to run CI on a GitLab instance. It currently only supports a handful of Livermore Computing machines, but more could be added without disrupting the design.

The pre-requisite for using the RADIUSS Shared CI Infrastructure, provided that your projects is hosted on GitHub, is that you can trigger your build and test process with one command line.

Strict separation between the build and test process and the CI infrastructure greatly helps with maintenance: it is much easier to debug a standalone script, which can be run outside CI, than when the process is encoded in the CI YAML files.

Note

In our own RADIUSS CI setups, we do not split the build and the test phases because it adds complexity. This typically saves us from using artifacts. However, we do not prevent projects from implementing more complex workflows. Please refer to Implement Complex CI Worflows for more details on multi-steps workflow with RADIUSS Shared CI.

Our RADIUSS projects typically leverage Spack to manage the installation of the dependencies and configure the project. We describe this as a Shared Build Infrastructure documented in RADIUSS Spack Configs. Then, building and testing is handled in custom scripts.

RADIUSS Shared CI

By sharing the CI definition, projects share the burden of maintaining it.

With a centralized CI configuration that is shared by projects, we create an interface between local and shared configurations. We try to keep this interface minimal, while allowing for project-specific customization.

Projects only need to set some variables and add CI jobs instances (inheriting from the job template) to get the CI to work. Files in the customization directory allow for fine tuning and extensibility.

Note

GitLab allows projects to include external files to configure their CI. We rely on this mechanism to share most of the CI configuration among projects.

File structure

../_images/Shared-CI-File-Structure.png

The RADIUSS Shared CI repository contains files that are included remotely as well as template files that needs to be copied over and completed. Each pipeline file also defines a job template (in the sense of GitLab CI YAML syntax) providing the core settings that your own jobs will extend.

The short version

The following sequence of steps and commands summarizes the process of integrating the RADIUSS Shared CI infrastructure into your project.

### Prerequisites
# write CI script

### CI Setup
cd my_project/..
git clone https://github.com/LLNL/radiuss-shared-ci.git
cd my_project
cp ../radiuss-shared-ci/customization/gitlab-ci.yml .gitlab-ci.yml
mkdir -p .gitlab/jobs
cp ../radiuss-shared-ci/customization/subscribed-pipelines.yml .gitlab
cp ../radiuss-shared-ci/customization/custom-jobs-and-variables.yml .gitlab
cp ../radiuss-shared-ci/customization/jobs/\<machine\>.yml .gitlab/jobs/<machine>.yml
# You may use the <machine>.yml file as a starting point to add jobs.
vim .gitlab/subscription-pipelines.yml
# comment the jobs associted to <CI_MACHINE> you don't want.
vim .gitlab/custom-jobs-and-variables.yml
# set the variables according to your needs.
vim .gitlab/jobs/<machine>.yml
# Add jobs or override some of the shared ones.

### Mirroring Setup
https://docs.gitlab.com/ee/ci/ci_cd_for_external_repos/github_integration.html#connect-with-personal-access-token

### Non-RADIUSS projects
open https://lc.llnl.gov/gitlab/<group>/<project>/-/settings/ci_cd
# Set CI/CD variable GITHUB_TOKEN to hold token with repo:status
# permissions.

Jump to a corresponding section for details on Customize the CI, Add jobs.

The detailed version

Our CI implementation can be divided in four parts:

  • local build-and-test script

  • shared files

  • customization files

  • jobs

Setting up the CI consists of four corresponding steps.

Write CI Script

The first step is to prepare a CI script that will be called using the JOB_CMD variable in the CI. Once you have that script, you are ready to move on to the CI setup.

Core CI implementation

Start by cloning the RADIUSS Shared CI project locally, for example next to the project you intend to add CI to.

cd my_project/..
git clone https://github.com/LLNL/radiuss-shared-ci.git
cd my_project

By default, GitLab expects a .gitlab-ci.yml file to interpret the CI setup. We provide one in customization/gitlab-ci.yml that projects can copy over (don’t forget to add a .). Place the file in the top-level directory of your project.

cp ../radiuss-shared-ci/customization/gitlab-ci.yml .gitlab-ci.yml

In the .gitlab-ci.yml file, there are some variables that you need to adapt to your project. They are described in the following table:

Parameter

Description

LLNL_SERVICE_USER

Project specific Service User Account used in CI (optional but recommeded)

CUSTOM_CI_BUILD_DIR

If not using a service user, where to locate the CI working directories (prevent exceeding your disk quota)

GIT_SUBMODULES_STRATEGY

Controls strategy for the clone performed by GitLab. Consider recursive if you have submodules, otherwise comment it.

BUILD_ROOT

Location (path) where the projects should be built. We provide a sensible default.

SHARED_CI_REF

The reference (branch, tag) you would like to use in RADIUSS Shared CI repository

GITHUB_PROJECT_NAME

The Project name on GitHub, used to send status updates

GITHUB_PROJECT_ORG

The Project organization on GitHub, used to send status updates

JOB_CMD

The command that runs the build and test script. Lets you name and store that script however you like.

ALWAYS_RUN_PATTERN

The regex pattern describing the branches that will skip the draft pull request filter test.

Note

If a variable is blank in the template file, then it does not require a value. If a variable has a value there (even “…”), it does require one.

Warning

We strongly recommend that you set your CI to use a service user account. This will enable you to add users to the associated service user account group so that they can interact with GitLab runners to restart test pipelines, for example. It will also simplify permissions and allocations management.

Your CI is now set up to include remote files from the GitLab mirror of the radiuss-shared-ci project.

We now need to complete the interface with the shared CI configuration. In particular, the .gitlab-ci.yml file requires some files to be present in your Git repository. These are described in the next few sections.

Customize the CI

We provide templates for the required customization files. You need to have a .gitlab subdirectory in the top-level of your Git repository. Then, you can copy the template files to that directory in your repo. For example:

mkdir -p .gitlab
cp ../radiuss-shared-ci/customization/subscribed-pipelines.yml .gitlab
cp ../radiuss-shared-ci/customization/custom-jobs-and-variables.yml .gitlab

We will now browse the files to see what changes they may require to suit your needs.

The .gitlab/subscribed-pipelines.yml file

In this file, you will select the machines you want to run tests on. Comment out the jobs (sections) for machines you don’t want, or don’t have access to.

Note

To add a new machine, please refer to Add a new machine.

The .gitlab/custom-jobs-and-variables.yml file

Here is a table that describes each variable present in the file. More details can be found in the file itself.

Parameter

Description

ALLOC_NAME

Name of the shared allocation. Should be unique, our default should be fine.

<MACHINE>_SHARED_ALLOC

Parameters for the shared allocation. You may extend the resource and time.

<MACHINE>_JOB_ALLOC

Parameters for the job allocation. You may extend the resource and time within the scope of the shared allocation.

PROJECT_<MACHINE>_VARIANTS

Global variants to be added to all the shared specs.

PROJECT_<MACHINE>_DEPS

Global dependencies to be added to all the shared specs.

Note

If a variable is blank in the template file, then it does not require a value. If a variable has a value there, it does require one.

You may modify to the .custom_job job that will then be included in all you CI jobs. This can be used to export jUnit test reports, for example. Changes to that section are not mandatory.

Add jobs

We provide a template file to add jobs to each machine. You should create one file per machine using this template. These files may be placed in your project’s .gitlab/jobs subdirectory and named <machine>.yml, where <machine> is the machine name. They are required as soon as the associated machine has been activated (uncommented) in the .gitlab/subscribed-pipelines file.

In the provided template, you may remove the variable definition, uncomment and duplicate the example job and complete it with the required information:

  • Unique job names that will appear in CI.

  • Custom variables that will make that job unique.

Warning

GitLab supports long and complex job names. Make sure to pick names that are unique so that your extra jobs do not override a shared job.

Note

It is possible to import jobs from another repository. This is what we do in our RADIUSS projects to share some jobs and thus make sure we build with the same toolchains. See the dedicated How-To section for more details Import shared jobs.

Mirroring Setup

RADIUSS Shared CI is primarily intended for projects hosted on GitHub that need to run tests on LLNL Livermore Computing (LC) systems through the LC GitLab instance. GitLab provides a mirroring feature with GitHub integration that will automate the synchronisation of local source with the remote GitHub repository.

The mirroring setup is described in detail in GitLab documentation_:

Non-RADIUSS Projects

RADIUSS Shared CI features a customized status report mechanism that reports to to GitHub when a machine is down, making it impossible to run the pipeline. This prevent users from having to connect to GitLab to find out.

If your project hasn’t been mirrored within the Radiuss group on LC GitLab, this feature will only work after you create a GitHub token with repo:status permissions, and registering it as a CI/CD variable named GITHUB_TOKEN in the project (or the group) on GitLab.

Visit https://lc.llnl.gov/gitlab/<group>/<project>/-/settings/ci_cd to create the variable once the token has been generated on GitHub.