A powerful, easily deployable network traffic analysis tool suite
Using GitHub runners to build Malcolm images
Users who have forked Malcolm on GitHub can use GitHub-hosted runners to build Malcolm docker images, as well as sensor and Malcolm installer ISOs, and push those images to GitHub’s ghcr.io container registry.
The resulting images are named according to the pattern ghcr.io/username/malcolm/image:branch (e.g., if the user has forked Malcolm with the GitHub user romeogdetlevjr, the Arkime container built for the main branch would be named ghcr.io/romeogdetlevjr/malcolm/arkime:main). To run a local instance of Malcolm using these images instead of the official ghcr.io/idaholab ones, users will need to edit their docker-compose.yml file(s) and replace the image: tags according to this new pattern, or use the Bash helper script ./shared/bin/github_image_helper.sh to pull and re-tag the images.
Workflow files are provided that contain instructions to build the docker images using GitHub-hosted runners, as well as sensor and Malcolm installer ISOs.
This document outlines the steps needed to configure and use GitHub runners to build Malcolm images.
GitHub actions
Configuring GitHub actions
Users should log into GitHub and navigate to their Malcolm fork. Click ⚙ Settings along the top of the page, then open the ▶ Actions menu from the left panel and select General.
Select Allow username, and select non-username, actions and reusable workflows and check the boxes for Allow actions created by GitHub and Allow actions by Marketplace verified creators. Malcolm’s workflows also use a third-party action (styfle/cancel-workflow-action) for canceling running workflows when new ones are started, so add the text styfle/cancel-workflow-action@*, under Allow specified actions and reusable workflows. Click Save underneath that section.
Under Fork pull request workflows from outside collaborators, select Require approval for all outside collaborators (or read Approving workflow runs from public forks and make another selection). Click Save for that section.
Under Workflow permissions, select Read and write permissions to allow the GitHub workflow actions to write packages to the container registry. Malcolm workflows do not create and approve pull request, so Allow GitHub Actions to create and approve pull requests may be left unchecked. Click Save for that section.
Secrets and variables
Expand ✴ Secrets and variables in the left menu panel under Security, then select Actions from that menu.
Malcolm uses MaxMind’s free GeoLite2 databases for GeoIP lookups. As of December 30, 2019, these databases are no longer available for download via a public URL. Instead, they must be downloaded using a MaxMind license key (available without charge from MaxMind). This license key should be specified here as a repository secret so that the workflows may download the GeoLite2 databases to embed into the container images that use them. Keep in mind that MaxMind has a daily rate for the number of downloads allowed per license key.
Click New repository secret and create a secret with the name MAXMIND_GEOIP_DB_LICENSE_KEY and the Secret value containing the user’s free license key obtained from MaxMind. Then click Add secret. Click New repository secret and create a secret with the name MAXMIND_GEOIP_DB_LICENSE_KEY and the Secret value containing the user’s free license key obtained from MaxMind. Then click Add secret.
Enabling workflows
Click the ▶ Actions tab along the top of the page. Users will be prompted with this message:
If desired, the Malcolm workflow files may be reviewed here prior to enabling them. Click I understand my workflows, go ahead and enable them to enable these workflows on the user’s Malcolm fork.
Testing a workflow run
Once workflows have been enabled, users are presented with a view of past workflow runs, which will initially be empty:
Initiate a test workflow run by clicking on a workflow from the menu on the left (dirinit-build-and-push-ghcr is one that will build quickly). Expand the Run workflow ⛛ menu on the right hand side of the page, then click the Run workflow button to initiate the build.
Users will be presented with a Workflow run was successfully requested message and will notice that a workflow run is in progress:
Depending on which workflow was run – some of Malcolm’s workflows will complete in less than a minute, while others may take up to an hour to finish – the icon to the left of the workflow run should show a white check mark with a green background (✅) to indicate it completed successfully. Clicking on the name of the workflow run (e.g., dirinit-build-and-push-ghcr) will show details about its execution:
Further details, including the output of the build jobs themselves, can be examined by clicking on a job from the menu on the left side of the screen and expanding the sections of interest that are displayed on the right. Most of the details relevant to a build will be under a step called Build and push.
Verifying the workflow run
As the product of Malcolm’s workflow runs are container images pushed to GitHub’s ghcr.io container registry, users can verify the image was built and pushed by navigating back to the main page of their Malcolm fork and clicking Packages from the right panel of that page, or by navigating to https://github.com/username?tab=packages&repo_name=Malcolm. A list of the packages published is displayed:
Clicking on a package name will show details about that package. Note that most Malcolm container images are built with two tags, one for x86-64 platforms tagged with the branch name (main) and another for arm64 platforms tagged with the branch name and -arm64 (main-arm64).
Modifying workflow files
Triggers
Malcolm’s workflow files are configured to build when any of the following triggers occur (found in the on: section at the top of each workflow file):
- Per container when files relevant to that container are pushed to the repository
- The file ‘.trigger_workflow_build’ may be modified and pushed (its contents are irrelevant) to trigger all containers’ builds.
- Per container upon a workflow dispatch (e.g., when clicking the Run workflow button as described in the example above or via API)
- Repository-wide upon a repository dispatch executed via API
These triggers may be modified according to the users’ needs.
Platforms
Each container build workflow actually runs two paths in parallel: one for building and pushing the x86-64 images and one for building and pushing the arm64 images. As of the writing of this document, Arm-based Linux runners are not yet publicly available (but are expected to be offered for open source and personal accounts by the end of 2024). For this reason, the arm64 builds are emulated with QEMU. These emulated builds take much longer than their native x86-64 counterparts. Users who do not need the arm64 images may comment out that platform (by adding a # to the beginning of its line) under jobs.docker.strategy.matrix.platform in the workflow YML file.
Convenience scripts for development
As mentioned earlier, Malcolm images built using the instructions in this document are are named according to the pattern ghcr.io/username/malcolm/image:branch. However, note that the image: values found in docker-compose.yml (and in the Kubernetes manifests) look like ghcr.io/idaholab/malcolm/opensearch:24.06.0, using the OpenSearch container as an example. To run a local instance of Malcolm using these images instead of the official ghcr.io/idaholab ones, users will need to edit their docker-compose.yml file(s) and replace the image: tags according to this new pattern, or use the bash helper script ./scripts/github_image_helper.sh to pull the repository images and re-tag them with ghcr.io/idaholab and the current Malcolm version (e.g., 24.06.0).
Before explaining that script, a discussion of the workflow files for the Hedgehog Linux (hedgehog-iso-build-docker-wrap-push-ghcr.yml
) and Malcolm (malcolm-iso-build-docker-wrap-push-ghcr.yml
) installer ISOs is warranted. The installer images are ISO 9660-formatted files, not docker container images, so one may reasonably wonder about the purpose of the ghcr.io/username/malcolm/malcolm:main and ghcr.io/username/malcolm/hedgehog:main images pushed to ghcr.io.
Examining malcolm-iso/Dockerfile and hedgehog-iso/Dockerfile, one will see that these container images are just thin wrappers around the ISO 9660 image files built during their respective workflows. These wrapper images serve two purposes:
- To provide an HTTP server from which the ISO itself can be downloaded
- To boot a live ISO image in QEMU (based on
tianon/qemu:native)
Since the Malcolm Malcolm (malcolm-iso-build-docker-wrap-push-ghcr.yml ) installer ISO workflow also includes a snapshot of the rest of the Malcolm container images (built from the user’s Malcolm fork as described above) in its file system, it should not be executed until all of the other container images have been built and pushed at least once. In other words, the Malcolm installer ISO will contain the container images from the fork’s most recent workflow runs.
Running github_image_helper.sh presents the following menu:
$ ./scripts/github_image_helper.sh
0 pull and extract everything
1 ExtractAndLoadImagesFromGithubWorkflowBuildISO
2 ExtractISOsFromGithubWorkflowBuilds
3 GithubTriggerPackagesBuild
4 PullAndTagGithubWorkflowImages
5 PullAndTagGithubWorkflowISOImages
These menu options are described below:
- pull and extract everything
- This option is the same as if the user had selected steps 5, 1, 2, and 4 in that order. It will result in the pulling of the Malcolm and Hedgehog Linux installer ISO wrapper container images from ghcr.io and the loading of the rest of the Malcolm container images from the copy stored internally in the ISO’s file system (to conserve bandwidth).
- ExtractAndLoadImagesFromGithubWorkflowBuildISO
- This option extracts and loads the Malcolm container images (built from the user’s Malcolm fork) that were embedded into the file system of the Malcolm installer ISO wrapper container image.
- ExtractISOsFromGithubWorkflowBuilds
- This option extracts the ISO 9660 image files for the Malcolm and Hedgehog Linux installer ISOs from their respective wrapper container images, and writes these files (with the
.isoextension) to the local file system.
- This option extracts the ISO 9660 image files for the Malcolm and Hedgehog Linux installer ISOs from their respective wrapper container images, and writes these files (with the
- GithubTriggerPackagesBuild
- This option will trigger a repository dispatch via the GitHub API using
curl. In order for this operation to work, an environment variable namedGITHUB_OAUTH_TOKENmust be defined containing a personal access token created for your GitHub user account with the “Actions (write)” repository permission.
- This option will trigger a repository dispatch via the GitHub API using
- PullAndTagGithubWorkflowImages
- This option will pull latest Malcolm container images (excluding the installer ISO wrapper container images) from ghcr.io for the user’s Malcolm fork, and re-tag them with
ghcr.io/idaholaband the current Malcolm version (e.g.,24.06.0) so that they may be run without modifying the localdocker-compose.ymlfile. This is probably the option users will select most often. Note that this is different from the action performed in steps 1 and 0 above: this pulls the images directly from the container registry, it does not extract them from the Malcolm installer ISO wrapper container image.
- This option will pull latest Malcolm container images (excluding the installer ISO wrapper container images) from ghcr.io for the user’s Malcolm fork, and re-tag them with
- PullAndTagGithubWorkflowISOImages
- This option will pull latest Malcolm installer ISO wrapper container images from ghcr.io for the user’s Malcolm fork, and re-tag them with
ghcr.io/idaholaband the current Malcolm version (e.g.,24.06.0).
- This option will pull latest Malcolm installer ISO wrapper container images from ghcr.io for the user’s Malcolm fork, and re-tag them with
The script can also be run non-interactively by specifying the option number on the command line (e.g., ./scripts/github_image_helper.sh 4).
In addition to requiring Docker, some of the options above also require the xorriso and squashfs-tools packages to extract from the ISO 9660 image file embedded in the Malcolm installer ISO wrapper container image. Consult your distribution’s documentation for how to install these tools (something like sudo apt-get -y install xorriso squashfs-tools should work on distributions that use the apt package manager).
Example
For example, here is a workflow for building the Malcolm container images using GitHub runners and pulling them from the ghcr.io container registry.
First, a build is initiated for all containers:
$ ./scripts/github_image_helper.sh
0 pull and extract everything
1 ExtractAndLoadImagesFromGithubWorkflowBuildISO
2 ExtractISOsFromGithubWorkflowBuilds
3 GithubTriggerPackagesBuild
4 PullAndTagGithubWorkflowImages
5 PullAndTagGithubWorkflowISOImages
Operation:3
GithubTriggerPackagesBuild
Issuing repository_dispatch on romeogdetlevjr/Malcolm
The user would then wait for about 45 minutes for the workflows to complete. These may be monitored at https://github.com/username/malcolm/actions, with an is:in_progress filter to only show the workflows that are still executing. Remember that the arm64 builds will take much longer than the x86-64 builds; users working in an x86-64 environment may check on the status of a particular running workflow and know that it’s safe to continue if the linux/amd64 job is complete even if the linux/arm64 job is still running.
Once all the workflows are finished, the containers may be pulled:
$ ./scripts/github_image_helper.sh
0 pull and extract everything
1 ExtractAndLoadImagesFromGithubWorkflowBuildISO
2 ExtractISOsFromGithubWorkflowBuilds
3 GithubTriggerPackagesBuild
4 PullAndTagGithubWorkflowImages
5 PullAndTagGithubWorkflowISOImages
Operation:4
PullAndTagGithubWorkflowImages
Pulling images from ghcr.io/romeogdetlevjr (main) and tagging as 24.06.0...
main: Pulling from romeogdetlevjr/malcolm/api
Digest: sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Status: Image is up to date for ghcr.io/romeogdetlevjr/malcolm/api:main
ghcr.io/romeogdetlevjr/malcolm/api:main
xxxxxxxxxxxx: Pull complete
...
main: Pulling from romeogdetlevjr/malcolm/zeek
Digest: sha256:xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Status: Image is up to date for ghcr.io/romeogdetlevjr/malcolm/zeek:main
ghcr.io/romeogdetlevjr/malcolm/zeek:main
xxxxxxxxxxxx: Pull complete
Verify that the images were pulled. Note that users will see two tags for each image: one tagged with the username and branch (e.g., ghcr.io/romeogdetlevjr/malcolm/api:main) and another tagged with ghcr.io/idaholab and the Malcolm version (e.g., ghcr.io/idaholab/malcolm/api:24.06.0).
$ docker images | grep romeogdetlevjr/malcolm
ghcr.io/idaholab/malcolm/zeek 24.06.0 xxxxxxxxxxxx 10 minutes ago 1.39GB
ghcr.io/romeogdetlevjr/malcolm/zeek main xxxxxxxxxxxx 10 minutes ago 1.39GB
ghcr.io/idaholab/malcolm/dashboards 24.06.0 xxxxxxxxxxxx 13 minutes ago 1.55GB
ghcr.io/romeogdetlevjr/malcolm/dashboards main xxxxxxxxxxxx 13 minutes ago 1.55GB
ghcr.io/idaholab/malcolm/suricata 24.06.0 xxxxxxxxxxxx 14 minutes ago 339MB
ghcr.io/romeogdetlevjr/malcolm/suricata main xxxxxxxxxxxx 14 minutes ago 339MB
ghcr.io/idaholab/malcolm/file-monitor 24.06.0 xxxxxxxxxxxx 15 minutes ago 712MB
ghcr.io/romeogdetlevjr/malcolm/file-monitor main xxxxxxxxxxxx 15 minutes ago 712MB
ghcr.io/idaholab/malcolm/redis 24.06.0 xxxxxxxxxxxx 15 minutes ago 55.4MB
ghcr.io/romeogdetlevjr/malcolm/redis main xxxxxxxxxxxx 15 minutes ago 55.4MB
ghcr.io/idaholab/malcolm/nginx-proxy 24.06.0 xxxxxxxxxxxx 16 minutes ago 160MB
ghcr.io/romeogdetlevjr/malcolm/nginx-proxy main xxxxxxxxxxxx 16 minutes ago 160MB
ghcr.io/idaholab/malcolm/pcap-capture 24.06.0 xxxxxxxxxxxx 16 minutes ago 137MB
ghcr.io/romeogdetlevjr/malcolm/pcap-capture main xxxxxxxxxxxx 16 minutes ago 137MB
ghcr.io/idaholab/malcolm/htadmin 24.06.0 xxxxxxxxxxxx 16 minutes ago 246MB
ghcr.io/romeogdetlevjr/malcolm/htadmin main xxxxxxxxxxxx 16 minutes ago 246MB
ghcr.io/romeogdetlevjr/malcolm/file-upload main xxxxxxxxxxxx 16 minutes ago 250MB
ghcr.io/idaholab/malcolm/file-upload 24.06.0 xxxxxxxxxxxx 16 minutes ago 250MB
ghcr.io/idaholab/malcolm/logstash-oss 24.06.0 xxxxxxxxxxxx 16 minutes ago 1.49GB
ghcr.io/romeogdetlevjr/malcolm/logstash-oss main xxxxxxxxxxxx 16 minutes ago 1.49GB
ghcr.io/idaholab/malcolm/netbox 24.06.0 xxxxxxxxxxxx 17 minutes ago 1.66GB
ghcr.io/romeogdetlevjr/malcolm/netbox main xxxxxxxxxxxx 17 minutes ago 1.66GB
ghcr.io/romeogdetlevjr/malcolm/filebeat-oss main xxxxxxxxxxxx 18 minutes ago 405MB
ghcr.io/idaholab/malcolm/filebeat-oss 24.06.0 xxxxxxxxxxxx 18 minutes ago 405MB
ghcr.io/romeogdetlevjr/malcolm/postgresql main xxxxxxxxxxxx 18 minutes ago 303MB
ghcr.io/idaholab/malcolm/postgresql 24.06.0 xxxxxxxxxxxx 18 minutes ago 303MB
ghcr.io/idaholab/malcolm/arkime 24.06.0 xxxxxxxxxxxx 18 minutes ago 802MB
ghcr.io/romeogdetlevjr/malcolm/arkime main xxxxxxxxxxxx 18 minutes ago 802MB
ghcr.io/idaholab/malcolm/opensearch 24.06.0 xxxxxxxxxxxx 18 minutes ago 1.42GB
ghcr.io/romeogdetlevjr/malcolm/opensearch main xxxxxxxxxxxx 18 minutes ago 1.42GB
ghcr.io/idaholab/malcolm/pcap-monitor 24.06.0 xxxxxxxxxxxx 18 minutes ago 176MB
ghcr.io/romeogdetlevjr/malcolm/pcap-monitor main xxxxxxxxxxxx 18 minutes ago 176MB
ghcr.io/idaholab/malcolm/dashboards-helper 24.06.0 xxxxxxxxxxxx 18 minutes ago 233MB
ghcr.io/romeogdetlevjr/malcolm/dashboards-helpermain xxxxxxxxxxxx 18 minutes ago 233MB
ghcr.io/idaholab/malcolm/freq 24.06.0 xxxxxxxxxxxx 18 minutes ago 153MB
ghcr.io/romeogdetlevjr/malcolm/freq main xxxxxxxxxxxx 18 minutes ago 153MB
ghcr.io/idaholab/malcolm/api 24.06.0 xxxxxxxxxxxx 18 minutes ago 169MB
ghcr.io/romeogdetlevjr/malcolm/api main xxxxxxxxxxxx 18 minutes ago 169MB