Enforce profile correctness with spread

Spread is an integration test runner that lets you attach a dedicated test to every profile and replay it on every change. Each test is a short .yaml file that lists the commands to execute and the results to expect. The upstream AppArmor project already uses spread to exercise its own profiles: when a software update introduces a new behavior the profile does not cover, the next run catches it immediately, and the profiles can be fixed accordingly. Spread drives real cloud images (Ubuntu, openSUSE, Fedora, ...), so the same test confirms the profile behaves identically across distributions and environments.

This tutorial complements the autopkgtest tutorial and the custom pipeline tutorial, that allows respectively to check apparmor coverage for autopkgtest tools and upstream tests. By design, spread makes sure that profiles keep adapted to the software they are built for. It typically happens later in the profile deployment cycle.

In this tutorial you will learn how to:

Write a per-profile task.yaml exercising the application
Allow-list intentional denials with EXPECT_DENIALS

Prerequisites

Install the Go toolchain, QEMU, and a build environment for image-garden:

sudo apt-get install -y golang-go qemu-system-x86 build-essential git

If you don't already have it, clone the AppArmor repository into ~/apparmor (the path the rest of this tutorial assumes):

git clone https://gitlab.com/apparmor/apparmor.git ~/apparmor

Install spread itself with go install. It lands in ~/go/bin/spread, which is the path the rest of this tutorial assumes:

go install github.com/snapcore/spread/cmd/spread@latest

Install image-garden from source:

git clone https://gitlab.com/zygoon/image-garden
cd image-garden
sudo make install prefix=/usr/local

Writing a task

In order to add a test for your profile you can simply write a test file for your profile file in tests/profiles/<your_profile_name>/task.yaml.

The simplest possible task exercises the application and asserts that apparmor find no denials for this profile.

summary: smoke test for the curl profile
execute: |
    # set up fake HTTP server
    echo -ne "HTTP/1.0 200 OK\nContent-type: text/html; charset=utf-8\nContent-Length: 12\n\nhello, world" > res
    nc -lvp 8080 < res &

    # HTTP GET to server, save result
    curl http://localhost:8080/ -o /tmp/res

    # assert result is correct
    test "$(cat /tmp/res)" = "hello, world"

    # The profile is attached based on the program path.
    "$SPREAD_PATH"/tests/bin/actual-profile-of curl | MATCH 'curl \(enforce\)'

This example creates a fake http server, exercises curl and checks that it works. It also checks that the curl profile is enforced. This test is successful if AppArmor generates no denial for this test.

Task sections

A task.yaml supports the following lifecycle:

summary: one-line description
environment:
    PROFILE_NAME: ...      # override the default (task directory name)
    PROGRAM_NAME: ...      # used by the suite's debug-each
    EXPECT_DENIALS: |      # optional; multi-line regex list, one per line
        operation="open".*name="/var/log/foo"
prepare: |
    # task-specific setup (create files, start services)
execute: |
    # the actual test
restore: |
    # cleanup (stop services, remove files)

Task-level prepare and restore run after the suite's prepare-each and before the suite's restore-each respectively, so the profile is already loaded when prepare starts and is still loaded when restore ends.

Expected denials

Some tasks deliberately exercise a path the profile is supposed to block, to confirm the profile actually blocks it. As a simple example, curl should not modify sensitive files e.g. /etc/passwd. A task that points curl's output there should trigger a denial, and the absence of that denial would be a bug in itself.

To allow-list the denial without weakening the default "any denial fails the task" assertion, set EXPECT_DENIALS to a regex (or a newline-separated list of regexes) that matches every denial line you expect. Do not forget to escape double quotes " as \".

summary: curl must not be allowed to overwrite system files
environment:
    PROFILE_NAME: curl
    EXPECT_DENIALS: 'operation=\"open\".*profile=\"curl\".*name=\"/etc/passwd\".*requested_mask=\"[wc]+\"'
execute: |
    echo -ne "HTTP/1.0 200 OK\nContent-Length: 3\n\nevil" > res
    nc -lvp 8080 < res &
    sleep 1

    # Attempt to overwrite /etc/passwd inside the VM. The curl profile denies
    # writes under /etc, so this must fail and leave /etc/passwd untouched.
    curl http://localhost:8080/ -o /etc/passwd || true

    grep -q '^root:' /etc/passwd    # sanity: the real file is intact

The assertion is two-way: every observed denial must match some regex, and every regex must match at least one observed denial. If a future profile change started allowing the write, no denial would fire and the test would fail because the expected regex went unmatched, which is what you want.

Running multiple tests

A single task.yaml can group several related tests for the same profile. Spread creates one test variant per entry named TEST/<variant> under environment:, and runs each variant as an independent test through the same prepare / execute / restore lifecycle. Variant-specific settings (like EXPECT_DENIALS/<variant>) apply only to the matching variant.

summary: Spread tests for the curl profile
environment:
    PROFILE_NAME: curl

    # curl fetches a page from the local HTTP server.
    TEST/fetch: "curl http://localhost:8080/ -o /tmp/res && test \"$(cat /tmp/res)\" = hello"

    # curl must not be allowed to overwrite sensitive system files.
    TEST/write_passwd: "curl http://localhost:8080/ -o /etc/passwd"
    EXPECT_DENIALS/write_passwd: 'operation=\"open\".*profile=\"curl\".*name=\"/etc/passwd\".*requested_mask=\"[wc]+\"'

prepare: |
    echo -ne "HTTP/1.0 200 OK\nContent-Length: 5\n\nhello" > res
    nc -lvp 8080 < res </dev/null >/dev/null 2>&1 &
    echo $! > .nc.pid
    sleep 1

restore: |
    kill "$(cat .nc.pid)" 2>/dev/null || true
    rm -f .nc.pid res

execute: |
    if [ -z "${EXPECT_DENIALS:-}" ]; then
        eval "$TEST"
    else
        eval "$TEST" || true
    fi

The shared execute block runs the variant's $TEST command and picks the expectation based on whether EXPECT_DENIALS is set: variants without it must succeed (no denials allowed), variants with it may fail because the profile is expected to block the operation.

You can run a single variant by appending its name to the task path with a colon:

sudo ~/go/bin/spread -vv garden:ubuntu-cloud-24.04:tests/profiles/curl:write_passwd

Omit the :<variant> suffix to run every variant in the task.

Run spread

From the AppArmor repository root, run the suite against a single system and a single task:

cd ~/apparmor
sudo ~/go/bin/spread -vv garden:ubuntu-cloud-24.04:tests/profiles/curl

The first run downloads the Ubuntu 24.04 cloud image (several minutes, cached afterwards), boots it, builds the AppArmor userspace from the repository (needed by the suite's prepare), and runs the task. Subsequent runs reuse the cached image and the built artefacts; a single-task iteration drops to tens of seconds.

Interpreting failures

A task failure prints the denial lines that triggered it:

Error executing garden:ubuntu-cloud-24.04:tests/profiles/curl:
-----
Unexpected denials:
[ 84.112] audit: apparmor="DENIED" operation="open"
  profile="curl" name="/etc/ssl/private/custom.crt"
  requested_mask="r" denied_mask="r"
-----

From there, the flow is the same as when refining a profile in complain mode: decide whether the access is legitimate (add a rule), an attack surface you want to keep blocked (add an EXPECT_DENIALS regex), or a test artefact (fix the test).

Contributing a test upstream

If you created a spread test for a profile you might want to contribute upstream so that the profile then gets tested on every upstream change, by everyone running the suite.

You can open a merge request against gitlab.com/apparmor/apparmor with your spread test.

Keep tasks short, deterministic, and focused on one profile. The upstream tasks under tests/profiles/ are a good reference for idiom and scope.