testing.md

Testing

Overview

Type	Tool	Command	Requires Cluster
Go unit tests	go test	make test	No
CRD validation (CEL)	envtest	go test ./api/...	No
Webhook unit tests	envtest	go test ./internal/webhook/...	No
Helm chart lint	helm lint	make helm-lint	No
Helm chart unit tests	helm unittest	make helm-unittest	No
Go linting	golangci-lint	make lint	No
Go vulnerability check	govulncheck	make vulncheck	No
CRD/deepcopy drift check	controller-gen	make check-manifests	No
Container image lint	hadolint	via _container-build.yaml	No
Shell script lint	shellcheck	via _shellcheck.yaml	No
Unicode lint	grep -P	via _unicode-lint.yaml	No
E2E tests	Chainsaw	make e2e-group-*	Yes

Run all offline checks at once:

make ci

CI/CD Workflows

Workflow Overview

Workflow	File	Trigger	What it does
CI	ci.yaml	Push to main/develop, PRs	Linting + all 6 image builds; pushes :develop tag on develop branch
E2E Images	e2e-images.yaml	Manual only	Builds + pushes all 6 E2E images to ghcr.io
E2E	e2e.yaml	Manual only	Runs all E2E test groups sequentially against pre-built images
E2E (single)	e2e-single.yaml	Manual only	Run a single test, group, or all tests for an operator variant
Release	release.yaml	Manual (main only)	semantic-release, builds operator/server/db with version tag + :latest, publishes Helm charts

Reusable Workflows

Workflow	File	Purpose
Container Build	_container-build.yaml	Multi-arch image build, optional push, signing, SBOM
E2E Run	_e2e-run.yaml	Reusable: operator setup, matrix test execution, cleanup
Go	_go.yaml	Go build, test, vet, vulncheck, lint
Helm	_helm.yaml	Helm lint + unittest
Shellcheck	_shellcheck.yaml	Shell script linting
Unicode Lint	_unicode-lint.yaml	Detect suspicious Unicode characters

Image Tagging Strategy

Context	Images	Push	Tag	:latest
CI - PR	all 6	No	-	-
CI - push develop	all 6	Yes	:develop	No
CI - push main	all 6	No	-	-
E2E Images - manual	all 6	Yes	custom image_tag or branch name	No
Release	operator, server, db	Yes	:x.y.z	Yes

Key points:

• :develop images are unstable, rebuilt on every push to develop (by the CI workflow)
• :latest is only set by release.yaml and always points to the latest release
• E2E tests run against pre-built images; no images are built or cleaned up during test runs

Which Images Are Released?

Only the three production images and three Helm charts are released:

Artifact	Type	Released
openvox-operator	Container image	Yes
openvox-server	Container image	Yes
openvox-db	Container image	Yes
openvox-agent	Container image	No (E2E only)
openvox-e2e-code	Container image	No (E2E only)
openvox-mock	Container image	No (E2E only)
openvox-operator	Helm chart	Yes
openvox-stack	Helm chart	Yes
openvox-db-postgres	Helm chart	Yes

Go Unit Tests

make test

Runs go test ./... with coverage. Tests include:

• Controller reconciliation (internal/controller/)
• Certificate signing and CA HTTP client
• Webhook validation (internal/webhook/)
• Duration parsing (api/v1alpha1/duration.go)
• Volume helpers, hash functions, image resolution
• Label generation
• Signing policy evaluation (cmd/autosign/policy.go)
• ENC HTTP client (cmd/enc/classifier.go)
• PuppetDB Wire Format v8 transformation (cmd/report/puppetdb.go)

CRD Validation Tests

The api/v1alpha1/ package contains envtest-based tests that validate CEL rules and enum constraints against a real API server. These run as part of go test ./... but require envtest binaries.

For local development:

go install sigs.k8s.io/controller-runtime/tools/setup-envtest@latest
source <(setup-envtest use -p env)
go test ./api/...

In CI, envtest is set up automatically.

Helm Tests

Linting

make helm-lint

Lints all three charts: openvox-operator, openvox-stack, openvox-db-postgres.

Unit Tests

make helm-unittest

Runs helm-unittest for all charts. Test files are in charts/*/tests/.

Container Image Lint

All Containerfiles are linted with hadolint as part of _container-build.yaml. This runs automatically in CI for every image build.

E2E Tests

E2E tests use Chainsaw to deploy the operator and OpenVox stack scenarios against a real Kubernetes cluster and verify the full resource lifecycle.

Container Images

E2E tests require 6 container images pulled from ghcr.io at runtime:

Image	Purpose
openvox-operator	The operator itself
openvox-server	OpenVox Server (CA + compiler)
openvox-db	OpenVox DB (PuppetDB)
openvox-e2e-code	OCI image with Puppet environments
openvox-agent	Puppet agent (UBI9)
openvox-mock	Mock server for ENC, reports, OpenVox DB endpoints

OCI image volumes require Kubernetes 1.35+ (ImageVolume feature gate).

Test Groups

Tests are organized into groups. Each group installs the operator with specific settings, runs its tests, and cleans up.

Group	Operator Settings	Tests	Make Target
base	webhook=false, gatewayAPI=false	single-node, multi-server, agent-basic, agent-broken, agent-idempotent, agent-concurrent, agent-report, database-cnpg, readonly-rootfs, code-pvc, autosign-policy, cert-rotation	make e2e-group-base
enc	webhook=false, gatewayAPI=false	agent-enc, agent-full	make e2e-group-enc
gateway	webhook=false, gatewayAPI=true	pool-gateway	make e2e-group-gateway
webhooks-cm	webhook=true, cert-manager	webhook-validation-server, webhook-validation-config, webhook-validation-database, webhook-smoke	make e2e-group-webhooks-cm

Run all groups sequentially:

make e2e-all

External Dependencies

Some tests require external operators:

• CloudNativePG -- PostgreSQL operator for database-cnpg test
• Envoy Gateway -- Gateway API implementation for pool-gateway test
• cert-manager -- Webhook TLS automation for webhooks-cm test

In CI, these are pre-installed on the persistent E2E cluster via ArgoCD. The workflow only waits for readiness (make e2e-wait).

For local development, install them with setup.sh:

bash tests/e2e/setup.sh all              # Install all dependencies
bash tests/e2e/setup.sh install-cnpg      # CNPG operator only
bash tests/e2e/setup.sh install-envoy-gateway  # Envoy Gateway only
bash tests/e2e/setup.sh install-cert-manager   # cert-manager only
bash tests/e2e/setup.sh status            # Check status

Or via Make:

make e2e-setup    # Install all dependencies (local dev)
make e2e-wait     # Only check readiness (used in CI)

Versions can be overridden via environment variables: CNPG_VERSION, ENVOY_GATEWAY_VERSION, CERT_MANAGER_VERSION.

Running Locally

# Run a single test (operator must be installed first)
make e2e-operator-base IMAGE_TAG=develop
make e2e-run-test TEST=cert-rotation IMAGE_TAG=develop

# Run a test group (installs operator automatically)
make e2e-group-base IMAGE_TAG=develop

# Run all groups
make e2e-all IMAGE_TAG=develop

Feature branches: E2E images are only pushed for the develop branch. To run E2E tests for a feature branch, you need to build and push images manually (e.g. via gh workflow run e2e-images.yaml --ref <branch>). A future in-cluster registry feature will make this easier -- see the tracking issue for details.

Running in CI

The E2E workflow connects to a persistent K3S cluster via E2E_KUBECONFIG secret. External dependencies (CNPG, Envoy Gateway, cert-manager) are managed by ArgoCD on the cluster -- the workflow only verifies they are available before starting tests. Each test runs as its own matrix job with max-parallel: 1 for better visibility.

Images are built by the CI workflow on push to develop (:develop tag) or manually via e2e-images.yaml. The E2E workflows only run tests against already-pushed images.

There are three ways to trigger E2E tests:

Goal	Command
Run all groups	gh workflow run e2e.yaml -f image_tag=develop
Run a test group	gh workflow run e2e-single.yaml -f operator=base -f group=base
Run a single test	gh workflow run e2e-single.yaml -f operator=base -f tests='["cert-rotation"]'

The e2e-single.yaml workflow selects tests using this priority:

group	tests	What runs
(empty)	(empty)	All tests for the selected operator variant
base	(empty)	All tests in the base group
(empty)	["cert-rotation"]	Only cert-rotation
base	["cert-rotation"]	Only cert-rotation (tests overrides group)

Chainsaw Configuration

Global settings in tests/e2e/chainsaw-config.yaml:

• skipDelete: true -- namespaces are preserved on failure for debugging
• parallel: 1 -- tests run sequentially (single shared cluster)
• Timeouts: apply 1m, assert 5m, cleanup 2m

Test Scenarios

Tests are in tests/e2e/<scenario>/chainsaw-test.yaml.

Stack Deployment Tests

• single-node -- Minimal CA+Server stack, verifies all resources reach terminal phases
• multi-server -- Dedicated CA + 2 compiler replicas, verifies pod count and phases

Agent Tests

• agent-basic -- Single Puppet agent run with autosign and code deployment
• agent-concurrent -- Three agents running in parallel
• agent-idempotent -- Two consecutive runs verifying idempotency
• agent-broken -- Agent with broken environment, verifies expected failure
• agent-enc -- Agent with ENC via mock server, verifies classification request
• agent-report -- Agent with report forwarding to mock server
• agent-full -- Full integration: ENC + reports + OpenVox DB via mock

Database Tests

• database-cnpg -- CNPG PostgreSQL cluster + OpenVox DB deployment, verifies connectivity

Gateway Tests

• pool-gateway -- Gateway API TLSRoute with Envoy Gateway, verifies SNI routing

Webhook Validation Tests

• webhook-validation-server -- Server ref validation (configRef, certificateRef, poolRef)
• webhook-validation-config -- Config ref validation (authorityRef, databaseRef/serverUrls mutual exclusion)
• webhook-validation-database -- Database field validation (certificateRef, host, credentialsSecretRef, sslMode)
• webhook-smoke -- ValidatingWebhookConfiguration exists, Service has endpoints, smoke create/reject

Writing New Tests

Create a new directory under tests/e2e/ with a chainsaw-test.yaml:

tests/e2e/
+-- chainsaw-config.yaml
+-- my-scenario/
    +-- chainsaw-test.yaml

Key patterns:

• Use spec.namespace to set an isolated namespace
• Use script operations for Helm install/uninstall
• Use assert operations with resource to wait for CRD status fields
• Place cleanup in the finally block of the last step
• Reference chart paths via $(git rev-parse --show-toplevel) for portability

Add the test to the appropriate group in the Makefile (e2e-group-* target).

See the Chainsaw documentation for the full API reference.