Skip to content

fix: add strict deb version match in VHD containerd content test#8960

Open
abigailliang-aks-sig-node wants to merge 11 commits into
mainfrom
fix/strict-containerd-version-content-test
Open

fix: add strict deb version match in VHD containerd content test#8960
abigailliang-aks-sig-node wants to merge 11 commits into
mainfrom
fix/strict-containerd-version-content-test

Conversation

@abigailliang-aks-sig-node

@abigailliang-aks-sig-node abigailliang-aks-sig-node commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

Summary

Add strict full-version matching in the VHD content test for containerd and cri-tools, catching version drift at VHD build time rather than in e2e.

Problem

The VHD content test (testContainerd/testCriCtl) previously only compared major.minor.patch (stripping the hotfix suffix via cut -d- -f1). This meant:

  • components.json says 2.3.2-ubuntu24.04u1
  • VHD build installs 2.3.2-ubuntu24.04u2 (due to apt wildcard moby-containerd=2.3.2*)
  • Content test passes (only checks 2.3.2 == 2.3.2)
  • E2e validator fails (exact match of full version string)

Changes

  • Add assertPackageVersion helper (inline in linux-vhd-content-test.sh): uses dpkg-query (deb) or rpm (RPM-based) to compare the full installed version against components.json
  • Use explicit hardcoded package name mapping per OS (not renovateTag parsing, since renovateTag is metadata for Renovate bot and its format is not guaranteed stable)
    • kubernetes-cri-toolskubernetes-cri-tools (same across all OS)
    • containerdmoby-containerd (Ubuntu/Mariner) or containerd2 (Azure Linux 3.0+)
  • Fail fast when package name cannot be resolved
  • Fix err() call signatures to match the two-argument format ($1=test name, $2=message)
  • Rename debPackageNameinstalledPackageName since the check now covers both deb and rpm
  • Script is fully self-contained — no external helper files needed (runs on VM without dependencies)

Follow-up items

  1. ARM64 support: testPackagesInstalled currently skips entirely on ARM64 (line 209), meaning ARM64 VHDs have no package version validation. Opening this up to ARM64 would require verifying that all packages in components.json have correct ARM64 entries and that the download/install paths work for arm64.

Test plan

  • VHD build with mismatched version fails at content test (not at e2e) — verified by intentionally setting containerd version to u99:
    testContainerd:Error: installed moby-containerd version '2.3.2-ubuntu24.04u2' does not match expected '2.3.2-ubuntu24.04u99' from components.json
    
    Failed in "Test, Scan, and Cleanup" stage as expected.
  • shellspec CI passes locally with no regressions (804 examples, 0 failures)
  • VHD build with matching components.json version passes content test (will be verified once CI runs with correct version)

This comment was marked as duplicate.

Copilot AI review requested due to automatic review settings July 15, 2026 21:29
@abigailliang-aks-sig-node
abigailliang-aks-sig-node force-pushed the fix/strict-containerd-version-content-test branch from 6af5478 to f541322 Compare July 15, 2026 21:29

This comment was marked as duplicate.

Copilot AI review requested due to automatic review settings July 16, 2026 00:57

This comment was marked as off-topic.

@@ -0,0 +1,96 @@
#!/bin/bash

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This script file is not called or referenced by anyone, right?

@abigailliang-aks-sig-node abigailliang-aks-sig-node Jul 17, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this a local test file starting as test_....

I originally planned to add a ShellSpec test, but ces_hepler.sh does not work well in the ShellSpec Docker environment. As a temporary solution, I’m keeping this as a local test script for manual validation.

And one of the follow-up is to figure out a way to unit test vhdbuilder/packer/test/linux-vhd-content-test.sh in ci

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If it's not added any CI, or local test bundle (like make generate). I am afraid that a few weeks later you will be the only one who knows this presence of this test script😅 and it will become dangling. Why did you say cse_helper.sh doesn't work well? There is an existing cse_helper_spec.sh.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, i think it would be best to add as a shellspec. If not possible, i honestly don't think its necessary since its a test for a test, not super vital if we don't have it since the vhd content test will catch it

@abigailliang-aks-sig-node abigailliang-aks-sig-node Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason I initially thought integrating cse_helpers.sh with ShellSpec would be tricky was that I was trying to use source to load the script, which failed in the Docker environment due to missing system dependencies like lsb_release. So I left it as a local-only test script and planned to figure out the CI integration in a follow-up PR.

but your concerns here make sense. i could figure out a way to run a shell spec as the sample you mentioned here. thx.

also one thing i dont fully understand here. make generate doesn't corelated to this change —you mean we have an cse_helpers_spec.sh file can run the cse_helpers.sh

@abigailliang-aks-sig-node abigailliang-aks-sig-node Jul 20, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update: I removed the ShellSpec test since it had become redundant. It was only comparing hardcoded strings, and now that the package mapping lives directly in the content test cases, there isn't much left for it to test.

Thanks for the big-picture perspective—that was really helpful! 🙏 @Devinwong and @lilypan26

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the changes!
But as Lily suggested, it's rare to see unit testing a test-only function.
IMO, we generally just test production codes. Although it's no harm to test a test-only function, however if one day your spec/vhdbuilder/packer/test/assert_package_version_spec.sh fails, you will need to check 2 places (your test codes and the production codes). You can have local test to make sure your test-only function is valid. But once it's confirmed, we will trust it's always reliable.
So I will just remove the spec/vhdbuilder/packer/test/assert_package_version_spec.sh if that makes sense to you.

Comment thread vhdbuilder/packer/test/linux-vhd-content-test.sh Outdated
Copilot AI review requested due to automatic review settings July 20, 2026 19:56

This comment was marked as duplicate.

Per review feedback, replace renovateTag parsing with hardcoded
package name mapping. renovateTag is metadata for Renovate bot and
its format is not guaranteed stable — parsing it makes the test
fragile to unrelated changes.

Also remove the shellspec test since the mapping is now trivially
explicit and doesn't warrant a separate unit test.
Copilot AI review requested due to automatic review settings July 20, 2026 22:19
@abigailliang-aks-sig-node
abigailliang-aks-sig-node force-pushed the fix/strict-containerd-version-content-test branch from 3c89aac to 7b302f9 Compare July 20, 2026 22:19

This comment was marked as duplicate.

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Copilot AI review requested due to automatic review settings July 20, 2026 22:23

This comment was marked as duplicate.

Copilot AI review requested due to automatic review settings July 20, 2026 22:49
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

This comment was marked as duplicate.

Copilot AI review requested due to automatic review settings July 20, 2026 22:52

This comment was marked as duplicate.

This comment was marked as duplicate.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (2)

vhdbuilder/packer/test/linux-vhd-content-test.sh:321

  • testPackagesInstalled resolves containerd’s installed package name based only on $AZURELINUX_OS_NAME, but getCurrentPackageTestOS can set OS to $AZURELINUX_KATA_OS_NAME (AzureLinux kata). In that case this falls through to moby-containerd, causing the strict package-version check to query the wrong package name and fail on kata images.
            case "$OS" in
              "$AZURELINUX_OS_NAME")
                pkgName="containerd2"
                ;;
              *)

vhdbuilder/packer/test/linux-vhd-content-test.sh:36

  • The PR description says err() call signatures were updated to the 2-argument format, but this script still has multiple 1-argument call sites (e.g. around lines ~1754, ~1764, ~1955, ~2245, ~2333). With the current err() implementation, those emit malformed output like <message>:Error: with an empty message. Either update remaining call sites or make err() tolerant of being called with a single argument.
err() {
  echo "$1:Error: $2" >>/dev/stderr
}

# assertPackageVersion verifies that the installed deb/rpm package version matches

local crictl_version
crictl_version=$(crictl --version)
# the output of crictl_version looks like this "crictl version 1.32.0", need to extract the version number.
crictl_version=$(echo $crictl_version | cut -d' ' -f3)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

components This pull request updates cached components on Linux or Windows VHDs

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants