Skip to content

fix: allow DNS failover during CSE connectivity checks#8985

Merged
r2k1 merged 2 commits into
mainfrom
r2k1-fix-cse-dns-failover
Jul 21, 2026
Merged

fix: allow DNS failover during CSE connectivity checks#8985
r2k1 merged 2 commits into
mainfrom
r2k1-fix-cse-dns-failover

Conversation

@r2k1

@r2k1 r2k1 commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

The Linux CSE wraps hostname-based connectivity checks with per-attempt timeouts. With multiple custom DNS servers, an unresponsive primary can consume the timeout before libc tries a responsive secondary.

This gives both affected checks 15 seconds per attempt while preserving or reducing their total retry budgets:

  • Outbound registry check: 60 attempts x 5 seconds becomes 20 attempts x 15 seconds. Including one-second sleeps, the worst-case window decreases from about 359 seconds to 319 seconds.
  • API server HTTPS check: 50/100 attempts x 10 seconds becomes 34/68 attempts x 15 seconds. The public/private worst-case windows decrease from about 549/1099 seconds to 543/1087 seconds.

Each attempt can now cover the normal primary DNS timeout, secondary DNS fallback, and TCP/TLS setup. The rendered endpoints and failure behavior are unchanged.

Tests:

  • ShellSpec: spec/parts/linux/cloud-init/artifacts/cse_main_spec.sh
  • GENERATE_TEST_DATA=true go test ./pkg/agent...
  • repository shellcheck rules for cse_main.sh

Which issue(s) this PR fixes:

N/A

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8618f8d1-4a31-43a9-92d7-0ba79b33675f

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adjusts the Linux CSE outbound connectivity preflight retry parameters in cse_main.sh to better tolerate DNS failover scenarios (e.g., an unresponsive primary custom DNS server) while slightly reducing the overall worst-case retry window. It fits into the Linux node provisioning path (nodePrep) where outbound connectivity checks are performed during cluster integration.

Changes:

  • Update the outbound preflight retry shape from 60 x 5s to 20 x 15s in parts/linux/cloud-init/artifacts/cse_main.sh.
  • Add a ShellSpec assertion to ensure the outbound preflight uses the intended retry parameters.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File Description
parts/linux/cloud-init/artifacts/cse_main.sh Adjusts outbound connectivity preflight retry/timeout parameters to allow DNS fallback within an attempt.
spec/parts/linux/cloud-init/artifacts/cse_main_spec.sh Adds a test that validates the retry parameters embedded in cse_main.sh.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

Copilot-Session: 8618f8d1-4a31-43a9-92d7-0ba79b33675f
Copilot AI review requested due to automatic review settings July 20, 2026 22:30
@r2k1 r2k1 changed the title fix: allow DNS failover during outbound preflight fix: allow DNS failover during CSE connectivity checks Jul 20, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

fi
else
logs_to_events "AKS.CSE.apiserverCurl" "retrycmd_if_failure ${API_SERVER_CONN_RETRIES} 1 10 curl -v --cacert /etc/kubernetes/certs/ca.crt https://${API_SERVER_NAME}:443" || time curl -v --cacert /etc/kubernetes/certs/ca.crt "https://${API_SERVER_NAME}:443" || VALIDATION_ERR=$ERR_K8S_API_SERVER_CONN_FAIL
logs_to_events "AKS.CSE.apiserverCurl" "retrycmd_if_failure ${API_SERVER_CONN_RETRIES} 1 15 curl -v --cacert /etc/kubernetes/certs/ca.crt https://${API_SERVER_NAME}:443" || time curl -v --cacert /etc/kubernetes/certs/ca.crt "https://${API_SERVER_NAME}:443" || VALIDATION_ERR=$ERR_K8S_API_SERVER_CONN_FAIL

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Existing behaviour. I have no willpower to change it

@r2k1
r2k1 enabled auto-merge (squash) July 20, 2026 23:36
@r2k1
r2k1 merged commit 2395857 into main Jul 21, 2026
36 checks passed
@r2k1
r2k1 deleted the r2k1-fix-cse-dns-failover branch July 21, 2026 00:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants