From 0bc7a71ec378849906dc3946c7572a4a11b4e5d4 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Mon, 15 Jun 2026 14:41:23 -0500 Subject: [PATCH 1/8] add: AWS ARM node tooling upgrade guide for Orka 3.6 (OK-5476) Documents the new Ansible-based in-place upgrade path for ARM EC2 Mac nodes, replacing the AMI replacement approach. Covers SSH/SSM prereqs, required node tag (role=orka-arm), what's preserved during upgrade, and what changes in 3.5 to 3.6 for AWS deployments. Co-Authored-By: Claude Sonnet 4.6 --- docs.json | 1 + .../upgrading-orka-on-aws.mdx | 92 +++++++++++++++++++ 2 files changed, 93 insertions(+) create mode 100644 orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx diff --git a/docs.json b/docs.json index a2f24cb..50a498a 100644 --- a/docs.json +++ b/docs.json @@ -275,6 +275,7 @@ "group": "Upgrade Guides", "pages": [ "orka/orka-upgrades-and-release-notes/orka-upgrades", + "orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws", "orka/orka-upgrades-and-release-notes/kubernetes-upgrade-guide" ] }, diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx new file mode 100644 index 0000000..0659c19 --- /dev/null +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -0,0 +1,92 @@ +--- +title: "Upgrading Orka on AWS" +description: "Upgrade your Orka cluster on AWS from 3.5 to 3.6. Covers ARM node tooling updates, SSH and SSM requirements, what's preserved during the upgrade, and what changes with the new Ansible-based upgrade path." +--- + +Upgrading Orka on AWS follows the same general process as MSDC-hosted upgrades: submit a support ticket, schedule a maintenance window, and MacStadium handles the upgrade. However, AWS deployments have specific requirements and a different upgrade mechanism for ARM Mac nodes. + +## Before you upgrade + +### Check your ARM node SSH configuration + +Starting with the 3.5 to 3.6 upgrade, MacStadium uses an Ansible-based process to update tooling on your ARM Mac nodes rather than replacing the AMI. This approach is significantly faster and preserves your node configuration. + +For this to work, your ARM EC2 Mac instances must accept SSH connections on port 22 for the `ec2-user` account using key-based authentication. EC2 Mac instances launched with a key pair have this enabled by default. Your security group must allow inbound TCP port 22 from MacStadium's CodeBuild subnets — confirm this with your MacStadium support contact when scheduling. + +If your nodes cannot accept SSH, the upgrade can run over SSM instead. SSM upgrades require an S3 bucket in the same region as your ARM nodes for Ansible file transfer, and can take significantly longer (up to 4 hours). SSH is strongly recommended. + +If you're unsure how your nodes are configured, check with your AWS administrator before scheduling. + +### Verify your node tags + +MacStadium uses a dynamic Ansible inventory to identify your ARM nodes during the upgrade. Each ARM EC2 Mac instance must have the EC2 tag: + +* **Key:** `role` +* **Value:** `orka-arm` + +Instances without this tag will not be selected by the inventory and will be skipped during the upgrade. Verify this tag is applied to all ARM nodes before scheduling. + +## Requesting an upgrade + +1. Review the [3.6 release notes](/orka/orka-upgrades-and-release-notes/orka-36-release-notes) and complete any preparation steps listed there. +2. Submit a ticket through the MacStadium portal. +3. Schedule a maintenance window via the link in your ticket. + +## What changes in 3.5 to 3.6 + +### ARM node tooling updates no longer require AMI replacement + +Previously, updating tooling on ARM nodes required replacing the EC2 Mac AMI: the instance had to be deleted, a new one provisioned (a process that takes approximately 2 hours), and the node's name, namespace, and custom tags had to be manually reapplied. + +Starting with the 3.5 to 3.6 upgrade path, MacStadium updates ARM node tooling in place using Ansible over SSH. The upgrade takes under 10 minutes per node over SSH. The following are read from the running node and reapplied automatically: node name, node IP, cluster registration, license key, VM quota, and storage layout (including data volumes on instances with local NVMe). Running VMs are not interrupted. + +### Upgrade Service is installed + +As part of the 3.6 upgrade, the Orka Upgrade Service is deployed to your cluster. This enables smoother tooling updates in future Orka releases without requiring AMI replacement. + +### AWS credentials no longer required for artifact distribution + +Orka binaries and container images are now distributed publicly via CloudFront. You no longer need AWS credentials configured to pull Orka artifacts during upgrades or deployments. + +### cert-manager behavior change + +Orka no longer installs its own cert-manager if one is already present in the cluster. If your cluster runs its own cert-manager and you previously experienced version or configuration conflicts with Orka's bundled installation, those conflicts are resolved in 3.6. + +If your automation or tooling depends on Orka's cert-manager specifically, verify your setup before upgrading. + +### Credential scoping: action required + +The permissions required for Orka to run have been tightened to least-privilege. Separate, minimal credential sets are now defined for: + +* Orka configuration +* Virtual Kubelet +* ECR access +* Backup operations + +If your deployment uses broad IAM credentials for Orka, you will need to update your IAM policies to align with the new scoped credential sets before or as part of the upgrade. MacStadium support will provide the updated permission requirements when scheduling your maintenance window. + +## During the maintenance window + +| Aspect | Notes | +|--------|-------| +| Maintenance window | Up to 3 hours. Must be scheduled with MacStadium support. | +| Access to the environment | Unavailable during the window. Restored at the original endpoint after completion. | +| Orka users | Persist. | +| Service Accounts | Persist. Tokens must be regenerated after the upgrade. | +| VMs and VM configs | VM configs persist. Running VMs on ARM nodes are preserved during the in-place tooling upgrade. Redeploy from VM configs after the upgrade if needed. | +| Images | Persist. | +| Image cache | Cleared from each node. | +| Namespaces and RoleBindings | Persist. | +| ARM node names and tags | Preserved with the Ansible-based upgrade path. | + + + Service Account tokens must be regenerated after this upgrade. Any automated workflows using service account tokens will fail until tokens are regenerated with `orka3 serviceaccount token `. + + +## After the upgrade + +Once the upgrade is complete: + +1. [Download and install](/orka/orka-overview/tools-integrations) the Orka 3.6 CLI if you haven't already. +2. Regenerate Service Account tokens for any automated workflows. +3. Repopulate the image cache on your ARM nodes if needed: `orka3 imagecache add --all` From 40b987a0cd9ca46d8bc49e35ce9c25a1b1353caa Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Mon, 15 Jun 2026 15:55:31 -0500 Subject: [PATCH 2/8] update: add SSH-via-SSM path, multi-region note, and node value overrides to AWS upgrade guide Co-Authored-By: Claude Sonnet 4.6 --- .../upgrading-orka-on-aws.mdx | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx index 0659c19..91f177d 100644 --- a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -17,6 +17,34 @@ If your nodes cannot accept SSH, the upgrade can run over SSM instead. SSM upgra If you're unsure how your nodes are configured, check with your AWS administrator before scheduling. +### Enabling SSH on nodes launched without a key pair + +If your nodes were not launched with a key pair and do not accept SSH connections, you can use SSM to install MacStadium's CodeBuild public SSH key first, then proceed with the faster SSH upgrade path. This requires your instances to be SSM-managed: the `AmazonSSMManagedInstanceCore` policy must be attached to the instance profile, as recommended in the Orka on AWS setup guide. + +To set this up: + +1. Contact MacStadium support to obtain the CodeBuild public SSH key. +2. Run the following SSM command to install it on all tagged ARM nodes (replace `` with the key from support): + + ```shell + aws ssm send-command \ + --document-name "AWS-RunShellScript" \ + --targets "Key=tag:role,Values=orka-arm" \ + --parameters '{"commands":[ + "mkdir -p /Users/ec2-user/.ssh", + "echo \"\" >> /Users/ec2-user/.ssh/authorized_keys", + "chmod 700 /Users/ec2-user/.ssh", + "chmod 600 /Users/ec2-user/.ssh/authorized_keys", + "chown -R ec2-user /Users/ec2-user/.ssh" + ]}' + ``` + + The identity running `send-command` requires `ssm:SendCommand` on the target instances and on the `AWS-RunShellScript` document, plus `ssm:GetCommandInvocation`. + +3. Ensure your security group allows inbound TCP port 22 from MacStadium's CodeBuild subnets. + +Once the key is installed, MacStadium can proceed with the SSH upgrade path. + ### Verify your node tags MacStadium uses a dynamic Ansible inventory to identify your ARM nodes during the upgrade. Each ARM EC2 Mac instance must have the EC2 tag: @@ -26,12 +54,27 @@ MacStadium uses a dynamic Ansible inventory to identify your ARM nodes during th Instances without this tag will not be selected by the inventory and will be skipped during the upgrade. Verify this tag is applied to all ARM nodes before scheduling. +### Multi-region deployments + +The upgrade discovers ARM nodes in the region configured on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, MacStadium runs a separate upgrade pass per region. When scheduling, provide the full list of regions where you have ARM nodes. + ## Requesting an upgrade 1. Review the [3.6 release notes](/orka/orka-upgrades-and-release-notes/orka-36-release-notes) and complete any preparation steps listed there. 2. Submit a ticket through the MacStadium portal. 3. Schedule a maintenance window via the link in your ticket. +## Changing node values during the upgrade + +By default, the upgrade reads all configuration from the running node and reapplies it automatically. If you want to change any of the following values as part of the upgrade, request the override when scheduling: + +| Value | Notes | +|-------|-------| +| Node hostname | The node re-registers under the new name in Kubernetes. | +| Node IP | Use if the node's IP address is changing. | +| License key | Replaces the license key applied to the node. | +| VM quota | Changes the number of VMs the node is permitted to run. | + ## What changes in 3.5 to 3.6 ### ARM node tooling updates no longer require AMI replacement From 6af39df888b3eef54b15c9d50adaf7639efa77dc Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Tue, 16 Jun 2026 08:14:35 -0500 Subject: [PATCH 3/8] update: fix SSH key flow, remove unpublished node overrides per Ivan review Co-Authored-By: Claude Sonnet 4.6 --- .../upgrading-orka-on-aws.mdx | 24 +++++++++++++------ 1 file changed, 17 insertions(+), 7 deletions(-) diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx index 91f177d..3c89a65 100644 --- a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -23,8 +23,8 @@ If your nodes were not launched with a key pair and do not accept SSH connection To set this up: -1. Contact MacStadium support to obtain the CodeBuild public SSH key. -2. Run the following SSM command to install it on all tagged ARM nodes (replace `` with the key from support): +1. Generate an SSH key pair (ed25519 or RSA). +2. Run the following SSM command to install the public key on all tagged ARM nodes: ```shell aws ssm send-command \ @@ -32,7 +32,7 @@ To set this up: --targets "Key=tag:role,Values=orka-arm" \ --parameters '{"commands":[ "mkdir -p /Users/ec2-user/.ssh", - "echo \"\" >> /Users/ec2-user/.ssh/authorized_keys", + "echo \"\" >> /Users/ec2-user/.ssh/authorized_keys", "chmod 700 /Users/ec2-user/.ssh", "chmod 600 /Users/ec2-user/.ssh/authorized_keys", "chown -R ec2-user /Users/ec2-user/.ssh" @@ -41,9 +41,21 @@ To set this up: The identity running `send-command` requires `ssm:SendCommand` on the target instances and on the `AWS-RunShellScript` document, plus `ssm:GetCommandInvocation`. -3. Ensure your security group allows inbound TCP port 22 from MacStadium's CodeBuild subnets. +3. Store the private key in AWS Secrets Manager as a plaintext secret. +4. Grant MacStadium's CodeBuild service role permission to read the secret: -Once the key is installed, MacStadium can proceed with the SSH upgrade path. + ```json + { + "Effect": "Allow", + "Action": "secretsmanager:GetSecretValue", + "Resource": "arn:aws:secretsmanager:*:*:secret:*" + } + ``` + +5. Ensure your security group allows inbound TCP port 22 from MacStadium's CodeBuild subnets. +6. When scheduling, provide MacStadium support with the Secrets Manager secret ARN. + +MacStadium will use the stored private key to connect over SSH for the upgrade. ### Verify your node tags @@ -71,9 +83,7 @@ By default, the upgrade reads all configuration from the running node and reappl | Value | Notes | |-------|-------| | Node hostname | The node re-registers under the new name in Kubernetes. | -| Node IP | Use if the node's IP address is changing. | | License key | Replaces the license key applied to the node. | -| VM quota | Changes the number of VMs the node is permitted to run. | ## What changes in 3.5 to 3.6 From b92293275c9c5a55c38968f665f2ed33b137959a Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Tue, 16 Jun 2026 09:46:51 -0500 Subject: [PATCH 4/8] update: rewrite AWS upgrade guide as self-service with full CodeBuild/IAM examples Co-Authored-By: Claude Sonnet 4.6 --- .../upgrading-orka-on-aws.mdx | 178 +++++++++++++----- 1 file changed, 128 insertions(+), 50 deletions(-) diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx index 3c89a65..e66b04e 100644 --- a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -1,25 +1,32 @@ --- title: "Upgrading Orka on AWS" -description: "Upgrade your Orka cluster on AWS from 3.5 to 3.6. Covers ARM node tooling updates, SSH and SSM requirements, what's preserved during the upgrade, and what changes with the new Ansible-based upgrade path." +description: "Upgrade your Orka cluster on AWS from 3.5 to 3.6. Covers the Orka services upgrade and the new Ansible-based in-place tooling upgrade for ARM Mac nodes, including SSH and SSM requirements." --- -Upgrading Orka on AWS follows the same general process as MSDC-hosted upgrades: submit a support ticket, schedule a maintenance window, and MacStadium handles the upgrade. However, AWS deployments have specific requirements and a different upgrade mechanism for ARM Mac nodes. +Orka on AWS upgrades are self-service. You run the upgrade using the same CodeBuild project used for installation, pointed at the Orka 3.6 Ansible image. There are two parts: upgrading the Orka Kubernetes services, and upgrading the tooling on your ARM Mac nodes. + +Review the [3.6 release notes](/orka/orka-upgrades-and-release-notes/orka-36-release-notes) before upgrading. ## Before you upgrade -### Check your ARM node SSH configuration +### Verify your node tags -Starting with the 3.5 to 3.6 upgrade, MacStadium uses an Ansible-based process to update tooling on your ARM Mac nodes rather than replacing the AMI. This approach is significantly faster and preserves your node configuration. +The upgrade uses a dynamic Ansible inventory to identify your ARM nodes. Each ARM EC2 Mac instance must have the EC2 tag: -For this to work, your ARM EC2 Mac instances must accept SSH connections on port 22 for the `ec2-user` account using key-based authentication. EC2 Mac instances launched with a key pair have this enabled by default. Your security group must allow inbound TCP port 22 from MacStadium's CodeBuild subnets — confirm this with your MacStadium support contact when scheduling. +* **Key:** `role` +* **Value:** `orka-arm` -If your nodes cannot accept SSH, the upgrade can run over SSM instead. SSM upgrades require an S3 bucket in the same region as your ARM nodes for Ansible file transfer, and can take significantly longer (up to 4 hours). SSH is strongly recommended. +Instances without this tag will not be selected by the inventory and will be skipped. Verify this tag is applied to all ARM nodes before proceeding. -If you're unsure how your nodes are configured, check with your AWS administrator before scheduling. +### Check your ARM node SSH configuration + +Starting with the 3.5 to 3.6 upgrade, ARM node tooling is updated in place using Ansible rather than replacing the AMI. For this to work, your ARM EC2 Mac instances must accept SSH connections on port 22 for the `ec2-user` account using key-based authentication. EC2 Mac instances launched with a key pair have this enabled by default. Your security group must allow inbound TCP port 22 from your CodeBuild subnets. + +If your nodes cannot accept SSH, the upgrade can run over SSM instead. SSM upgrades require an S3 bucket in the same region as your ARM nodes for Ansible file transfer, and can take significantly longer (up to 4 hours). SSH is strongly recommended. ### Enabling SSH on nodes launched without a key pair -If your nodes were not launched with a key pair and do not accept SSH connections, you can use SSM to install MacStadium's CodeBuild public SSH key first, then proceed with the faster SSH upgrade path. This requires your instances to be SSM-managed: the `AmazonSSMManagedInstanceCore` policy must be attached to the instance profile, as recommended in the Orka on AWS setup guide. +If your nodes were not launched with a key pair and do not accept SSH connections, you can use SSM to install a public SSH key first, then proceed with the faster SSH upgrade path. This requires your instances to be SSM-managed: the `AmazonSSMManagedInstanceCore` policy must be attached to the instance profile. To set this up: @@ -42,7 +49,7 @@ To set this up: The identity running `send-command` requires `ssm:SendCommand` on the target instances and on the `AWS-RunShellScript` document, plus `ssm:GetCommandInvocation`. 3. Store the private key in AWS Secrets Manager as a plaintext secret. -4. Grant MacStadium's CodeBuild service role permission to read the secret: +4. Grant your CodeBuild service role permission to read the secret: ```json { @@ -52,38 +59,123 @@ To set this up: } ``` -5. Ensure your security group allows inbound TCP port 22 from MacStadium's CodeBuild subnets. -6. When scheduling, provide MacStadium support with the Secrets Manager secret ARN. +5. Ensure your security group allows inbound TCP port 22 from your CodeBuild subnets. -MacStadium will use the stored private key to connect over SSH for the upgrade. +Once the key is installed, proceed with the SSH upgrade path below. -### Verify your node tags +### Multi-region deployments -MacStadium uses a dynamic Ansible inventory to identify your ARM nodes during the upgrade. Each ARM EC2 Mac instance must have the EC2 tag: +The upgrade discovers ARM nodes in the region set by `AWS_DEFAULT_REGION` on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, run a separate upgrade pass per region and set `AWS_DEFAULT_REGION` to the target region before each run. -* **Key:** `role` -* **Value:** `orka-arm` +## Upgrading the Orka services -Instances without this tag will not be selected by the inventory and will be skipped during the upgrade. Verify this tag is applied to all ARM nodes before scheduling. +The Orka Kubernetes services are upgraded the same way they were installed: run the CodeBuild project pointed at the Orka 3.6 Ansible image. No additional configuration is required. -### Multi-region deployments +## Upgrading ARM node tooling + +### Over SSH (recommended) + +Store your SSH private key in AWS Secrets Manager as a plaintext secret. Give your CodeBuild service role permission to read it: + +```json +{ + "Effect": "Allow", + "Action": "secretsmanager:GetSecretValue", + "Resource": "arn:aws:secretsmanager:*:*:secret:*" +} +``` + +Update your CodeBuild buildspec to pull the key and run the upgrade playbook: + +```yaml +version: 0.2 + +env: + shell: bash + secrets-manager: + SSH_PRIVATE_KEY: "" + +phases: + install: + commands: + - apt-get update + - apt-get install -y openssh-client + - mkdir -p ~/.ssh + - printf "%s\n" "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa + - chmod 600 ~/.ssh/id_rsa + - ssh-keygen -y -f ~/.ssh/id_rsa >/dev/null + build: + commands: + - ansible-playbook -i arm.ssh.aws_ec2.yml configure-arm.yml --private-key ~/.ssh/id_rsa +``` + +Start the CodeBuild project. The upgrade takes under 10 minutes per node. + +### Over SSM -The upgrade discovers ARM nodes in the region configured on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, MacStadium runs a separate upgrade pass per region. When scheduling, provide the full list of regions where you have ARM nodes. +Use this method only if SSH cannot be enabled. SSM upgrades may take up to 4 hours per run. -## Requesting an upgrade +Requirements: -1. Review the [3.6 release notes](/orka/orka-upgrades-and-release-notes/orka-36-release-notes) and complete any preparation steps listed there. -2. Submit a ticket through the MacStadium portal. -3. Schedule a maintenance window via the link in your ticket. +1. Instances are SSM-managed (`AmazonSSMManagedInstanceCore` policy attached to the instance profile). +2. An S3 bucket in the same region as your ARM nodes. Set the `ANSIBLE_AWS_SSM_BUCKET` environment variable on the CodeBuild project to the bucket name. +3. CodeBuild service role permissions for Session Manager: + + ```json + { + "Effect": "Allow", + "Action": [ + "ssm:StartSession", + "ssm:TerminateSession", + "ssm:ResumeSession", + "ssm:DescribeSessions", + "ssm:GetConnectionStatus" + ], + "Resource": [ + "arn:aws:ec2:::instance/*", + "arn:aws:ssm:::document/SSM-SessionManagerRunShell", + "arn:aws:ssm:::session/*" + ] + } + ``` + +4. CodeBuild service role permissions on the S3 bucket: + + ```json + [ + { + "Effect": "Allow", + "Action": ["s3:PutObject", "s3:GetObject", "s3:DeleteObject"], + "Resource": "arn:aws:s3:::bucket-name/*" + }, + { + "Effect": "Allow", + "Action": ["s3:ListBucket", "s3:GetBucketLocation"], + "Resource": "arn:aws:s3:::bucket-name" + } + ] + ``` + +Run the upgrade playbook: + +```shell +ansible-playbook -i arm.ssm.aws_ec2.yml configure-arm.yml +``` ## Changing node values during the upgrade -By default, the upgrade reads all configuration from the running node and reapplies it automatically. If you want to change any of the following values as part of the upgrade, request the override when scheduling: +By default, the upgrade reads all configuration from the running node and reapplies it automatically. To change a value during the upgrade, pass it as an extra variable: -| Value | Notes | -|-------|-------| -| Node hostname | The node re-registers under the new name in Kubernetes. | -| License key | Replaces the license key applied to the node. | +| Value | Variable | +|-------|----------| +| Node hostname | `-e override_node_hostname=` | +| License key | `-e override_orka_engine_license_key=` | + +Example: + +```shell +ansible-playbook -i arm.ssh.aws_ec2.yml configure-arm.yml --private-key ~/.ssh/id_rsa -e override_node_hostname=arm-node-1 +``` ## What changes in 3.5 to 3.6 @@ -91,7 +183,9 @@ By default, the upgrade reads all configuration from the running node and reappl Previously, updating tooling on ARM nodes required replacing the EC2 Mac AMI: the instance had to be deleted, a new one provisioned (a process that takes approximately 2 hours), and the node's name, namespace, and custom tags had to be manually reapplied. -Starting with the 3.5 to 3.6 upgrade path, MacStadium updates ARM node tooling in place using Ansible over SSH. The upgrade takes under 10 minutes per node over SSH. The following are read from the running node and reapplied automatically: node name, node IP, cluster registration, license key, VM quota, and storage layout (including data volumes on instances with local NVMe). Running VMs are not interrupted. +Starting with the 3.5 to 3.6 upgrade path, ARM node tooling is updated in place using Ansible over SSH. The upgrade takes under 10 minutes per node. The following are read from the running node and reapplied automatically: node name, node IP, cluster registration, license key, VM quota, and storage layout (including data volumes on instances with local NVMe). Running VMs are not interrupted. + +AMI replacement is still required when the host operating system needs to be upgraded. Release notes will call this out explicitly when it applies. ### Upgrade Service is installed @@ -116,30 +210,14 @@ The permissions required for Orka to run have been tightened to least-privilege. * ECR access * Backup operations -If your deployment uses broad IAM credentials for Orka, you will need to update your IAM policies to align with the new scoped credential sets before or as part of the upgrade. MacStadium support will provide the updated permission requirements when scheduling your maintenance window. +If your deployment uses broad IAM credentials for Orka, update your IAM policies to align with the new scoped credential sets before upgrading. -## During the maintenance window +## After the upgrade -| Aspect | Notes | -|--------|-------| -| Maintenance window | Up to 3 hours. Must be scheduled with MacStadium support. | -| Access to the environment | Unavailable during the window. Restored at the original endpoint after completion. | -| Orka users | Persist. | -| Service Accounts | Persist. Tokens must be regenerated after the upgrade. | -| VMs and VM configs | VM configs persist. Running VMs on ARM nodes are preserved during the in-place tooling upgrade. Redeploy from VM configs after the upgrade if needed. | -| Images | Persist. | -| Image cache | Cleared from each node. | -| Namespaces and RoleBindings | Persist. | -| ARM node names and tags | Preserved with the Ansible-based upgrade path. | +1. [Download and install](/orka/orka-overview/tools-integrations) the Orka 3.6 CLI if you haven't already. +2. Regenerate Service Account tokens for any automated workflows: `orka3 serviceaccount token ` +3. Repopulate the image cache on your ARM nodes if needed: `orka3 imagecache add --all` Service Account tokens must be regenerated after this upgrade. Any automated workflows using service account tokens will fail until tokens are regenerated with `orka3 serviceaccount token `. - -## After the upgrade - -Once the upgrade is complete: - -1. [Download and install](/orka/orka-overview/tools-integrations) the Orka 3.6 CLI if you haven't already. -2. Regenerate Service Account tokens for any automated workflows. -3. Repopulate the image cache on your ARM nodes if needed: `orka3 imagecache add --all` From a3331537be154b00b71665e8dc0b84b31a3bbc45 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Tue, 16 Jun 2026 09:47:57 -0500 Subject: [PATCH 5/8] update: add per-run region override example for multi-region deployments Co-Authored-By: Claude Sonnet 4.6 --- .../upgrading-orka-on-aws.mdx | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx index e66b04e..466f15a 100644 --- a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -65,7 +65,12 @@ Once the key is installed, proceed with the SSH upgrade path below. ### Multi-region deployments -The upgrade discovers ARM nodes in the region set by `AWS_DEFAULT_REGION` on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, run a separate upgrade pass per region and set `AWS_DEFAULT_REGION` to the target region before each run. +The upgrade discovers ARM nodes in the region set by `AWS_DEFAULT_REGION` on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, run a separate upgrade pass per region. Override the region per run without editing the project: + +```shell +aws codebuild start-build --project-name \ + --environment-variables-override name=AWS_DEFAULT_REGION,value=us-west-2,type=PLAINTEXT +``` ## Upgrading the Orka services From 98d2591321115f8847cf6a7748f4fe308dda3cd6 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Tue, 16 Jun 2026 09:54:42 -0500 Subject: [PATCH 6/8] update: remove credential scoping section pending published permission sets Co-Authored-By: Claude Sonnet 4.6 --- .../upgrading-orka-on-aws.mdx | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx index 466f15a..28965a7 100644 --- a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -206,16 +206,6 @@ Orka no longer installs its own cert-manager if one is already present in the cl If your automation or tooling depends on Orka's cert-manager specifically, verify your setup before upgrading. -### Credential scoping: action required - -The permissions required for Orka to run have been tightened to least-privilege. Separate, minimal credential sets are now defined for: - -* Orka configuration -* Virtual Kubelet -* ECR access -* Backup operations - -If your deployment uses broad IAM credentials for Orka, update your IAM policies to align with the new scoped credential sets before upgrading. ## After the upgrade From d13253a82c4e1ccceaa33bc6b7d687f35caa73db Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Tue, 16 Jun 2026 09:56:14 -0500 Subject: [PATCH 7/8] update: clarify multi-region options per Ivan review Co-Authored-By: Claude Sonnet 4.6 --- orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx index 28965a7..739290f 100644 --- a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -65,7 +65,7 @@ Once the key is installed, proceed with the SSH upgrade path below. ### Multi-region deployments -The upgrade discovers ARM nodes in the region set by `AWS_DEFAULT_REGION` on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, run a separate upgrade pass per region. Override the region per run without editing the project: +The upgrade discovers ARM nodes in the region set by `AWS_DEFAULT_REGION` on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, you can either configure multiple steps in a single CodeBuild project (one per region), or trigger separate runs with a region override: ```shell aws codebuild start-build --project-name \ From 5d68369d1cd2358d0ebc41ef6c6fc9337111a8b5 Mon Sep 17 00:00:00 2001 From: Rin Oliver Date: Wed, 17 Jun 2026 13:39:10 -0500 Subject: [PATCH 8/8] update: address Ivan's second round of PR #260 review comments - Add AWS blog link for SSH key pair rotation in Enabling SSH section - Add link to installation guide in Upgrading the Orka services section - Add AWS_DEFAULT_REGION example variable in buildspec env.variables - Rename "Changing node values during the upgrade" to "Changing node values"; clarify playbook can run independently to rename nodes - Fix timing: Ansible runs nodes in parallel, typical deployment under 10 minutes total (not per node) - Update section heading and Upgrade Service copy: "no longer require provisioning a new EC2 Mac instance" instead of "AMI replacement" (AMI replacement still required for host OS upgrades) - Fix cert-manager language: opt-out via explicit config, not auto-detected presence - Remove SA token regeneration and image cache items from After the upgrade (MSDC-specific, not applicable to AWS self-service); remove Warning block Co-Authored-By: Claude Sonnet 4.6 --- .../upgrading-orka-on-aws.mdx | 28 +++++++++---------- 1 file changed, 13 insertions(+), 15 deletions(-) diff --git a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx index 739290f..3b880c0 100644 --- a/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx +++ b/orka/orka-upgrades-and-release-notes/upgrading-orka-on-aws.mdx @@ -63,6 +63,8 @@ To set this up: Once the key is installed, proceed with the SSH upgrade path below. +To rotate the stored SSH key pair automatically, see [How to use AWS Secrets Manager to securely store and rotate SSH key pairs](https://aws.amazon.com/blogs/security/how-to-use-aws-secrets-manager-securely-store-rotate-ssh-key-pairs/). + ### Multi-region deployments The upgrade discovers ARM nodes in the region set by `AWS_DEFAULT_REGION` on the CodeBuild project, defaulting to `us-east-1`. If your ARM nodes span multiple AWS regions, you can either configure multiple steps in a single CodeBuild project (one per region), or trigger separate runs with a region override: @@ -74,7 +76,7 @@ aws codebuild start-build --project-name \ ## Upgrading the Orka services -The Orka Kubernetes services are upgraded the same way they were installed: run the CodeBuild project pointed at the Orka 3.6 Ansible image. No additional configuration is required. +The Orka Kubernetes services are upgraded the same way they were installed: run the CodeBuild project pointed at the Orka 3.6 Ansible image. For setup details, see [Getting started with Orka on AWS](/orka/orka-on-aws-and-on-prem/orka-on-aws-getting-started). No additional configuration is required. ## Upgrading ARM node tooling @@ -97,6 +99,8 @@ version: 0.2 env: shell: bash + variables: + AWS_DEFAULT_REGION: "us-east-1" secrets-manager: SSH_PRIVATE_KEY: "" @@ -114,7 +118,7 @@ phases: - ansible-playbook -i arm.ssh.aws_ec2.yml configure-arm.yml --private-key ~/.ssh/id_rsa ``` -Start the CodeBuild project. The upgrade takes under 10 minutes per node. +Start the CodeBuild project. Ansible runs nodes in parallel; a typical deployment completes in under 10 minutes. For large numbers of nodes, timing scales with node count. ### Over SSM @@ -167,9 +171,9 @@ Run the upgrade playbook: ansible-playbook -i arm.ssm.aws_ec2.yml configure-arm.yml ``` -## Changing node values during the upgrade +## Changing node values -By default, the upgrade reads all configuration from the running node and reapplies it automatically. To change a value during the upgrade, pass it as an extra variable: +By default, all configuration is read from the running node and reapplied automatically. You can also run the playbook independently to change a value without a full upgrade, for example to rename a node. Pass the value as an extra variable: | Value | Variable | |-------|----------| @@ -184,17 +188,17 @@ ansible-playbook -i arm.ssh.aws_ec2.yml configure-arm.yml --private-key ~/.ssh/i ## What changes in 3.5 to 3.6 -### ARM node tooling updates no longer require AMI replacement +### ARM node tooling updates no longer require provisioning a new EC2 Mac instance Previously, updating tooling on ARM nodes required replacing the EC2 Mac AMI: the instance had to be deleted, a new one provisioned (a process that takes approximately 2 hours), and the node's name, namespace, and custom tags had to be manually reapplied. -Starting with the 3.5 to 3.6 upgrade path, ARM node tooling is updated in place using Ansible over SSH. The upgrade takes under 10 minutes per node. The following are read from the running node and reapplied automatically: node name, node IP, cluster registration, license key, VM quota, and storage layout (including data volumes on instances with local NVMe). Running VMs are not interrupted. +Starting with the 3.5 to 3.6 upgrade path, ARM node tooling is updated in place using Ansible over SSH. Ansible runs nodes in parallel; a typical deployment completes in under 10 minutes. The following are read from the running node and reapplied automatically: node name, node IP, cluster registration, license key, VM quota, and storage layout (including data volumes on instances with local NVMe). Running VMs are not interrupted. AMI replacement is still required when the host operating system needs to be upgraded. Release notes will call this out explicitly when it applies. ### Upgrade Service is installed -As part of the 3.6 upgrade, the Orka Upgrade Service is deployed to your cluster. This enables smoother tooling updates in future Orka releases without requiring AMI replacement. +As part of the 3.6 upgrade, the Orka Upgrade Service is deployed to your cluster. This enables smoother tooling updates in future Orka releases without requiring you to provision new EC2 Mac instances. ### AWS credentials no longer required for artifact distribution @@ -202,17 +206,11 @@ Orka binaries and container images are now distributed publicly via CloudFront. ### cert-manager behavior change -Orka no longer installs its own cert-manager if one is already present in the cluster. If your cluster runs its own cert-manager and you previously experienced version or configuration conflicts with Orka's bundled installation, those conflicts are resolved in 3.6. +Orka no longer installs its own cert-manager when configured to skip the bundled installation. This is not auto-detected: if you want Orka to skip cert-manager, configure the installation explicitly. If your cluster runs its own cert-manager and you previously experienced version or configuration conflicts with Orka's bundled installation, those conflicts can now be avoided by opting out. If your automation or tooling depends on Orka's cert-manager specifically, verify your setup before upgrading. ## After the upgrade -1. [Download and install](/orka/orka-overview/tools-integrations) the Orka 3.6 CLI if you haven't already. -2. Regenerate Service Account tokens for any automated workflows: `orka3 serviceaccount token ` -3. Repopulate the image cache on your ARM nodes if needed: `orka3 imagecache add --all` - - - Service Account tokens must be regenerated after this upgrade. Any automated workflows using service account tokens will fail until tokens are regenerated with `orka3 serviceaccount token `. - +[Download and install](/orka/orka-overview/tools-integrations) the Orka 3.6 CLI if you haven't already.