Skip to content

feat(clickhouse): Terraform-managed Altinity operator, dedicated Node… - #1138

Open
meetreks wants to merge 1 commit into
awslabs:spark-workshopfrom
meetreks:add-clickhouse-lab
Open

feat(clickhouse): Terraform-managed Altinity operator, dedicated Node…#1138
meetreks wants to merge 1 commit into
awslabs:spark-workshopfrom
meetreks:add-clickhouse-lab

Conversation

@meetreks

Copy link
Copy Markdown
Contributor

…Pool, tuned storage

Add an in-workshop ClickHouse lab that sits alongside the Spark labs (and, when enabled, the Kafka lab) on the same EKS Auto Mode cluster created by analytics/terraform/spark-k8s-operator/.

Terraform side (behind var.enable_clickhouse_lab, default true):

  • clickhouse-operator.tf — Altinity Cluster Operator via Helm chart https://helm.altinity.com/altinity-clickhouse-operator@0.24.5, plus a tuned clickhouse-gp3 StorageClass provisioned at 16000 IOPS / 1000 MiB/s with reclaimPolicy: Retain and allowVolumeExpansion: true.
  • variables.tf — new vars: enable_clickhouse_lab (bool) and altinity_operator_version (string, defaults to 0.24.5 — later 0.27.x versions had watch-event reliability issues on Auto Mode during earlier testing).
  • manifests/automode/nodepool-clickhouse.yaml — dedicated Karpenter NodePool with workload=clickhouse:NoSchedule taint. m/r Gen 5+ Nitro, On-Demand only, sizes 2xlarge..8xlarge. Auto-discovered by the existing fileset('nodepool*.yaml') pattern in eks.tf, so no eks.tf change is needed.

Lab manifests + scripts (analytics/clickhouse/):

  • keeper.yaml — ClickHouseKeeperInstallation with 3 replicas, AZ-level podAntiAffinity, karpenter.sh/do-not-disrupt annotation, PVCs on the clickhouse-gp3 StorageClass.
  • cluster.yaml — ClickHouseInstallation with 1 shard x 3 replicas, ReplicatedMergeTree coordinated through ClickHouse Keeper (not ZooKeeper — the daemon behind the zookeeper.nodes field is Keeper's ZK-compatible C++ implementation). Points at the Keeper service at keeper-keeper.clickhouse.svc.cluster.local:2181 — the Altinity operator names Services as -, worth noting because keeper.clickhouse.svc.cluster.local doesn't resolve.
  • deploy-clickhouse.sh — preflight-checks the operator, StorageClass, and NodePool, then applies both CRs and waits for status=Completed.
  • cleanup.sh — deletes the CRs and their PVCs, leaves the operator, StorageClass, and NodePool for Terraform to reclaim on destroy.
  • install-altinity.sh — fallback operator install for clusters not built by this workshop (adds the Altinity Helm repo and installs the pinned operator version into the clickhouse namespace).
  • README.md — architecture overview, deploy/verify/cleanup walkthrough, storage tier and sizing sections mirroring the Kafka lab's shape.

Verified end-to-end against a scratch spike on an Auto Mode cluster: Karpenter provisioned 3 m5a.2xlarge nodes across us-west-2a/2b/2c, one CH replica and one Keeper member on each node. CREATE TABLE ON CLUSTER 'replicated' with ReplicatedMergeTree accepted, INSERT on replica 0 propagated to replicas 1 and 2 via Keeper coordination within seconds.

What does this PR do?

🛑 Please open an issue first to discuss any significant work and flesh out details/direction. When we triage the issues, we will add labels to the issue like "Enhancement", "Bug" which should indicate to you that this issue can be worked on and we are looking forward to your PR. We would hate for your time to be wasted.
Consult the CONTRIBUTING guide for submitting pull-requests.

Motivation

More

  • Yes, I have tested the PR using my local account setup (Provide any test evidence report under Additional Notes)
  • Mandatory for new blueprints. Yes, I have added a example to support my blueprint PR
  • Mandatory for new blueprints. Yes, I have updated the website/docs or website/blog section for this feature
  • Yes, I ran pre-commit run -a with this PR. Link for installing pre-commit locally

For Moderators

  • E2E Test successfully complete before merge?

Additional Notes

…Pool, tuned storage

Add an in-workshop ClickHouse lab that sits alongside the Spark labs
(and, when enabled, the Kafka lab) on the same EKS Auto Mode cluster
created by analytics/terraform/spark-k8s-operator/.

Terraform side (behind var.enable_clickhouse_lab, default true):

- clickhouse-operator.tf — Altinity Cluster Operator via Helm chart
  https://helm.altinity.com/altinity-clickhouse-operator@0.24.5, plus a
  tuned clickhouse-gp3 StorageClass provisioned at 16000 IOPS / 1000
  MiB/s with reclaimPolicy: Retain and allowVolumeExpansion: true.
- variables.tf — new vars: enable_clickhouse_lab (bool) and
  altinity_operator_version (string, defaults to 0.24.5 — later 0.27.x
  versions had watch-event reliability issues on Auto Mode during
  earlier testing).
- manifests/automode/nodepool-clickhouse.yaml — dedicated Karpenter
  NodePool with workload=clickhouse:NoSchedule taint. m/r Gen 5+ Nitro,
  On-Demand only, sizes 2xlarge..8xlarge. Auto-discovered by the
  existing fileset('nodepool*.yaml') pattern in eks.tf, so no eks.tf
  change is needed.

Lab manifests + scripts (analytics/clickhouse/):

- keeper.yaml — ClickHouseKeeperInstallation with 3 replicas, AZ-level
  podAntiAffinity, karpenter.sh/do-not-disrupt annotation, PVCs on the
  clickhouse-gp3 StorageClass.
- cluster.yaml — ClickHouseInstallation with 1 shard x 3 replicas,
  ReplicatedMergeTree coordinated through ClickHouse Keeper (not
  ZooKeeper — the daemon behind the zookeeper.nodes field is Keeper's
  ZK-compatible C++ implementation). Points at the Keeper service at
  keeper-keeper.clickhouse.svc.cluster.local:2181 — the Altinity
  operator names Services as <CR-name>-<cluster-name>, worth noting
  because keeper.clickhouse.svc.cluster.local doesn't resolve.
- deploy-clickhouse.sh — preflight-checks the operator, StorageClass,
  and NodePool, then applies both CRs and waits for status=Completed.
- cleanup.sh — deletes the CRs and their PVCs, leaves the operator,
  StorageClass, and NodePool for Terraform to reclaim on destroy.
- install-altinity.sh — fallback operator install for clusters not
  built by this workshop (adds the Altinity Helm repo and installs
  the pinned operator version into the clickhouse namespace).
- README.md — architecture overview, deploy/verify/cleanup walkthrough,
  storage tier and sizing sections mirroring the Kafka lab's shape.

Verified end-to-end against a scratch spike on an Auto Mode cluster:
Karpenter provisioned 3 m5a.2xlarge nodes across us-west-2a/2b/2c, one
CH replica and one Keeper member on each node. CREATE TABLE ON CLUSTER
'replicated' with ReplicatedMergeTree accepted, INSERT on replica 0
propagated to replicas 1 and 2 via Keeper coordination within seconds.
@meetreks
meetreks force-pushed the add-clickhouse-lab branch from e585b76 to 9e5c491 Compare July 26, 2026 11:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant