Skip to content

feat: add liveness and readiness probes to dialog deployment - #398

Open
HussainAther wants to merge 1 commit into
Hubs-Foundation:masterfrom
HussainAther:add-dialog-probes
Open

feat: add liveness and readiness probes to dialog deployment#398
HussainAther wants to merge 1 commit into
Hubs-Foundation:masterfrom
HussainAther:add-dialog-probes

Conversation

@HussainAther

Copy link
Copy Markdown

What?

This PR adds adds Kubernetes readiness and liveness probes to the dialog deployment in community-edition/generate_script/hcce.yam.


Why?

The dialog service currently =does not have any Kubernetes health monitoring. This PR adds the readiness and liveness that the probes allows Kubernetes to:

  • delay traffic until the service is ready
  • automatically restart the container if it becomes unresponsive

For these reasons, this is aligns with standard Kubernetes best practices for service reliability for creating a PR.


Notes

  • This work the uses the health endpoint referenced in Hubs-Foundation/dialog#61.
  • The initial timing values are also intentionally conservative, since the readiness timing is not yet well characterized. (Coudl be future work?)
  • Probes are then configured over HTTPS on port 4443, matching the existing dialog deployment.

Examples

N/A


How to test

  1. Apply the updated deployment (or regenerate manifests if using the CE scripts).

  2. Deploy to a Kubernetes cluster:

    kubectl apply -f community-edition/generate_script/hcce.yam
  3. Verify the dialog pod:

    kubectl get pods
    kubectl describe pod <dialog-pod-name>
  4. Confirm that , based on the results of these actions :

    • readiness and liveness probes are present in the pod spec
    • the pod transitions to Ready
  5. Optionally:

    • simulate failure (e.g., put some sort of block on the endpoint or a similar sort of signal one may have to look for) and then look to confirm if Kubernetes restarts the container

Documentation of functionality

No additional documentation changes are required. This is a deployment configuration improvement within hcce.yam.


Limitations

  • The exact readiness timing characteristics are not fully known, so probe timings may need adjustment based on real-world behavior.
  • This assumes the health endpoint referenced in dialog#61 is stable and available.

Alternative implementations considered

  • Using more advanced health signals (e.g., memory usage thresholds) for liveness detection, but a simple HTTP probe was chosen as a practical and reliable starting point.

Open questions

  • Whether /health is the best long-term endpoint or if /healthz (used elsewhere in the repo) should be standardized across services.
  • Whether probe timing values should be tuned further based on observed startup behavior.

Additional details or related context

@DougReeder DougReeder left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can check that the HTTP endpoints return useful values by running the container locally. (The DNS name may be different than the service DNS name that Kubernetes provides.)

Comment on lines +1110 to +1112
path: https://localhost/health
port: 4443
scheme: HTTPS

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The probes are sent from the Kubernetes control plane; they are not run inside the container. So, they use the internal DNS names and the pod ports. Also, there is no /health path in dialog. So, I think these values need to be:

path: /meta
port: 7000
scheme: HTTP

path: https://localhost/health
port: 4443
scheme: HTTPS
initialDelaySeconds: 10

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you timed how long it usually takes dialog to start up?

port: 4443
scheme: HTTPS
initialDelaySeconds: 30
periodSeconds: 10

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 564 of generate_script/hcce.yam has the livenessProbe for reticulum. The values there are more typical for httpGet probes.

https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants