Node Storage Cleanup

[defo89]

Summary

When Server transitions from Reserved to Available metal-operator should support wiping the storage.

Current behaviour:

• During the rolling update, ServerClaim is removed and Server transitions to Available.
• Server was using disk install -> data on disk is persisted including possible operational data (kubelet certs, static IP address configuration).
• Server is used by another ServerClaim but network boot fails:
  • Server might re-join the cluster without Cluster machinery (Gardener, Cluster-API) being aware of this.
  • Server operates with previously configured IP address which is already "released" by IPAM controller.
  • Cluster operations is disrupted because of two above issues (wrong scheduling, duplicate IP addressing).

Possible options:

• Cleanup via Redfish/BMC
• Cleanup via ServerBootConfiguration (risk of network boot failing -> same issues as described above)

There should be 2 policies but this is open for discussion:

• Cleanup root disk only. This will help to redeploy rook/ceph clusters and persists OSDs on node redeployment.
• Multi-tenancy: all storage disks will be wiped on Server and it can be safely used by next tenant.

[coderabbitai]

📝 CodeRabbit Plan Mode

Generate an implementation plan and prompts that you can use with your favorite coding agent.

• Create Plan

Examples

• Example 1
• Example 2

---

🔗 Related PRs

#581 - Transition Server to Maintenance in case of pending BIOS settings exist [merged]

---

🧪 Issue enrichment is currently in open beta.

You can configure auto-planning by selecting labels in the issue_enrichment configuration.

To disable automatic issue enrichment, add the following to your .coderabbit.yaml:

issue_enrichment:
  auto_enrich:
    enabled: false

💬 Have feedback or questions? Drop into our discord!

[stefanhipfel]

A disk secure erase can be used via redfish. I think in most cases it will create a job, which does the erase.
So the controller needs to watch over a job and wait until finished.
@afritzler i think this would be done in a separate controller. But we also talked about doing this via our image in discovery phase.

[afritzler]

Isn't disk sanitization only supported on HPE machines via Redfish? I think the idiomatic for us to achieve that would be to introduce a Cleanup state (between Reserved and Available). This could be optional at the beginning. Here we would just do yet another network boot with an agent which does the cleanup on the host side.

[stefanhipfel]

We should make the cleanup state configurable.
Btw dell allows disk erase via redfish as well

https://github.com/dell/iDRAC-Redfish-Scripting/blob/master/Redfish%20Python/SecureEraseDevicesREDFISH.py

[defo89]

Isn't disk sanitization only supported on HPE machines via Redfish? I think the idiomatic for us to achieve that would be to introduce a Cleanup state (between Reserved and Available). This could be optional at the beginning. Here we would just do yet another network boot with an agent which does the cleanup on the host side.

If we rely on network boot with an agent to perform the cleanup, there is a risk of failed network boot causing described issues. I agree with @stefanhipfel that it should be configurable and end user can choose the approach based on the Redfish support in their fleet.

[afritzler]

Fair point. But those will be all oem extension operations right? But you are right maybe we should start with those, as they are "hopefully" less error prone.