fix(tracking): finalize MLflow runs on SIGTERM instead of leaving them RUNNING

[EazyReal]

What

When a training process is killed by a signal (e.g. SLURM scancel → SIGTERM), its MLflow run is left in status="RUNNING" with end_time=null forever. Tracking.finish() (which calls mlflow.end_run()) is only invoked from Tracking.__del__, and Python does not guarantee __del__ runs on signal-driven termination.

Fix

Implements the approach suggested in the issue:

• Tracking.finish(status="FINISHED") and the MLflow adapter forward status to end_run.
• Register an atexit hook and a SIGTERM handler that finalize the run (status="KILLED" on signal). The handler chains to the previous disposition so normal process teardown is preserved, and is skipped off the main thread (e.g. Ray workers) where signal.signal raises.
• finish() is idempotent, so the signal / atexit / __del__ paths end the run exactly once.

CPU tests cover default/forwarded status, idempotency, and the SIGTERM path.

Fixes #1516

[gemini-code-assist]

Code Review

This pull request introduces robust signal and exit handling to the Tracking class to ensure MLflow runs are properly finalized (e.g., marked as KILLED) during abnormal process termination, such as a SIGTERM from SLURM. It also adds corresponding unit tests to verify status forwarding, idempotency, and signal handling. The review feedback highlights a critical issue where registering bound methods directly with atexit and signal creates strong reference cycles, preventing garbage collection of the Tracking instance. To resolve this, the reviewer suggests using weakref wrappers, unregistering the handlers upon completion, and updating the tests and imports accordingly.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[EazyReal]

Good catch on the reference cycle. Fixed: the atexit hook and SIGTERM handler now reference self through a weakref (so the instance stays collectable and __del__ keeps working), and finish() unregisters the atexit hook and restores the previous SIGTERM handler. Added a test asserting the instance is GC'd after finish().

[EazyReal]

@SumanthRH @erictang000 Could I get a quick review on this tracking finalization fix? It closes MLflow runs on SIGTERM so interrupted jobs do not stay RUNNING forever.