feat: keep machine ingestion time linear as fleet size scales

[shayan1995]

Is this a new feature, an enhancement, or a change to existing functionality?

Enhancement

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem this feature solves

Machine ingestion time grows superlinearly with fleet size. In the machine-a-tron scale testing campaign (docs/development/machine-a-tron-scale-testing.md), ingesting 3,000 machines (1,000 hosts × 2 DPUs) completed end-to-end in ~25 minutes, but 13,500 machines (4,500 hosts × 2 DPUs) took on the order of hours — far beyond the ~2 hours that linear scaling would predict, even after discounting environmental noise in that run.

The mechanisms are known and documented in the scale-testing notes:

1. Machine identification/creation only runs at the end of a completed explore_site cycle, and identification rebuilds explored_managed_hosts from scratch every cycle. Per-cycle overhead therefore grows with fleet size while the creation opportunity per cycle stays capped, producing a sawtooth of 40–240 machines per cycle at 13.5k endpoints.
2. explorations_per_run trades sweep throughput against creation latency: a full sweep of 13,500 endpoints takes ~113 cycles at the default of 120, versus ~25 cycles for 3,000 endpoints, so creation latency compounds with scale.
3. DHCP discovery is effectively serial at ~110 interfaces/min, which is ~2 hours of discovery alone at 13,500 interfaces before downstream phases can drain.

Feature Description

As an operator standing up a large site (~4,500 hosts / 13,500 BMC endpoints), I want ingestion time to scale approximately linearly with the number of endpoints, so that fleet bring-up time is predictable and large sites do not pay a superlinear penalty.

Describe your ideal solution

• Decouple machine identification/creation from the exploration cycle boundary (or run it incrementally as endpoints complete), so creation throughput is not gated on full-cycle completion (see open question 6 in docs/development/machine-a-tron-scale-testing.md).
• Make identification incremental instead of rebuilding explored_managed_hosts every cycle.
• Investigate parallelizing/batching DHCP discovery so it is not a serial bottleneck at large interface counts.
• Add per-phase timing metrics (DHCP, exploration, preingestion, creation) so linearity can be verified in CI/scale runs and regressions detected.

Describe any alternatives you have considered

Tuning explorations_per_run per site size mitigates but does not remove the superlinearity, and the right value is non-obvious (documented in the scale-testing notes as "cycle economics").

Additional context

Measured data from the scale campaign on a 3-node dev cluster (dev-sized Postgres):

Fleet	Endpoints	End-to-end ingestion
1,000 hosts × 2 DPUs	3,000	~25 min (≈120 machines/min end-to-end; creation phase peaked at ≈240 machines/min)
4,500 hosts × 2 DPUs	13,500	~15 h wall clock (unattended, including connectivity outages; linear extrapolation predicts ~2 h)

See docs/development/machine-a-tron-scale-testing.md for the full issue log and stage-3 observations.

Code of Conduct

• I agree to follow NVIDIA Infra Controller's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request