feat(provider): add External Metrics provider

[jlore-decathlon]

Proposed addition

The current Datadog metric provider relies on their Metric API.
However, this API has pretty low rate limits, and people with a moderately sized infrastructure tend to reach these limits quite easily when scaling their usage of Flagger or datadog-based autoscaling (like KEDA).

Datadog offers a more scalable alternative by making its Cluster Agent batch requests by groups of 35 see Cluster Agent Autoscaling Metrics. It then makes these metrics available within the cluster by exposing an endpoint following Kubernetes External Metrics API.

Note

This endpoint is not documented by Datadog, as they expect people to have the agent register against the control plane as the cluster's external metrics provider and then making these metrics available through k8s API Server, removing the need to query the endpoint directly.
However, by implementing a kubernetes API, its behavior is predictable and stable enough to be used directly.

We've relied on the way KEDA implemented a similar feature during design and implementation. However, Flagger is not an autoscaling solution so we're not going to mimic the metric proxy Keda operates. We simply propose to query the external metric server directly. By doing this, we also chose to make the provider generic and compatible with any external metrics server. The downside is that we cannot abstract the way datadog names its metrics which isn't trivial.

fix: #1235

Any alternatives you've considered?

We've pondered modifying the Datadog metric provider instead of making an external metrics provider. But we felt that this had the benefit of making other external metric providers compatible and kept the code datadog-agnostic.

We could theoretically make it even more generic and use any kubernetes metric API (standard, Custom or External), but I think Flagger already offers this

Disclaimer

• This PR was peer programmed with my colleague @mveroone.
• We're not Go developers, yet we did our best to follow the project's coding conventions and guidelines. Any feedback is welcome and we'll be happy to rework any part of that contribution you think needs it.
• AI disclosure : AIL 1 (see https://danielmiessler.com/blog/ai-influence-level-ail). Minor code autocomplete, but mostly manual coding and writing.
• We've built the docker image and tested end-to-end on one of our GKE clusters against Datadog cluster agent endpoint. it seems to work as expected, but the lack of feedback from Flagger leaves room for some doubts, but we were unsure if adding some logging was a good idea.

[aryan9600]

thank you for opening this PR!

[mveroone]

Note : if that's okay, we'll squash commits after a few rounds of review so we can fix the DCO

[aryan9600]

looking good - a few nits

[aryan9600]

lgtm! 🎖️

please squash this into 1-2 commits and sign them off, thanks

[mveroone]

lgtm! 🎖️

please squash this into 1-2 commits and sign them off, thanks

@aryan9600 Done !
Thanks for the support.

[aryan9600]

ci is failing because of unformatted code - could you run make fmt and push again? thanks

[jlore-decathlon]

ci is failing because of unformatted code - could you run make fmt and push again? thanks

@aryan9600 Done

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 87.30159% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 30.19%. Comparing base (12ee6cb) to head (9b37d18).
⚠️ Report is 91 commits behind head on main.

Files with missing lines	Patch %	Lines
pkg/metrics/providers/externalmetrics.go	87.30%	4 Missing and 4 partials ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1863      +/-   ##
==========================================
- Coverage   39.44%   30.19%   -9.25%     
==========================================
  Files         287      288       +1     
  Lines       22706    18532    -4174     
==========================================
- Hits         8956     5596    -3360     
+ Misses      12777    12201     -576     
+ Partials      973      735     -238     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.