feat: client side tracing upload path

[hweawer]

Summary

Implement end-to-end distributed tracing for Docker blob upload paths in Kraken, enabling observability from proxy → origin. This PR adds OpenTelemetry instrumentation to the client upload flow with hierarchical span creation and trace context propagation.

Motivation

Debugging slow or failed Docker pushes currently requires piecing together logs across multiple services with no correlation. This PR enables:

• End-to-end visibility: Trace a blob upload from Docker client through proxy, origin, and writeback
• Performance analysis: Identify bottlenecks in chunked uploads, replication, or backend writes
• Retry debugging: See which origins were attempted and why they failed

[Anton-Kalpakchiev]

considering we will be trying to debug which part of the upload failed, does it make sense to enrich this err message?

[Anton-Kalpakchiev]

considering the normal client's tracer is called kraken-origin-client, wouldnt it make sense to call this one kraken-origin-cluster-client?

[Copilot]

Pull request overview

Adds OpenTelemetry-based tracing to the Docker blob upload flow so a blob push can be correlated across proxy → origin via propagated trace context and structured spans.

Changes:

• Introduces httputil.SendTracingContext to wrap HTTP transports with otelhttp for trace propagation (and HTTP client spans).
• Threads context.Context through blob upload APIs (blobclient.Client.UploadBlob, blobclient.ClusterClient.UploadBlob, server replicate-to-remote) and updates relevant tests/mocks.
• Adds upload-path spans/attributes in the proxy transferer and origin cluster/origin clients.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
utils/httputil/httputil.go	Adds SendTracingContext option and wraps transport with otelhttp.NewTransport.
origin/blobclient/uploader.go	Propagates tracing context through chunked upload HTTP calls (POST/PATCH/PUT).
origin/blobclient/client.go	Adds tracer + span around upload and changes UploadBlob signature to accept context.Context.
origin/blobclient/cluster_client.go	Adds client-side span around multi-origin upload attempts and threads context into per-origin UploadBlob calls.
lib/dockerregistry/transfer/rw_transferer.go	Starts a new span for registry upload and passes context into origin cluster upload.
origin/blobserver/server.go	Passes request context into replicate-to-remote upload call.
origin/blobserver/server_test.go	Updates tests for new context parameter and gomock expectations.
origin/blobserver/cluster_client_test.go	Updates tests to pass context and adjust gomock expectations.
mocks/origin/blobclient/client.go	Updates mock for Client.UploadBlob(ctx, ...).
mocks/origin/blobclient/clusterclient.go	Updates mock for ClusterClient.UploadBlob(ctx, ...).
lib/persistedretry/writeback/task.go	Minor formatting change.

Comments suppressed due to low confidence (1)

origin/blobclient/uploader.go:178

• runChunkedUploadHelper calls patch once per chunk; because each patch uses SendTracingContext, this will generate an otelhttp client span for every chunk request. For large uploads or small chunk sizes this can create very large traces and high telemetry volume. Consider propagating trace headers without creating per-chunk HTTP spans (or gating per-chunk spans behind sampling/config), while keeping higher-level spans for the overall upload.

func (c *uploadClient) patch(
	d core.Digest, uid string, start, stop int64, chunk io.Reader,
) error {
	_, err := httputil.Patch(
		fmt.Sprintf("http://%s/namespace/%s/blobs/%s/uploads/%s",
			c.addr, url.PathEscape(c.namespace), d, uid),
		httputil.SendTracingContext(c.ctx),
		httputil.SendBody(chunk),
		httputil.SendHeaders(map[string]string{
			"Content-Range": fmt.Sprintf("%d-%d", start, stop),
		}),
		httputil.SendTLS(c.tls))

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

SendTracingContext wraps the request transport with otelhttp.NewTransport, which does more than header propagation (it also creates an HTTP client span per request). The comment currently says it only enables propagation; please update the doc to reflect span creation, or change the implementation to inject trace headers without generating per-request spans if propagation-only is the intent.

[Copilot]

This interface change requires updating all call sites to pass a context. The repo still has callers using the old signature (e.g., origin/blobclient/client_test.go calls client.UploadBlob(namespace, d, ...)), which will fail to compile.

Suggested change

	DuplicateUploadBlob(namespace string, d core.Digest, blob io.Reader, delay time.Duration) error
	DuplicateUploadBlob(ctx context.Context, namespace string, d core.Digest, blob io.Reader, delay time.Duration) error