Failed to fetch blobs because they do not exist remotely

[tdarci]

We are running into this issue where digests are missing. Here's an example log line:

ERROR: /var/lib/jenkins/workspace/godel/users/server/BUILD.bazel:61:8: GoCompilePkg users/server/go_default_test.internal.a failed: Failed to fetch blobs because they do not exist remotely.: Missing digest: 85d96d201e4303b96b1e10e4e6bdbe5d86daf9a3dec7ee8f79258adfa3655013/3483454 for bazel-out/k8-fastbuild/bin/protobufs/protos/events/go_default_library.x

What is happening when we get this message?
And how do we find our way out when this starts happening (once it begins, it's all over the place)?

Any help much appreciated.

We are running bazel-remote-cache in kubernetes. Cache directory is on an EFS volume.
Prior to this issue starting, our remote cache was unavailable due to crashing. Issue arose once cache started back up.

[mostynb]

What might have happened is that some of the files in bazel-remote's cache directory have become corrupt. One simple way to recover from this issue would be to stop bazel-remote, delete the contents of the cache directory, and then restart bazel-remote.

Which version of bazel-remote are you using? The latest version (v2.6.1) deletes invalid cache items as they are found, so the cache can recover over time without intervention.

[tdarci]

Thank you. It seems we are on 2.6.1, so that's good:

❯ curl http://bazel-remote-cache.OURDOMAIN/status
{
 "CurrSize": 57902690304,
 "UncompressedSize": 148748591104,
 "ReservedSize": 0,
 "MaxSize": 1073741824000,
 "NumFiles": 95327,
 "ServerTime": 1761864876,
 "GitCommit": "f46bc2030d3f30604d79ef4bf040e3a9c7a4ff89",
 "GitTags": "v2.6.1",
 "NumGoroutines": 11
}

When you say

delete the contents of the cache directory

I'm assuming you mean the directory we have in BAZEL_REMOTE_DIR for the bazel-remote server, correct? We did try this, and it made the problem go away for now. But, of course, it makes for some downtime and headache.

I just added these to our build/test commands, in hopes they will help when the remote cache is sorting itself out:

 --remote_local_fallback --remote_retries=2

This is in addition to these:

--remote_cache=${BAZEL_REMOTE_CACHE} --experimental_remote_cache_eviction_retries=10

Thanks again for the assistance.

[mostynb]

When you say

delete the contents of the cache directory

I'm assuming you mean the directory we have in BAZEL_REMOTE_DIR for the bazel-remote server, correct? We did try this, and it made the problem go away for now. But, of course, it makes for some downtime and headache.

Yes- exactly. If there is a lot of corruption and your cache is large, this is probably the solution with the shortest downtime.

Do you have any details of the crash?

I just added these to our build/test commands, in hopes they will help when the remote cache is sorting itself out:

 --remote_local_fallback --remote_retries=2

This is in addition to these:

--remote_cache=${BAZEL_REMOTE_CACHE} --experimental_remote_cache_eviction_retries=10

Thanks again for the assistance.

No problem. If these bazel flags seem to help, we can add them to the documentation.

[tdarci]

I will report back on whether --remote_local_fallback and --remote_retries=2 help. So far not seeing any "missing digest" messages, so nothing to verify.

As for crashing, I unfortunately don't have details, other than the pod was failing liveness check. (this is running in kubernetes). We took the liveness check out to get the pods up again, but have since reinstated it, and all is happy.

          liveness_probe {
            failure_threshold     = 5
            initial_delay_seconds = 5
            period_seconds        = 1
            success_threshold     = 1
            timeout_seconds       = 1
            grpc {
              port    = "9092"
              service = "/grpc.health.v1.Health/Check"
            }
          }

Thanks again for the help.