SetupWellKnownPaths wwwroot enumeration fails only on PowerShell 7.6 (.NET 10) worker — Directory.EnumerateFiles regression in .NET 10.0.9, fixed in 10.0.11

[andystaples]

Summary

FunctionLoader.SetupWellKnownPaths throws Could not find file 'C:\home\site\wwwroot' only on the PowerShell 7.6 worker, while 7.2 / 7.4 workers on the same stamps are unaffected. The root cause is a behavioral change in Directory.EnumerateFiles between .NET 8 and .NET 10 (specifically introduced in .NET 10.0.9), interacting with certain Azure Files / content-share filesystem drivers. This issue is the root-cause counterpart to the hardening work in #1146 — #1146 makes the enumeration resilient; this issue documents why the enumeration started failing in the first place on 7.6.

Observed failure

Microsoft.Azure.WebJobs.Script.Workers.Rpc.RpcException : Result: Failure
Exception: Could not find file 'C:\home\site\wwwroot'.
Stack:    at System.IO.Enumeration.FileSystemEnumerator`1.FindNextEntry()
   at System.IO.Enumeration.FileSystemEnumerator`1.MoveNext()
   at System.Linq.Enumerable.TryGetFirstNonIterator[TSource](IEnumerable`1 source, Boolean& found)
   at System.Linq.Enumerable.FirstOrDefault[TSource](IEnumerable`1 source)
   at Microsoft.Azure.Functions.PowerShellWorker.FunctionLoader.SetupWellKnownPaths(FunctionLoadRequest request, String managedDependenciesPath)
   at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.SetupAppRootPathAndModulePath(FunctionLoadRequest functionLoadRequest, String managedDependenciesPath)
   at Microsoft.Azure.Functions.PowerShellWorker.RequestProcessor.ProcessFunctionLoadRequest(StreamingMessage request)

Note the failure is at FindNextEntry() / MoveNext() — the directory handle opens and enumeration starts, then throws mid-iteration. The resolved path is the correct, fully-qualified C:\home\site\wwwroot, so this is not a path-computation bug.

Relevant code

src/FunctionLoader.cs:

// Resolve the FunctionApp profile path
var options = new EnumerationOptions { MatchCasing = MatchCasing.CaseInsensitive };
var profiles = Directory.EnumerateFiles(FunctionAppRootPath, "profile.ps1", options);
FunctionAppProfilePath = profiles.FirstOrDefault();

Directory.EnumerateFiles is lazy, so the directory is walked when FirstOrDefault() pulls the first entry — which is exactly where the stack trace lands.

Evidence pointing at a worker-version / runtime change

Telemetry correlation (production, all stamps/regions) of the SetupWellKnownPaths failure against each app's own worker-launch line (...\workers\powershell\<ver>\...):

• 100% of failing apps run the PowerShell 7.6 worker, which runs on .NET 10.
• No 7.2 / 7.4 app (which run on .NET 8) appears in the failing set in any time window examined.
• Affected apps span multiple regions (ln1, am2, ty1, db3, zrh) and multiple stamps, i.e. it is not a single degraded stamp.
• On the same instance during the same startup, the Functions host reads C:\home\site\wwwroot\host.json and indexes functions successfully seconds before the worker fails to enumerate the same directory — so the content share is mounted and readable at that moment.
• The worker's own init (Worker init request completed) succeeds; only the later ProcessFunctionLoadRequest enumeration fails.

Root cause

This is a regression in .NET 10.0.9 and is fixed in .NET 10.0.11.

What changed. Historically, on Windows, Directory.EnumerateFiles(dir, "profile.ps1") passed null as the FileName filter to NtQueryDirectoryFile and did the pattern matching in managed code. As a performance optimization, .NET 10 began passing the search pattern to NtQueryDirectoryFile as an OS-level filter on the first call per directory:

• Optimization: dotnet/runtime #122947, backported to release/10.0 in #127974 — shipped in 10.0.9.

Why it throws. When the OS-level filter matches zero files, most filesystem drivers report STATUS_NO_SUCH_FILE (0xC000000F), which the enumerator treats as "no matches" and returns an empty sequence. But some drivers — notably the ones backing certain Azure Files / mounted content shares — report the zero-match condition as STATUS_OBJECT_NAME_NOT_FOUND (0xC0000034) instead. In 10.0.9/10.0.10 the enumerator did not handle that status, so it fell through and threw. RtlNtStatusToDosError(0xC0000034) maps to ERROR_FILE_NOT_FOUND (2), which surfaces as FileNotFoundException: "Could not find file '<dir>'." — exactly the exception and message seen here, naming the directory itself.

• Fix: dotnet/runtime #130196 (adds handling for STATUS_OBJECT_NAME_NOT_FOUND), backported to release/10.0 in #130327, milestone 10.0.11. Original .NET report: dotnet/runtime#130134.

How this explains the symptoms

• 7.6-only. Only the 7.6 worker runs on .NET 10; 7.2/7.4 run on .NET 8, which never passes an OS-level filter and is therefore unaffected. The failing population being 100% 7.6 is a direct consequence.
• Throws mid-iteration at FindNextEntry()/MoveNext(). The directory handle opens fine; the throw comes from the first NtQueryDirectoryFile filtered query returning 0xC0000034, which is pulled lazily by FirstOrDefault().
• FileNotFoundException naming the directory, not DirectoryNotFoundException. This corresponds to ERROR_FILE_NOT_FOUND (2), which is the mapping for STATUS_OBJECT_NAME_NOT_FOUND — i.e. a zero-match filtered enumeration, not a missing/unmounted directory.
• Host can read the share moments earlier. Consistent with the above: the mount is visible and readable; the failure is purely how a zero-match filtered enumeration is surfaced by the driver on .NET 10.9/10.10.
• Intermittent across stamps. Whether the throw occurs depends on the specific content-share driver returning 0xC0000034 vs 0xC000000F for the zero-match case, which is why it is not tied to a single degraded stamp and why only apps whose wwwroot has no profile.ps1 (a zero-match) are exposed.

Path forward

• The .NET runtime available to the PowerShell worker is sourced from the Functions platform, not bundled by this worker. Because the worker is framework-dependent, it runs on whichever .NET 10 patch the platform provides. The fix therefore lands when the platform's .NET 10 runtime is updated to 10.0.11 or later; we are working with the platform team to pick that up.
• In the meantime, Make SetupWellKnownPaths resilient so a missing profile.ps1 cannot terminate the PowerShell 7.6 (.NET 10) worker #1146 tracks worker-side hardening of SetupWellKnownPaths (existence check + not caching a transient/enumeration error as a terminating failure) so the worker is resilient regardless of the underlying runtime behavior.
• Customer/immediate mitigation: move the app to PowerShell 7.4 (which runs on .NET 8 and is unaffected) until the platform's .NET 10 runtime is updated. Note 7.6 is still a preview runtime.

Related

• Make SetupWellKnownPaths resilient so a missing profile.ps1 cannot terminate the PowerShell 7.6 (.NET 10) worker #1146 — defensive hardening of SetupWellKnownPaths.

Environment

• Worker: PowerShell 7.6 (.NET 10)
• Unaffected: PowerShell 7.4 (.NET 8), 7.2
• Regressed .NET runtime: 10.0.9 and 10.0.10; fixed in 10.0.11