AssemblyCleanup runs before all ClassCleanups finish

[obligaron]

Describe the bug

AssemblyCleanup may begin executing while some ClassCleanup methods are still running.
This only happens when tests are run in parallel (ExecutionScope.MethodLevel).

Steps To Reproduce

Running the following program:

[assembly: Parallelize(Scope = ExecutionScope.MethodLevel)]

namespace TestProjectParallel;

[TestClass]
public static class GlobalSettings
{
    public static int MissingClassCleanups;

    [AssemblyCleanup]
    public static async Task AssemblyCleanup()
    {
        int foundMissingClassCleanups = MissingClassCleanups;
        await Task.Delay(1000);
        if (foundMissingClassCleanups > 0)
        {
            int foundMissingClassCleanupsAfter1Second = MissingClassCleanups;
            throw new InvalidOperationException($"AssemblyCleanup started while ClassCleanup is not finished (MissingClassCleanups: {foundMissingClassCleanups} after 1 sec: {foundMissingClassCleanupsAfter1Second})");
        }
    }
}

[TestClass]
public sealed class Test1
{
    [ClassCleanup]
    public static async Task ClassCleanup()
    {
        await Task.Delay(Random.Shared.Next(1, 100));
        Interlocked.Decrement(ref GlobalSettings.MissingClassCleanups);
    }

    [TestMethod]
    public void TestMethod1()
    {
        Interlocked.Increment(ref GlobalSettings.MissingClassCleanups);
    }
}

[TestClass]
public sealed class Test2
{
    [ClassCleanup]
    public static async Task ClassCleanup()
    {
        await Task.Delay(Random.Shared.Next(1, 100));
        Interlocked.Decrement(ref GlobalSettings.MissingClassCleanups);
    }

    [TestMethod]
    public void TestMethod1()
    {
        Interlocked.Increment(ref GlobalSettings.MissingClassCleanups);
    }
}

Given the following csproj:

<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <TargetFramework>net10.0</TargetFramework>
    <LangVersion>latest</LangVersion>
    <ImplicitUsings>enable</ImplicitUsings>
    <Nullable>enable</Nullable>
    <EnableMSTestRunner>true</EnableMSTestRunner>
    <OutputType>Exe</OutputType>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="MSTest" Version="4.0.2" />
  </ItemGroup>

  <ItemGroup>
    <Using Include="Microsoft.VisualStudio.TestTools.UnitTesting" />
  </ItemGroup>

</Project>

Often (try multiple runs) it executes GlobalSettings.AssemblyCleanup() while Test1.ClassCleanup() or Test2.ClassCleanup() is still running.
System.InvalidOperationException: AssemblyCleanup started while ClassCleanup is not finished (MissingClassCleanups: 1 after 1 sec: 0).

Expected behavior

AssemblyCleanup runs after all ClassCleanup are finished, e.g.

Test2.TestMethod()
Test1.TestMethod()
Test2.ClassCleanup() - start
Test2.ClassCleanup() - finish
Test1.ClassCleanup() - start
Test1.ClassCleanup() - finish
GlobalSettings.AssemblyCleanup() - start
GlobalSettings.AssemblyCleanup() - finish

Note: The order of the TestMethod/ClassCleanup between Test1 and Test2 is not relevant.

Actual behavior

Test2.TestMethod()
Test1.TestMethod()
Test2.ClassCleanup() - start
Test2.ClassCleanup() - finish
Test1.ClassCleanup() - start
GlobalSettings.AssemblyCleanup() - start
Test1.ClassCleanup() - finish
GlobalSettings.AssemblyCleanup() - finish

Additional context

This behavior affects users of Reqnroll: reqnroll/Reqnroll#951.

[nohwnd]

Thanks for reporting. That looks serious, and I don't see any obvious mistake in your code. We will have a look as soon as possible.

[engyebrahim]

The issue is that when using
[assembly: Parallelize(Scope = ExecutionScope.MethodLevel)],
AssemblyCleanup is being called after each test class, instead of once after all tests.

This behavior started occurring in version 3.11.1