Sort tests based on how many mutants they killed

[Otto-AA]

Just a hypothesis: If there are tests that detect many mutants, we should run them first.

To understand if a mutation gets killed, we execute all tests until the first one fails. We currently optimize this by running fast tests first and slow tests later on.

We could try to detect which tests are effective mutation killers and run those first.

Statistically:

• A test that has a 1% chance of killing a mutant and takes 1s to execute, kills on average 0.01/1=0.01 mutants per second
• A test that has a 20% chance of killing a mutant kills and takes 5s to execute, kills on average 0.2/5 = 0.04 mutants per second

So the second test would be more efficient at killing mutants.

The problem is, that we know how long a test approximately executes in advance based on the initial test runs, but we do not know how many mutants it kills in advance. We could:

• start with the order based on execution time. Gradually include the statistics of how many they killed
• reuse the killing statistics from the last mutmut run execution

In both scenarios, tests which are executed early on, will get many more mutation kills, so the statistic would be biased. Not sure if it's worth it then.