Smart-Test

Git Test Selection

Use code-coverage to determine, which line of code is executed by which tests. Take the git diff of changed lines, take the set-union of the sets of tests per line and run them. 

Flaky-Detection and Handling

If a test fails, rerun it n times. If it passes k times, mark it as green but increase its flakiness-counter by some sensible amount. Report the statistics.

Fail Fast

Remember which tests failed if we touched a file. This creates a database. If we run the tests, we order the tests in a way, that the most "dangerous" run first. This should move the moment of failure nearer to typing of the line.

React quickly

Rerun the tests as soon as the file-system changed. If production code changed, calculate the line-diffs and execute the tests first, that use that line (like in Git Test Selection). Report now, that these passed and test some random n tests in the background. The goal is to finish faster than the eyes can focus the test-output window. Prepare the next test run as soon as we finished and not when the file was saved.

When a test-file was changed, simply execute that with the same principles in min.