From: "Eregon (Benoit Daloze)" <noreply@...>
Date: 2022-04-14T15:38:12+00:00
Subject: [ruby-core:108240] [Ruby master Bug#18731] Parallel test-all sometimes does not run at all some tests

Issue #18731 has been updated by Eregon (Benoit Daloze).


Here is a quick search of test classes with the same name in different places:
```
class Test_Bignum
class TestDateParse
class TestEmojiBreaks
class TestFileUtils
class TestFuncall
class TestGraphemeBreaksFromFile
class TestMethod
class TestMkmf
class TestOptionParser
```
From `ack '^class \w+ < Test::Unit::TestCase' test/mri/tests | grep -o -P 'class \w+' | sort  > test.txt` +
```ruby
puts File.readlines("tests.txt", chomp: true).select { |t| tests.count(t) > 1 }.uniq
```

----------------------------------------
Bug #18731: Parallel test-all sometimes does not run at all some tests
https://bugs.ruby-lang.org/issues/18731#change-97259

* Author: Eregon (Benoit Daloze)
* Status: Open
* Priority: Normal
* Backport: 2.6: UNKNOWN, 2.7: UNKNOWN, 3.0: UNKNOWN, 3.1: UNKNOWN
----------------------------------------
In TruffleRuby I've noticed that some CRuby tests sometimes run or not, non-deterministically.
The TruffleRuby CI currently runs CRuby tests with `-j4`.

Today I investigated and I think I've found the reason for it.

One occurrence of this bug is for:
* https://github.com/ruby/ruby/blob/master/test/ruby/test_method.rb
* https://github.com/ruby/ruby/blob/master/test/ruby/test_inlinecache.rb

Both define a test class `TestMethod`.

The parallel runner distributes files across processes.

If the same worker process gets both files, then all tests in the second file are not run at all! (the file is still loaded).
The reason seems this line:
https://github.com/ruby/ruby/blob/8751c5c2672d1391c73d9dec590063d27bed7e4c/tool/lib/test/unit/parallel.rb#L128
(it's also possible to reproduce with just these 2 files, `-j2` and not the full test suite, and having one worker start very slowly so the same worker gets both files)

`Test::Unit::TestCase.test_suites` is an array of classes, and so here because `TestMethod` was already defined by the first file loaded, the result of `Test::Unit::TestCase.test_suites-suites` is empty, and nothing gets run.

This doesn't seem an issue when not running in parallel/using `-j`.
But that's rare/impractical because test-all is very slow without `-j`.

For this specific case it seems we should rename the class in `test_inlinecache.rb`, but I suspect there are more name conflicts and the parallel runner should be fixed to handle this.

---

&lt;rant&gt;
FWIW, this test/unit code seems pretty messy, very long lines, [duplication](https://github.com/ruby/ruby/blob/8751c5c2672d1391c73d9dec590063d27bed7e4c/tool/lib/test/unit.rb#L680-L699), hard to follow with the mix of minitest/test-unit, etc. I don't understand how this code is any better than `mspec`. It seems far more complex and hacky.
And that's probably why Ruby implementations except CRuby run MRI tests only when they have no choice, it's so annoying to work with, so many unnecessary subprocesses (very slow to run, annoying to debug), many CRuby-specific tests, so many unreliable tests, many metaprogramming-defined tests which are very brittle, lots of global state and coupling, etc.
&lt;/rant&gt;



-- 
https://bugs.ruby-lang.org/

Unsubscribe: <mailto:ruby-core-request@ruby-lang.org?subject=unsubscribe>
<http://lists.ruby-lang.org/cgi-bin/mailman/options/ruby-core>