Abort on system stack overflow during GC

Buggy native extensions could have mark functions that cause stack
overflow. When a stack overflow happens during GC, Ruby used to recover
by raising an exception, which runs the interpreter. It's not safe to
run the interpreter during GC since the GC is in an inconsistent state.
This could cause object allocation during GC, for example.

Instead of running the interpreter and potentially causing a crash down
the line, fail fast and abort.

1 files changed, 4 insertions, 1 deletions

diff --git a/vm_insnhelper.c b/vm_insnhelper.c
index 275e5f7394..9eedc10172 100644
--- a/vm_insnhelper.c
+++ b/vm_insnhelper.c
@@ -83,7 +83,10 @@ NORETURN(MJIT_STATIC void rb_ec_stack_overflow(rb_execution_context_t *ec, int c
 MJIT_STATIC void
 rb_ec_stack_overflow(rb_execution_context_t *ec, int crit)
 {
-    if (crit || rb_during_gc()) {
+    if (rb_during_gc()) {
+        rb_bug("system stack overflow during GC. Faulty native extension?");
+    }
+    if (crit) {
 	ec->raised_flag = RAISED_STACKOVERFLOW;
 	ec->errinfo = rb_ec_vm_ptr(ec)->special_exceptions[ruby_error_stackfatal];
 	EC_JUMP_TAG(ec, TAG_RAISE);