From: Eric Wong Date: 2018-02-03T09:33:56+00:00 Subject: [ruby-core:85362] Re: [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid sam.saffron@gmail.com wrote: > Issue #13618 has been updated by sam.saffron (Sam Saffron). > > > I'm leaning towards Thread::Green, so existing users can do > > s/Thread.new/Thread::Green.new/ in many cases. > > Yes I think this works the problem though is that people will > expect this to work like green threads, meaning they also > should auto-yield regularly. You should be allowed to have > two green threads doing expensive computations. One tight loop > in a reactor now and you blow up everything (unlike normal threads) Good point. rb_thread_call_without_gvl could be used to migrate work to a thread pool (so we end up with M:N threads), and maybe that's not horrible as a default behavior. Data migration across native threads would hurt locality-wise for short-lived tasks (e.g. rb_stat), though... Fwiw, I was planning on adding a hinting mechanism to rb_thread_call_without_gvl anyways later on (for GC, maybe); but hints could be added to prevent/encourage migration based on the expected duration/bottleneck of the function. > This would mean you would have to pull in the 1.8 scheduler > or something. But then this stops being a proper reactor :(. > > I guess this is the underlying reason you just wanted to call this auto > yielding fibers instead of threads to start with. Right, the predictability of not having a timer switch threads automatically is appealing, sometimes. Having rb_thread_call_without_gvl become a scheduling point of some sort for green threads would be fine, however, since all callers already assume a context switch will happen. > A question for Matz and Koichi is if they expect the scheduler from 1.8 > to be brought back, if this is "safe" by default and "opt-in" for unsafe. > > > > I expect PG to be able to benefit from rb_wait_for_single_fd when > > using sockets. I know mysql2 uses rb_wait_for_single_fd, at least. > > I am not sure about this, libpq abstracts all of this stuff away from you > this is why Sean G wrote a complete binary protocol implementation in rust, > to gain control. pg gem does not use rb_wait_for_single_fd it just releases gvl. > > We have to make sure there is some sort of path forward with Postgres here > it is a huge issue. I don't know how expensive it is to parse the Pg protocol; but I remember in the 1.8 days pg was one of the few gems to use rb_thread_select and it played nicely with 1.8 green threads. Can't say I know Pg well these days, it's been over a decade since I used it with Ruby. > MiniRacer is CPU bound its basically packaging libv8 into > Ruby. I am on the fence here On one hand it would be nice to > auto yield so we feel reduced GVL pain and Ruby code can run > while v8 does it's thing. On the other hand the semantics of > one thread at 100% suddenly becomes 2 threads at 100% is not > ideal. Hard to decide. How long does it release the GVL for? The thread pool / workqueue idea I mentioned above might be a good fit for this if the communications overhead can masked by the length of the task. Nothing wrong with 2 threads at 100% if they're getting work done faster than 1 thread at 100%. I wouldn't want a native pool to be used for something like getaddrinfo, however, that's hugely inefficient (but exactly what getaddrinfo_a does internally in glibc). Unsubscribe: