From: Eric Wong Date: 2017-07-13T22:31:08+00:00 Subject: [ruby-core:82040] Re: [Ruby trunk Feature#13618] [PATCH] auto fiber schedule for rb_wait_for_single_fd and rb_waitpid ko1@atdot.net wrote: > sorry for long absent about this topic. it is hard task (hard > to start writing up because of problem difficulties and my > English skil ;p ) to summarize about this topic. No problem, thank you for summarizing. > I try to write step by step. > > ---- > > # Discussion at last developers meeting > > ## Thread/Fiber switch safety > > Koichi: (repeat my opinion about difficulty of thread/fiber safety) > > akr: providing better synchronize mechanism (such as go-lang > has) and encouraging safe parallel computation seems better. > > Koichi: It is one possible solution but my position is "if > people can shoot their foot, people will shoot". I think your approach is too cautious. We already have many dangerous things in Ruby, even in single-threaded code. For example: File.read, IO#read, IO#gets are all dangerous with no size limit: they can cause out-of-memory or swapping on gigantic inputs, leading to DoS. Fork and inadvertant sharing of open files/sockets can also cause problems. And there are also pathological Regexp which can cause unbound CPU usage. > Matz: I don't like to force people to use lock and so on. > > (the point is Matz doesn't reject "-safe" approach) > > ## Introduce restriction > > (The following idea is not available at last meeting (only > part of idea I showed)) > > Koichi: > > The problem of this feature is mind gap using auto-fiber user > and script writer. This is same as thread-safety. Person A > consider the code is auto-fiber safe, and other person B (can > be same as A) write a code without auto-fiber safety, then it > will be problem. > > In general, most of existing libraries are not auto-fiber safe > code (it doesn't mean most of libraries are not auto-fiber > safe. Many code are auto-fiber safe without any care). Right; most code does not have to care; and all these dangers already exist with native Threads. > If we can know a code (and code called by this code) is > auto-fiber safe, we can use auto-fiber in safe. > > There are three type of code. > > * (1) don't care about auto-fiber > * (2) auto-fiber aware code (assume switching is not allowed at the beginning) > * (3) auto-fiber aware code (don't care it is allowed or not allowed to switch) > > There are three types of status. > > * (a) can't switch > * (b) can enable to switch, but don't switch > * (c) can switch > > in matrix > > ``` > can switch / can enable switch > (a) can't / can't > (b) can't / can > (c) can / ?? > ``` > > matrix with (1-3) and (a-c) > > ``` > (a) (b) (c) > (1) OK NG NG > (2) OK OK NG > (3) OK(*1) OK(*1) OK > ``` > > (1)-(b) and (1)-(c) is not accepted because other method called from this code can switch the context. > (2)-(c) is also unacceptable because the beginning of code is not auto-fiber aware. > > *1) Possible problem: (3) can introduce dead-lock problem because it can stop forever. Perhaps holding Mutex lock should disable auto-fiber switching. This should prevent deadlocks, I think. Existing code has Mutexes, so I'm not sure how they should interact with auto-Fiber. I agree with Matz that we should discourage locking, so I guess disabling auto-Fiber switch while Mutex is held is the most straightforward solution. > Normal threads start from (a). Auto-fibers start from (b). > They are written in (1), (2) and (3). Maybe (2) is written for > auto fiber top-lelvel. This code will call some async methods > which can change context. > > My proposal is, to write down explicitly of (1) to (3) and (a) > to (c) in program. > > At the meeting, I proposed non-matured keywords(-like) to control them. > (and just now I don't have good syntax for it yet) > > akira: If we introduce such keywords, we need to rewrite all > of code if we want to use auto-fiber web application request > handler (for example, we need to rewrite Rails to run on > auto-fiber based rack server). > > Matz: it is unacceptable to introduce huge rewriting for existing code. I agree completely with akira's observation and Matz's opinion of this. > (IMO (not appeared in last meeting) we need to rewrite all of > code even if we don't introduce keywords to make sure the > auto-fiber safety) I don't agree with this. A lot of code is already auto-fiber safe because they are written with GVL+Threads in mind. (see my original Net::HTTP example); and we also have a lot of code (webrick, net/*) which worked fine with green Threads in 1.8 Worst case is we release GVL in a native Thread and forget to yield to other Fibers in the same Thread. However, that is already a problem with existing code when run inside Fibers (e.g. getaddrinfo, IO operations on NFS/slow-disk, ...) I am working on making rb_thread_fd_select auto-fiber aware, too. (done for iom_select/iom_epoll, working on iom_kqueue) > Matz and I discussed about this issue, and we conclude that it > is too early to introduce this feature on Ruby 2.5. OK, I will continue to work on implementation improvements and keep patches rebased to trunk. > I want to consider this issue further. auto-fiber based guild > is one possibility, this mean we can introduce object > isolation and context switching each other. Do you think this is in the 2.5 timeline? Thank you. Unsubscribe: