Add support for kqueue() and epoll() to event loop #203
Description
From g.rodola on January 25, 2012 18:35:26
Right now the internal poller depends on asyncore module; as such it can only
use select() and poll() system calls which don't scale/perform well with
thousands of concurrent clients.
This is a benchmark using poll():
pyftpdlib 0.7.0:
2000 concurrent clients (connect, login) 36.63 secs
2000 concurrent clients (RETR 10M file) 128.07 secs
2000 concurrent clients (STOR 10M file) 189.73 secs
2000 concurrent clients (quit) 0.39 secs
proftpd 1.3.4rc2:
2000 concurrent clients (connect, login) 44.59 secs
2000 concurrent clients (RETR 10M file) 33.90 secs
2000 concurrent clients (STOR 10M file) 138.94 secs
2000 concurrent clients (quit) 2.28 secs
2000 clients here actually means 4000 concurrent connections (control + data).
As noticeable, poll() clearly suffers a serious performance degradation.
select() on the other hand, wouldn't have been able to work at all as it has a
limit of 1024 fds.
epoll() (Linux) and kqueue() (BSD / OSX) are supposed to fix this problems
altogheter.
What I have in mind (for 1.0.0 version) is to add a "lib" package within a
modified version of asyncore.dispatcher and an asyncore.loop supporting
kqueue()/epoll().
A partial patch I wrote some time ago is here: http://bugs.python.org/issue6692
Also, tornado ( http://www.tornadoweb.org/ ) can be used as an example for the
epoll() implementation.
Original issue: http://code.google.com/p/pyftpdlib/issues/detail?id=203