TCP Server实现
一
TCP server用于管理Acceptor类接受的TCP连接
TCP Server接受连接时的时序图
二
muduo处理TcpConnection类的关闭采用的是被动关闭,即对方关闭连接,本地read(2)系统调用返回0,触发关闭逻辑,连接关闭的时序图如下
当TcpConnection从TcpServer中erase()后,如果用户不持有该TcpConnection对象的shared_ptr指针,则其引用计数为1(因为Poller类持有的Channel还有该TcpConnection对象的week_ptr指针,在调用Channel::handleEvent()方法时,会将week_ptr提升为shared_ptr,故引用计数为1)
以下与TcpConnection相关
三
读取数据时,将数据存放在Buffer类中,缓冲区初始化大小为8+1024字节;通过Buffer::readfd()方法将socketfd中的数据读取到缓冲区,如果初始缓冲区大小不够,该方法会先将数据读取的栈上的char extrabuf[65536]数组中,该数组保证了千兆网(1Gbps)满载时500us的数据输入量(230/8/2000≈2162^{30}/8/2000 ≈ 2^{16}230/8/2000≈216),因此通常只需调用一次readv(2)。
ssize_t Buffer::readFd(int fd, int* savedErrno)
{
// saved an ioctl()/FIONREAD call to tell how much to read
char extrabuf[65536];
struct iovec vec[2];
const size_t writable = writableBytes();
vec[0].iov_base = begin()+writerIndex_;
vec[0].iov_len = writable;
vec[1].iov_base = extrabuf;
vec[1].iov_len = sizeof extrabuf;
// when there is enough space in this buffer, don't read into extrabuf.
// when extrabuf is used, we read 128k-1 bytes at most.
const int iovcnt = (writable < sizeof extrabuf) ? 2 : 1;
const ssize_t n = sockets::readv(fd, vec, iovcnt);
if (n < 0)
{
*savedErrno = errno;
}
else if (implicit_cast<size_t>(n) <= writable)
{
writerIndex_ += n;
}
else
{
writerIndex_ = buffer_.size();
append(extrabuf, n - writable);
}
// if (n == writable + sizeof extrabuf)
// {
// read again;
// }
return n;
}
四
调用shutdown()只关闭了本地的写端,即将本地TcpConnection写功能关闭(半关闭,仍支持读),远程的TcpConnection的写关闭则根据第一节的方法触发。
void TcpConnection::shutdown()
{
// FIXME: use compare and swap
if (state_ == kConnected)
{
setState(kDisconnecting);
// FIXME: shared_from_this()?
loop_->runInLoop(std::bind(&TcpConnection::shutdownInLoop, this));
}
}
void TcpConnection::shutdownInLoop()
{
loop_->assertInLoopThread();
if (!channel_->isWriting()) //发送缓冲区中还有数据,则不关闭写端
{
// we are not writing
socket_->shutdownWrite();
}
}
五
多个重载的send方法最终调用了sendInLoop()方法。
发送数据时,先尝试在eventloop中直接发送数据,如果未发送完,则将剩余数据添加到outputBuffer_中,并且通过修改可写状态Channel::enableWriting()向poll中注册POLLOUT事件。(注意触发POLLOUT事件的情况)
writeComleteCallback是用于提供低水位算法实现。
highWaterMatterCallback则是用于提供高水位算法实现。
在发送端与接收端速率不匹配时使用。
void TcpConnection::send(const StringPiece& message)
{
if (state_ == kConnected)
{
if (loop_->isInLoopThread())
{
sendInLoop(message);
}
else
{
void (TcpConnection::*fp)(const StringPiece& message) = &TcpConnection::sendInLoop;
loop_->runInLoop(
std::bind(fp,
this, // FIXME
message.as_string()));
//std::forward<string>(message)));
}
}
}
void TcpConnection::sendInLoop(const StringPiece& message)
{
sendInLoop(message.data(), message.size());
}
void TcpConnection::sendInLoop(const void* data, size_t len)
{
loop_->assertInLoopThread();
ssize_t nwrote = 0;
size_t remaining = len;
bool faultError = false;
if (state_ == kDisconnected)
{
LOG_WARN << "disconnected, give up writing";
return;
}
// if no thing in output queue, try writing directly
if (!channel_->isWriting() && outputBuffer_.readableBytes() == 0)
{
nwrote = sockets::write(channel_->fd(), data, len);
if (nwrote >= 0)
{
remaining = len - nwrote;
if (remaining == 0 && writeCompleteCallback_)
{
loop_->queueInLoop(std::bind(writeCompleteCallback_, shared_from_this()));
}
}
else // nwrote < 0
{
nwrote = 0;
if (errno != EWOULDBLOCK)
{
LOG_SYSERR << "TcpConnection::sendInLoop";
if (errno == EPIPE || errno == ECONNRESET) // FIXME: any others?
{
faultError = true;
}
}
}
}
assert(remaining <= len);
if (!faultError && remaining > 0)
{
size_t oldLen = outputBuffer_.readableBytes();
if (oldLen + remaining >= highWaterMark_
&& oldLen < highWaterMark_
&& highWaterMarkCallback_)
{
loop_->queueInLoop(std::bind(highWaterMarkCallback_, shared_from_this(), oldLen + remaining));
}
outputBuffer_.append(static_cast<const char*>(data)+nwrote, remaining);
if (!channel_->isWriting())
{
channel_->enableWriting();
}
}
}
当触发了POLLOUT事件后,根据TcpConnection构造时为Channel注册的WriteCallback函数TcpConnection::handleWrite处理outputBuffer_中的数据。
//构造TcpConnection
TcpConnection::TcpConnection(EventLoop* loop,
const string& nameArg,
int sockfd,
const InetAddress& localAddr,
const InetAddress& peerAddr)
: loop_(CHECK_NOTNULL(loop)),
name_(nameArg),
state_(kConnecting),
reading_(true),
socket_(new Socket(sockfd)),
channel_(new Channel(loop, sockfd)),
localAddr_(localAddr),
peerAddr_(peerAddr),
highWaterMark_(64*1024*1024)
{
channel_->setReadCallback(
std::bind(&TcpConnection::handleRead, this, _1));
channel_->setWriteCallback(
std::bind(&TcpConnection::handleWrite, this));
channel_->setCloseCallback(
std::bind(&TcpConnection::handleClose, this));
channel_->setErrorCallback(
std::bind(&TcpConnection::handleError, this));
LOG_DEBUG << "TcpConnection::ctor[" << name_ << "] at " << this
<< " fd=" << sockfd;
socket_->setKeepAlive(true);
}
//向Channel注册的写回调函数
void TcpConnection::handleWrite()
{
loop_->assertInLoopThread();
if (channel_->isWriting())
{
ssize_t n = sockets::write(channel_->fd(),
outputBuffer_.peek(),
outputBuffer_.readableBytes());
if (n > 0)
{
outputBuffer_.retrieve(n);
if (outputBuffer_.readableBytes() == 0)
{
channel_->disableWriting();
if (writeCompleteCallback_)
{
loop_->queueInLoop(std::bind(writeCompleteCallback_, shared_from_this()));
}
if (state_ == kDisconnecting)
{
shutdownInLoop();
}
}
}
else
{
LOG_SYSERR << "TcpConnection::handleWrite";
// if (state_ == kDisconnecting)
// {
// shutdownInLoop();
// }
}
}
else
{
LOG_TRACE << "Connection fd = " << channel_->fd()
<< " is down, no more writing";
}
}
六
对TcpConnection进行流量控制时,可使用WriteCompleteCallback和HighWaterMarkCallback,这两个回调函数的作用位置分别为WriteCompleteCallback:TcpConnection::handleWrite()和TcpConnection::sendInLoop()函数中;HighWaterMarkCallback:TcpConnection::sendInLoop()函数中。
七
实现多线程TcpServer的方式是用1+N个 EventLoop实现的。
1即使用TcpServer中的eventloop接受连接;
N(N>=0)则为通过EventLoopThreadPool线程池,采用轮询方法为每个新的连接分配eventloop。