CentOS7 实现 Keepalived + Nginx 实现高可用 Web 负载均衡

一、基础环境
系统版本 nginx版本 keepalived版本 ip 作用
CentOS Linux release 7.6 nginx/1.18.0 keepalived-2.1.5 192.168.86.135 master
CentOS Linux release 7.6 nginx/1.18.0 keepalived-2.1.5 192.168.86.136 slave
VIP:192.168.86.137

二、编译环境安装
yum -y install make zlib zlib-devel gcc-c++ libtool openssl openssl-devel pcre pcre-devel

挂镜像到系统

mv /etc/yum.repos.d/CentOS-Base.repo /etc/yum.repos.d/CentOS-Base.repo.bak

mv /etc/yum.repos.d/CentOS-Debuginfo.repo /etc/yum.repos.d/CentOS-Debuginfo.repo.bak

mv /etc/yum.repos.d/CentOS-Vault.repo /etc/yum.repos.d/CentOS-Vault.repo.bak

cp /etc/yum.repos.d/CentOS-Media.repo /etc/yum.repos.d/CentOS-Media.repo.bak

sed -i ‘s#file:///media/cdrom/##g’ /etc/yum.repos.d/CentOS-Media.repo

sed -i ‘s#file:///media/cdrecorder/##g’ /etc/yum.repos.d/CentOS-Media.repo

sed -i ‘s/^ *enabled =.$/enabled=1/g’ /etc/yum.repos.d/CentOS-Media.repo

sed -i ‘s#^ *baseurl =.$#baseurl=file:///mnt/centos/#g’ /etc/yum.repos.d/CentOS-Media.repo

mount -t iso9660 -o loop /tools/CentOS-7-x86_64-DVD-1908.iso /mnt/centos

三、安裝nginx
安裝nignx

#进入目录
cd /tools #上传安装文件，并解压
tar -zxvf nginx-1.18.0.tar.gz
#进入安装目录
cd nginx-1.18.0
#检查配置
./configure --prefix=/usr/local/nginx --with-stream

make&&make install

修改nginx配置文件

修改 Nginx 欢迎首页内容（用于后面测试， 用于区分两个节点的 Nginx）：

master

echo ‘this is master 135’ > /usr/local/nginx/html/index.html
slave

echo ‘this is slave 136’ > /usr/local/nginx/html/index.html
修改nginx.conf

vi /usr/local/nginx/conf/nginx.conf

user root;
worker_processes 1;
#error_log logs/error.log;
#error_log logs/error.log notice;
#error_log logs/error.log info;
#pid logs/nginx.pid;
events {
worker_connections 1024;
}
http {
include mime.types;
default_type application/octet-stream;
#log_format main '$remote_addr - $remot{e}_{u}ser[$time_local] “$request” ’

'$status $bod{y}_{b}yte{s}_{s}ent"$http_referer" ’

‘“$htt{p}_{u}se{r}_{a}gent""$http_x_forwarded_for”’;

#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
server {
listen 88;
server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / {
root html;
index index.html index.htm;
}
#error_page 404 /404.html;

redirect server error pages to the static page /50x.html

error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
#user nobody;
worker_processes 2;
error_log logs/error.log;
error_log logs/error.log debug;
error_log logs/error.log info;
#pid logs/nginx.pid;
worker_rlimit_nofile 65535;
events {
use epoll;
multi_accept on;
accept_mutex on;
worker_connections 65535;
}
stream{
upstream tcp21001{
server 192.168.10.130:11001 weight=1 max_fails=2 fail_timeout=30s;#负载均衡的主机IP与端口 weight权重 代表分到资源的比例 max_fails=2 失败超过2次，剔除 fail_timeout=30s 30秒不会在连接些服务
}
server{
#listen [::1]:7777 #IPV6
listen 7777;#监听本机端口号IPV4
proxy_pass tcp11001;
proxy_connect_timeout 3s; #服务不通时 超时时间
proxy_timeout 3600s; #无数据传送，超时时间
upstream tcp21002{
server 192.178.17.130:11002 weight=1 max_fails=2 fail_timeout=30s;#负载均衡的主机IP与端口 weight权重 代表分到资源的比例 max_fails=2 失败超过2次，剔除 fail_timeout=30s 30秒不会在连接些服务
}
server{
#listen [::1]:8888 #IPV6
listen 8888;#监听本机端口号IPV4
proxy_pass tcp11002;
proxy_connect_timeout 3s; #服务不通时 超时时间
proxy_timeout 3600s; #无数据传送，超时时间
}
}
启动nginx

systemctl start nginx
测试nginx启动

curl localhost
this is master135

4.CentOS 7设置Nginx开机启动

第一步：在/lib/systemd/system目录下创建nginx.service文件
第二步：编辑nginx.service文件：
vi /lib/systemd/system/nginx.service
[Unit]
Description=nginx
After=network.target

[Service]
Type=forking
ExecStart=/usr/local/nginx/sbin/nginx
ExecReload=/usr/local/nginx/sbin/nginx reload
ExecStop=/usr/local/nginx/sbin/nginx quit
PrivateTmp=true

[Install]
WantedBy=multi-user.target
[Unit]:服务的说明
Description:描述服务
After:描述服务类别
[Service]服务运行参数的设置
Type=forking是后台运行的形式
ExecStart为服务的具体运行命令
ExecReload为重启命令
ExecStop为停止命令
PrivateTmp=True表示给服务分配独立的临时空间
注意：[Service]的启动、重启、停止命令全部要求使用绝对路径
[Install]运行级别下服务安装的相关设置，可设置为多用户，即系统运行级别为3
#赋予脚本执行权限
chmod +x /usr/lib/systemd/system/nginx.service
第三步：加入开机启动

systemctl enable nginx.service

取消开机自启动
#systemctl disable nginx.service
启动nginx服务
#systemctl start nginx.service
停止nginx服务
#systemctl stop nginx.service
重启nginx服务
#systemctl restart nginx.service
查看所有以启动的服务
#systemctl list-units --type=service
查看服务当前状态
#systemctl status nginx.service
遇到的错误
Warning: nginx.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.
按照提示执行命令systemctl daemon-reload即可。
生成系统service管理文件
vvim /usr/lib/systemd/system/nginx.service

[Unit]

Description=nginx-The High-performance HTTP Server
After=network.target

[Service]

Type=forking
PIDFile= /usr/local/nginx/logs/nginx.pid
ExecStartPre=/usr/local/nginx/sbin/nginx -t -c /usr/local/nginx/conf/nginx.conf
ExecStart=/usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
ExecReload=/usr/local/nginx/sbin/nginx -s reload
ExecStop=/usr/local/nginx/sbin/nginx -s stop
PrivateTmp=true

[Install]

WantedBy=multi-user.target

6.加防火墙

添加

firewall-cmd --zone=public --add-port=88/tcp --permanent

删除

firewall-cmd --zone=public --remove-port=80/tcp --permanent

更新防火墙规则：
firewall-cmd --reload

查看所有打开的端口：
firewall-cmd --zone=public --list-ports

四、安装keepalived
1、 创建依赖环境
yum -y install libnl libnl-devel gcc openssl-devel libnl3-devel pcre-devel net-snmp-agent-libs libnfnetlink-devel curl
wget
2、安装keepalived
tar -zxvf keepalived-2.1.5.tar.gz
cd keepalived-2.1.5
./configure --prefix=/usr/local/keepalived-2.1.5
make && make install
3、创建启动文件
ln -s /usr/local/keepalived-2.1.5 /usr/local/keepalived
mkdir /etc/keepalived/
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/
cp /usr/local/keepalived-2.1.5/etc/sysconfig/keepalived /etc/sysconfig/
ln -s /usr/local/keepalived-2.1.5/sbin/keepalived /usr/sbin/
下面文件要从keepalived源码目录复制，安装目录中没有cp
cp /tools/keepalived-2.1.5/keepalived/keepalived.service /etc/systemd/system/
cp /tools/keepalived-2.1.5/keepalived/etc/init.d/keepalived /etc/init.d/
4、创建配置文件
master

cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
#一个没重复的名字即可
router_id xxoo_master
}

检测nginx是否运行

vrrp_script chk_nginx {
script “/etc/keepalived/nginx_check.sh”
interval 2
weight -20}
vrrp_instance VI_1 {
# 此处不设置为MASTER，通过priority来竞争master
state BACKUP
# 网卡名字，文章下方会给出如何获取网卡名字的方法 interface enp0s3
# 同一个keepalived集群的virtual_router_id相同
virtual_router_id 51
# 权重，master要大于slave
priority 100
# 主备通讯时间间隔
advert_int 1
# 如果两节点的上联交换机禁用了组播，则采用vrrp单播通告的方式
# 本机ip
unicast_src_ip 192.168.0.182
unicast_peer {
# 其他机器ip 192.168.0.189
}
# 设置nopreempt防止抢占资源
nopreempt
# 主备保持一致
authentication {
auth_type PASS
auth_pass 1111
}
# 与上方nginx运行状况检测呼应
track_script {
chk_nginx
}
virtual_ipaddress {
# 虚拟ip地址（VIP，一个尚未占用的内网ip即可） 192.168.0.180
}
}
EOF
slave

cat > /etc/keepalived/keepalived.conf << EOF
! Configuration File for keepalived
global_defs {
#一个没重复的名字即可
router_id xxoo_slave
}

检测nginx是否运行

vrrp_script chk_nginx {
script “/etc/keepalived/nginx_check.sh”
interval 2
weight -20}
vrrp_instance VI_1 {
# 此处不设置为MASTER，通过priority来竞争master
state BACKUP
# 网卡名字，文章下方会给出如何获取网卡名字的方法 interface enp0s3
# 同一个keepalived集群的virtual_router_id相同
virtual_router_id 51
# 权重，master要大于slave
priority 90
# 主备通讯时间间隔
advert_int 1
# 如果两节点的上联交换机禁用了组播，则采用vrrp单播通告的方式
# 本机ip
unicast_src_ip 192.168.0.189
unicast_peer {
# 其他机器ip 192.168.0.182
}
# 设置nopreempt防止抢占资源
nopreempt
# 主备保持一致
authentication {
auth_type PASS
auth_pass 1111
}
# 与上方nginx运行状况检测呼应
track_script {
chk_nginx
}
virtual_ipaddress {
# 虚拟ip地址（VIP，一个尚未占用的内网ip即可） 192.168.0.180
}
}
EOF

5、启编写Nginx 状态检测脚本
编写 Nginx 状态检测脚本 /etc/keepalived/nginx_check.sh (已在 keepalived.conf 中配置)脚本要求：如果 nginx 停止运行，

尝试启动，如果无法启动则杀死本机的 keepalived 进程， keepalied将虚拟 ip 绑定到 BACKUP 机器上。 内容如下：

cat > /etc/keepalived/nginx_check.sh << EOF
#!/bin/bash
A=ps -C nginx --no-header | wc -l
if [ $A -eq 0 ];
then
systemctl start nginx

/usr/local/nginx/sbin/nginx #尝试重新启动nginx

sleep 2 #睡眠2秒
if [ ps -C nginx --no-header | wc -l -eq 0 ];
then
killall keepalived
fi
fi
保存后，给脚本赋执行权限：

chmod +x /etc/keepalived/nginx_check.sh

每5秒执行一次脚本
crontab -e

• • • • • /etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 5;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 10;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 15;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 20;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 25;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 30;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 35;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 40;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 45;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 50;/etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log
• • • • • sleep 55; /etc/keepalived/nginx_check.sh >>/etc/keepalived/check_nginx.log

systemctl reload crond

systemctl restart crond

6、启动keepalived
systemctl start keepalived
systemctl enable keepalived
7、日志处理
1.编辑配置文件/etc/sysconfig/keepalived，将第14行的KEEPALIVED_OPTIONS="-D"修改为KEEPALIVED_OPTIONS="-D -d -S 0"

说明：

–dump-conf -d 导出配置数据

–log-detail -D 详细日志信息

–log-facility -S 设置本地的syslog设备，编号0-7# -S 0 表示指定为local0设备

2.修改rsyslog的配置文件 vi /etc/rsyslog.conf，在结尾加入如下2行内容，将local0设备的所有日志都记录到/var/log/keepalived.log文件
echo “local0.* /var/log/keepalived.log” >> /etc/rsyslog.conf
在42行第一列尾加入";local0.none"，表示来自local0设备的所有日志信息不再记录于/var/log/messages里
3.重启rsyslog服务
systemctl restart rsyslog
4.测试keepalived日志记录结果。重启keepalived服务，查看日志信息。
tail -f /var/log/keepalived.log
systemctl start keepalived.service
配置日志轮转
/var/log/keepalived/*.log { #切分的两个文件名
daily #按天切分
rotate 7 #保留7份
create 0644 haproxy haproxy #创建新文件的权限、用户、用户组
compress #压缩旧日志
delaycompress #延迟一天压缩
missingok #忽略文件不存在的错误
dateext #旧日志加上日志后缀
sharedscripts #切分后的重启脚本只运行一次
postrotate #切分后运行脚本重载rsyslog，让rsyslog向新的日志文件中输出日志
/bin/kill -HUP $(/bin/cat /var/run/syslogd.pid 2>/dev/null) &>/dev/null
endscript
}
防止出现脑裂现象（主备同时获取了VIP地址）

复制代码

指定keepalived配置的网卡：enp0s3，固定的VRRP广播地址：224.0.0.18

firewall-cmd --direct --permanent --add-rule ipv4 filter INPUT 0 --in-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --direct --permanent --add-rule ipv4 filter OUTPUT 0 --out-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --reload

查看配置的规则

firewall-cmd --direct --get-rules ipv4 filter INPUT
firewall-cmd --direct --get-rules ipv4 filter OUTPUT

清除规则

firewall-cmd --direct --permanent --remove-rule ipv4 filter INPUT 0 --in-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT
firewall-cmd --direct --permanent --remove-rule ipv4 filter OUTPUT 0 --out-interface ens192 --destination 224.0.0.18 --protocol vrrp -j ACCEPT

五、测试
1、在两台服务器上测试
master

$ curl localhost
this is master
root@centos7[14:46:07]:~
$ curl 192.168.86.135
this is master
root@centos7[15:03:29]:~
slave

$ curl localhost
this is slave
root@centos7[15:03:59]:/etc/keepalived
$ curl 192.168.86.136
this is master
2、关闭master的keepalived模仿down机
master关闭keepalived

$ systemctl stop keepalived
在slave上面进行测试
$ curl localhost
this is slave
root@centos7[15:10:29]:/etc/keepalived
$ curl 192.168.86.136
this is slave
到此keepalived完成

4.NetworkManager-nmcli管理网络
nmcli general status #显示NetworkManager的整体状态
nmcli connection show #显示所有的连接
nmcli connection show -a #显示活动的连接
nmcli device status #显示NetworkManager识别的设备列表和它们当前的状态
nmcli device disconnect/connect eno16777736 #停止/启动网卡==ifup/down

4、关闭NetworkManager服务

#停止服务

service NetworkManager stop

#禁用服务，下次不自动启动
chkconfig NetworkManager off

问题处理：

【keepalived】关于keepalived配置中的 mcast_src_ip 和 unicast_src_ip

https://blog.csdn.net/mofiu/article/details/76644012

keepalived 配置文件参数详解

找到你要虚拟的网卡对应的域，并且开启vrrp协议即可，命令为

sudo firewall-cmd --zone=public --add-protocol=vrrp --permanent
#重载配置
sudo firewall-cmd --reload

来自 “开源世界 ” ，链接：http://ym.baisou.ltd/post/518.html，如需转载，请注明出处，否则将追究法律责任。