haproxy集群
HAProxy(High Availability Proxy)是一款开源的高性能负载均衡器和反向代理工具,专注于 HTTP、TCP 和 SSL/TLS 协议的流量分发,广泛应用于高并发、高可用的网络架构中。它以稳定性强、性能卓越、配置灵活著称,是构建大型分布式系统的核心组件之一。
环境
IP |
172.25.254.100 |
172.25.254.10 |
172.25.254.11 |
172.25.254.110 |
安装nginx
172.25.254.10
[root@localhost ~]# systemctl enable --now nginx
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/
[root@localhost ~]# echo RS2 - 172.25.254.10 >/usr/share/nginx/html/index.html
[root@localhost ~]# curl 172.25.254.10
RS2 - 172.25.254.10
172.25.254.11
[root@localhost ~]# systemctl enable --now nginx
Created symlink /etc/systemd/system/multi-user.target.wants/nginx.service → /usr/lib/
[root@localhost ~]# echo RS1 - 172.25.254.11 >/usr/share/nginx/html/index.html
[root@localhost ~]# curl 172.25.254.11
RS1 - 172.25.254.11
安装haproxy
172.25.254.100
[root@localhost ~]# dnf install haproxy
[root@localhost ~]# systemctl enable --now haproxy
Created symlink /etc/systemd/system/multi-user.target.wants/haproxy.service → /usr/lib/systemd/system/haproxy.service.
[root@localhost ~]# systemctl status firewalld
○ firewalld.service - firewalld - dynamic firewall daemonLoaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; preset: enaActive: inactive (dead)Docs: man:firewalld(1)
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg64 #---------------------------------------------------------------------65 frontend webcluster66 bind *:8067 mode http68 balance roundrobin69 use_backend webserver7071 backend webserver72 server web1 172.25.254.10:8073 server web2 172.25.254.11:8074 # main frontend which proxys to the backends75 #---------------------------------------------------------------------76 frontend main
[root@localhost ~]# systemctl restart haproxy.service
[root@localhost ~]# vim ~/.vimrc
[root@localhost ~]# cat ~/.vimrc
set nu ts=4 sw=4 ai et
客户端测试
开启nbproc 2
stats socket /var/lib/haproxy/stats #指定haproxy的套接字文件nbproc 2 #指定haproxy的work进程数量,默认是1个cpu-map 1 0 #指定第一个work绑定第一个cpu核心cpu-map 2 1 #指定第二个work绑定第二个cpu核心
开启nbthread 2
指定haproxy的线程数量,默认每个进程一个线程,此参数与nbproc互斥
算法策略
172.25.254.100
静态算法
static-rr
不支持运行时利用socat进行权重的动态调整(只支持0和1,不支持其它值) 不支持端服务器慢启动 其后端主机数量没有限制,相当于LVS中的 wrr
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg
77 listen webcluster78 bind *:8079 mode http80 balance static-rr81 server web1 172.25.254.10:80 check inter 5s fall 3 weight 282 server web2 172.25.254.11:80 check inter 5s fall 3 weight 1
[root@localhost ~]# systemctl restart haproxy.service
客户端测试
first
根据服务器在列表中的位置,自上而下进行调度 其只会当第一台服务器的连接数达到上限,新请求才会分配给下一台服务 其会忽略服务器的权重设置 不支持用socat进行动态修改权重,可以设置0和1,可以设置其它值但无效
77 listen webcluster78 bind *:8079 mode http80 # balance static-rr81 # balance roundrobin82 balance first83 server web1 172.25.254.10:80 maxconn 3 check inter 5s fall 3 weight 284 server web2 172.25.254.11:80 check inter 5s fall 3 weight 1
动态算法
roundrobin
1. 基于权重的轮询动态调度算法,
2. 支持权重的运行时调整,不同于lvs中的rr轮训模式,
3. HAProxy中的roundrobin支持慢启动(新加的服务器会逐渐增加转发数),
4. 其每个后端backend中最多支持4095个real server,
5. 支持对real server权重动态调整,
6. roundrobin为默认调度算法,此算法使用广泛
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg
77 listen webcluster78 bind *:8079 mode http80 balance roundrobin81 server web1 172.25.254.10:80 check inter 5s fall 3 weight 182 server web2 172.25.254.11:80 check inter 5s fall 3 weight 1
客户端测试
leastconn
leastconn加权的最少连接的动态 支持权重的运行时调整和慢启动,即:根据当前连接最少的后端服务器而非权重进行优先调度(新客户 端连接)
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg
77 listen webcluster78 bind *:8079 mode http80 balance leastconn81 server web1 172.25.254.10:80 check inter 5s fall 3 weight 282 server web2 172.25.254.11:80 check inter 5s fall 3 weight 1
测试
其他算法
source
源地址hash,基于用户源地址hash并将请求转发到后端服务器,后续同一个源地址请求将被转发至同一 个后端web服务器。此方式当后端服务器数据量发生变化时,会导致很多用户的请求转发至新的后端服 务器,默认为静态方式,但是可以通过hash-type支持的选项更改这个算法一般是在不插入Cookie的TCP 模式下使用,也可给拒绝会话cookie的客户提供最好的会话粘性,适用于session会话保持但不支持 cookie和缓存的场景源地址有两种转发客户端请求到后端服务器的服务器选取计算方式,分别是取模法 和一致性hash
81 listen webcluster82 bind *:8083 mode http84 balance source85 hash-type consistent86 server web1 172.25.254.10:80 check inter 5s fall 3 rise 5 weight 287 server web2 172.25.254.11:80 check inter 5s fall 3 rise 5 weight 1
uri
基于对用户请求的URI的左半部分或整个uri做hash,再将hash结果对总权重进行取模后 根据最终结果将请求转发到后端指定服务器 适用于后端是缓存服务器场景 默认是静态算法,也可以通过hash-type指定map-based和consistent,来定义使用取模法还是一致性 hash
81 listen webcluster82 bind *:8083 mode http84 balance uri85 hash-type consistent86 server web1 172.25.254.10:80 check inter 5s fall 3 rise 5 weight 287 server web2 172.25.254.11:80 check inter 5s fall 3 rise 5 weight 1
172.25.254.10 172.25.254.10
[root@localhost ~]# echo RS1 - 172.25.254.10 >/usr/share/nginx/html/index1.html
[root@localhost ~]# echo RS1 - 172.25.254.10 >/usr/share/nginx/html/index2.html[root@localhost ~]# echo RS2 - 172.25.254.11 >/usr/share/nginx/html/index1.html
[root@localhost ~]# echo RS2 - 172.25.254.11 >/usr/share/nginx/html/index2.html
url_param
url_param对用户请求的url中的 params 部分中的一个参数key对应的value值作hash计算,并由服务器 总权重相除以后派发至某挑出的服务器,后端搜索同一个数据会被调度到同一个服务器,多用与电商 通常用于追踪用户,以确保来自同一个用户的请求始终发往同一个real server
81 listen webcluster82 bind *:8083 mode http84 balance url_param name,username85 hash-type consistent86 server web1 172.25.254.10:80 check inter 5s fall 3 rise 5 weight 287 server web2 172.25.254.11:80 check inter 5s fall 3 rise 5 weight 1
hdr
针对用户每个http头部(header)请求中的指定信息做hash, 此处由 name 指定的http首部将会被取出并做hash计算, 然后由服务器总权重取模以后派发至某挑出的服务器,如果无有效值,则会使用默认的轮询调度。
81 listen webcluster82 bind *:8083 mode http84 balance hdr(User-Agent)85 hash-type consistent86 server web1 172.25.254.10:80 check inter 5s fall 3 rise 5 weight 287 server web2 172.25.254.11:80 check inter 5s fall 3 rise 5 weight 1
基于cookie的会话保持
cookie value:为当前server指定cookie值,实现基于cookie的会话黏性,相对于基于 source 地址hash 调度算法对客户端的粒度更精准,但同时也加大了haproxy负载,目前此模式使用较少, 已经被session 共享服务器代替.
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg
77 listen webcluster78 bind *:8079 mode http80 # balance static-rr81 balance roundrobin82 # balance first83 # balance leastconn84 # balance source85 # balance uri86 # balance uri_param name87 # balance hdr(User-Agent)88 hash-type consistent89 cookie WEBCOOKIE insert nocache indirect90 server web1 172.25.254.10:80 cookie servera check inter 5s fall 391 server web2 172.25.254.11:80 cookie serverb check inter 5s fall 3[root@localhost ~]# systemctl restart haproxy.service
状态页
通过web界面,显示当前HAProxy的运行状态
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg
listen stats:mode httpbind 0.0.0.0:8888stats enablelog globalstats uri /statusstats auth wan:123456
开启四层IP透传
172.25.254.100
[root@localhost ~]# systemctl restart haproxy.service
#添加send-proxy
91 listen webcluster92 bind *:8093 mode tcp #设置四层tcp协议94 #balance static-rr95 balance roundrobin96 # balance first97 # balance leastconn98 # balance source99 # balance uri
100 # balance uri_param name
101 # balance hdr(User-Agent)
102 hash-type consistent
103 cookie WEBCOOKIE insert nocache indirect
104 server web1 172.25.254.10:80 send-proxy cookie servera check inter 5s fall 3
105 server web2 172.25.254.11:80 send-proxy cookie serverb check inter 5s fall 3
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg
172.25.254.10 172.25.254.11
[root@localhost ~]# vim /etc/nginx/nginx.conf
#添加 ' "$proxy_protocol_addr"'
http {log_format main '$remote_addr - $remote_user [$time_local] "$request" '' "$proxy_protocol_addr"' '$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';
access_log /var/log/nginx/access.log main;
sendfile on;tcp_nopush on;tcp_nodelay on;keepalive_timeout 65;types_hash_max_size 4096;
include /etc/nginx/mime.types;default_type application/octet-stream;
# Load modular configuration files from the /etc/nginx/conf.d directory.# See http://nginx.org/en/docs/ngx_core_module.html#include# for more information.include /etc/nginx/conf.d/*.conf;
#添加proxy_protocolserver {listen 80 proxy_protocol; listen [::]:80;server_name _;root /usr/share/nginx/html;[root@localhost ~]# systemctl restart nginx.service
未开启时
开启之后
开启七层透传
172.25.254.100
[root@localhost ~]# vim /etc/haproxy/haproxy.cfglisten webcluster
103 bind *:80
104 mode http #设置七层http协议
105 #balance static-rr
106 balance roundrobin
107 option forwardfor
108 # balance first
109 # balance leastconn
110 # balance source
111 # balance uri
112 # balance uri_param name
113 # balance hdr(User-Agent)
114 hash-type consistent
115 cookie WEBCOOKIE insert nocache indirect
116 server web1 172.25.254.10:80 cookie servera check inter 5s fall 3
117 server web2 172.25.254.11:80 cookie serverb check inter 5s fall 3
[root@localhost ~]# systemctl restart haproxy.service
172.25.254.10 172.25.254.11
[root@localhost ~]# vim /etc/nginx/nginx.conf
http {log_format main #'$remote_addr - $remote_user [$time_local] "$request" ''"$proxy_add_x_forwarded_for" - $remote_user [$time_local] "$request" '' "$proxy_protocol_addr"''$status $body_bytes_sent "$http_referer" ''"$http_user_agent" "$http_x_forwarded_for"';
[root@localhost ~]# systemctl restart nginx.service
ACL参数
访问控制列表ACL,Access Control Lists) 是一种基于包过滤的访问控制技术 它可以根据设定的条件对经过服务器传输的数据包进行过滤(条件匹配)即对接收到的报文进行匹配和过滤,基于请求报文头部中的源地址、源端口、目标地址、目标端口、请求方法、URL、文件后缀等信息内 容进行匹配并执行进一步操作,比如允许其通过或丢弃。
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg75 frontend webcluster76 bind *:8077 mode http7879 acl test hdr_dom(host) -i www.wan.org8081 use_backend webservera if test8283 default_backend webserverb8485 backend webservera86 balance roundrobin87 server web1 172.25.254.10:80 check inter 5 fall 38889 backend webserverb90 balance roundrobin91 server web2 172.25.254.11:80 check inter 5 fall 3
[root@localhost ~]# systemctl restart haproxy.service
参数2:
acl test path_sub -m sub /a
参数3:
[root@localhost ~]# vim /etc/haproxy/haproxy.cfgacl test hdr_end(host) -i .org .com .cn
[root@localhost ~]# systemctl restart haproxy.service
[root@localhost ~]# vim /etc/hosts172.25.254.100 www.wan.org www.wan.com www.wan.cn
参数4:
acl badsrc src 172.25.254.120
http-request deny if badsrc
acl acceptsrc src 172.25.254.120
http-request deny if ! acceptsrc
自定义haproxy错误界面
[root@localhost ~]# mkdir -p /etc/haproxy/errorpage
[root@localhost ~]# vim /etc/haproxy/errorpage/503.http
HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html;charset=UTF-8
<html><body><h1>什么动物生气最安静</h1>
大猩猩!!
</body></html>
[root@localhost ~]# vim /etc/haproxy/haproxy.cfgerrorfile 503 /etc/haproxy/errorpage/503.http
[root@localhost ~]# systemctl restart haproxy.service
haproxy四层负载
123 listen mysql_port
124 bind :3306
125 mode tcp
126 balance roundrobin
127 server mysql1 172.25.254.10:3306 check
128 server mysql2 172.25.254.11:3306 check
172.25.254.10 172.25.254.11
[root@localhost ~]# dnf install myriadb-server -y
[root@localhost ~]# mysql -e "grant all on *.* to wan@'%' identified by 'wan';"
[root@localhost ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server_id=10
[root@localhost ~]# systemctl start mariadb.service
[root@localhost ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.5.22-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> select @@server_id-> ;
+-------------+
| @@server_id |
+-------------+
| 10 |
+-------------+
1 row in set (0.000 sec)
172.25.254.11
[root@localhost ~]# dnf install myriadb-server -y
[root@localhost ~]# mysql -e "grant all on *.* to wan@'%' identified by 'wan';"
[root@localhost ~]# vim /etc/my.cnf.d/mariadb-server.cnf
[mysqld]
server_id=11
[root@localhost ~]# systemctl start mariadb.service
[root@localhost ~]# mysql -u root -p
Enter password:
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MariaDB connection id is 5
Server version: 10.5.22-MariaDB MariaDB Server
Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
MariaDB [(none)]> select @@server_id-> ;
+-------------+
| @@server_id |
+-------------+
| 11 |
+-------------+
1 row in set (0.000 sec)
客户端测试
制作证书
172.25.254.100
[root@localhost certs]# openssl req -newkey rsa:2048 -nodes -sha256 -keyout /etc/haproxy/certs/wan.org^Cey -x509 -days 365 -out /etc/haproxy/certs/wan.org.crt
[root@localhost ~]# cd /etc/haproxy/certs/
[root@localhost certs]# ls
wan.org.crt wan.org.key
[root@localhost certs]# cat wan.org.key wan.org.crt >wan.pem
[root@localhost certs]# cat wan.org.key wan.org.crt >wan.pem
[root@localhost certs]# ls
wan.org.crt wan.org.key wan.pem
屏蔽其他frontend和backend内容
[root@localhost ~]# vim /etc/haproxy/haproxy.cfg
frontend webcluster-80bind *:80mode httpbalance roundrobinredirect scheme https if !{ ssl_fc }use_backend webserver
frontend webcluster-443bind *:443 ssl crt /etc/haproxy/certs/wan.pemmode httpbalance roundrobinuse_backend webserver
backend webserverserver web1 172.25.254.10:80 check inter 3s fall 3 rise 2server web2 172.25.254.11:80 check inter 3s fall 3 rise 2
[root@localhost ~]# systemctl restart haproxy.service