当前位置： 首页 > news >正文

Ceph入门到精通-大流量10GB/s支持OSPF(ECMP)-LVS 集群

news 2025/6/21 1:15:48
Keepalived-LVS 能够提高集群的高可用性并增加后端检测功能、简化配置，满足常规需求。但Keepalived-LVS集群中，同一个VIP只能由一台设备进行宣告，为一主多备的架构，不能横向拓展集群的性能，为此我们引入OSPF来解决该问题。

OSPF(ECMP)

ECMP（Equal-CostMultipathRouting）等价多路径，存在多条不同链路到达同一目的地址的网络环境中，如果使用传统的路由技术，发往该目的地址的数据包只能利用其中的一条链路，其它链路处于备份状态或无效状态，并且在动态路由环境下相互的切换需要一定时间，而等值多路径路由协议可以在该网络环境下同时使用多条链路，不仅增加了传输带宽，并且可以无时延无丢包地备份失效链路的数据传输。

特点：

  1. 基于流的均衡负载
  2. 最大链路数受设备限制（最高16）
  3. 所有链路都active，故障链路自动剔除

LVS+OSPF(ECMP)

利用ECMP以上特性，可以将LVS集群进行横向拓展，利用quagga启ospf
为模拟集群环境，我们准备了六台虚拟机分别为Client、LVS-1、LVS-2、RealServer1、RealServer2、Router，VIP设为192.168.0.100

  • Router ：192.168.0.1 192.168.1.1
  • Client ：192.168.1.2
  • LVS-1 ：192.168.0.2
  • LVS-2 ：192.168.0.3
  • RealServer1 ：192.168.0.4
  • RealServer2 ：192.168.0.5

Router 配置

LVS-1、LVS-2与Router需处于同一ospf域中，通过 IP 192.168.0.1 与LVS1/LVS2/RealServer1/RealServer2 通讯，IP 192.168.1.1 与 Client通讯

[root@router ~]# echo 1 >  /proc/sys/net/ipv4/ip_forward
// 开启转发
[root@router ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500inet 192.168.0.1  netmask 255.255.255.0  broadcast 192.168.0.255inet6 fe80::20c:29ff:fe6e:d10e  prefixlen 64  scopeid 0x20<link>ether 00:0c:29:6e:d1:0e  txqueuelen 1000  (Ethernet)RX packets 63921  bytes 5978914 (5.7 MiB)RX errors 0  dropped 0  overruns 0  frame 0TX packets 24354  bytes 2334494 (2.2 MiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0ens38: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500inet 192.168.1.1  netmask 255.255.255.0  broadcast 192.168.1.255inet6 fe80::20c:29ff:fe6e:d122  prefixlen 64  scopeid 0x20<link>ether 00:0c:29:6e:d1:22  txqueuelen 1000  (Ethernet)RX packets 60501  bytes 5206254 (4.9 MiB)RX errors 0  dropped 0  overruns 0  frame 0TX packets 18358  bytes 1432690 (1.3 MiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536inet 127.0.0.1  netmask 255.0.0.0inet6 ::1  prefixlen 128  scopeid 0x10<host>loop  txqueuelen 1000  (Local Loopback)RX packets 1119  bytes 88568 (86.4 KiB)RX errors 0  dropped 0  overruns 0  frame 0TX packets 1119  bytes 88568 (86.4 KiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
// ens33：192.168.0.1 与LVS1/LVS2/RealServer1/RealServer同一子网 ， ens38：192.168.1.1 与Client同一子网
[root@LVS1 ~]# yum install quagga.x86_64 -y
[root@router ~]# vi /etc/quagga/zebra.conf
hostname Router
password test
enable password test
log file /var/log/quagga/zebra.log
service password-encryption
interface ens33
interface ens38
access-list 1 permit 127.0.0.1
ip prefix-list ANY seq 5 permit 0.0.0.0/0 le 32
route-map ANY deny 10match ip address prefix-list ANY
ip protocol ospf route-map ANY
line vtyaccess-class 1
[root@router ~]# vi /etc/quagga/ospfd.conf
hostname Router
password test
log file /var/log/quagga/ospfd.log
log stdout
log syslog
service password-encryption
interface ens33ip ospf hello-interval 1ip ospf dead-interval 4ip ospf priority 1ip ospf cost 1
router ospfospf router-id 192.168.0.7log-adjacency-changesnetwork 192.168.0.0/24 area 0.0.0.0network 192.168.1.0/24 area 0.0.0.0
access-list 1 permit 127.0.0.1
line vtyaccess-class 1

RealServer 配置

realserver.sh 参考上一章节

[root@RealServer1 ~]# ./realserver.sh start
[root@RealServer1 ~]# ifconfig
ens33: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500inet 192.168.0.4  netmask 255.255.255.0  broadcast 192.168.0.255inet6 fe80::20c:29ff:febd:38da  prefixlen 64  scopeid 0x20<link>ether 00:0c:29:bd:38:da  txqueuelen 1000  (Ethernet)RX packets 48635  bytes 4087456 (3.8 MiB)RX errors 0  dropped 0  overruns 0  frame 0TX packets 46101  bytes 5700308 (5.4 MiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536inet 127.0.0.1  netmask 255.0.0.0inet6 ::1  prefixlen 128  scopeid 0x10<host>loop  txqueuelen 1000  (Local Loopback)RX packets 30  bytes 2613 (2.5 KiB)RX errors 0  dropped 0  overruns 0  frame 0TX packets 30  bytes 2613 (2.5 KiB)TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0lo:0: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536inet 192.168.0.100  netmask 255.255.255.255loop  txqueuelen 1000  (Local Loopback)

LVS 配置

由于我们使用ospf来实现高可用，不开启keepalived的vrrp功能（LVS2也同样配置），只是用其后端检测功能。
在keepalived-LVS集群的抢占模式下，Master在网卡上挂VIP并进行ARP广播，此时VIP对应的设备是唯一的。
但在OSPF-LVS集群中，Router根据ospf信息通过修改报文的目的mac地址转发到对应的LVS来实现均衡负载，并不根据VIP对应的ARP信息，所以对应的每台LVS将VIP挂在在lo上。

[root@LVS1 ~]# vi /etc/keepalived/keepalived.conf
global_defs {router_id LVS1 #路由器标识script_user rootenable_script_security
}virtual_server 192.168.0.100 80 {delay_loop 5lb_algo wrrlb_kind DRpersistence_timeout 60persistence_granularity 255.255.255.255protocol tcpinhibit_on_failure onha_suspendsorry_server 127.0.0.1 80real_server 192.168.0.4 80 {weight 10HTTP_GET{url{path /status_code 200}connect_port 80connect_timeout 2retry 1delay_before_retry 1}}real_server 192.168.0.5 80 {weight 10HTTP_GET{url{path /status_code 200}connect_port 80connect_timeout 2retry 1delay_before_retry 1}}
}
[root@LVS1 ~]# systemctl reload keepalived
[root@LVS1 ~]# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags-> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.100:80 wrr persistent 60-> 192.168.0.4:80               Route   10     0          0-> 192.168.0.5:80               Route   10     0          0
[root@LVS1 ~]# ifconfig lo:0 192.168.0.100 netmask 255.255.255.255 up
//添加VIP到lo
[root@LVS1 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00inet 127.0.0.1/8 scope host lovalid_lft forever preferred_lft foreverinet 192.168.0.100/32 scope global lo:0valid_lft forever preferred_lft foreverinet6 ::1/128 scope hostvalid_lft forever preferred_lft forever
2: ens33: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000link/ether 00:0c:29:af:6b:f7 brd ff:ff:ff:ff:ff:ffinet 192.168.0.2/24 brd 192.168.0.255 scope global noprefixroute ens33valid_lft forever preferred_lft foreverinet6 fe80::20c:29ff:feaf:6bf7/64 scope linkvalid_lft forever preferred_lft forever
[root@LVS1 ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.0.1     0.0.0.0         UG    100    0        0 ens33
192.168.0.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33

安装quagga并进行配置

[root@LVS1 ~]# yum install quagga.x86_64 -y
[root@LVS1 ~]# vi /etc/quagga/zebra.conf
hostname LVS1
# HOSTNAME改为IP也可以
password test
enable password test
log file /var/log/quagga/zebra.log
#log syslog
service password-encryption
interface ens33
access-list 1 permit 127.0.0.1
ip prefix-list ANY seq 5 permit 0.0.0.0/0 le 32
route-map ANY deny 10match ip address prefix-list ANY
ip protocol ospf route-map ANY
line vtyaccess-class 1
[root@LVS1 ~]# vi /etc/quagga/ospfd.conf
hostname LVS1
password test
log file /var/log/quagga/ospfd.log
log stdout
log syslog
service password-encryption
interface ens33ip ospf hello-interval 1ip ospf dead-interval 4ip ospf priority 0ip ospf cost 1
router ospfospf router-id 192.168.0.2log-adjacency-changesnetwork 192.168.0.2/24 area 0.0.0.0
access-list 1 permit 127.0.0.1
line vtyaccess-class 1
[root@LVS1 ~]# systemctl  start zebra
[root@LVS1 ~]# systemctl  start ospfd

同理我们配置完LVS2后进行检测ospf状态

[root@LVS1 ~]# vtysh
Hello, this is Quagga (version 0.99.22.4).
Copyright 1996-2005 Kunihiro Ishiguro, et al.
LVS1# show ip ospf  neighborNeighbor ID Pri State           Dead Time Address         Interface            RXmtL RqstL DBsmL
192.168.0.7       1 Full/DR            3.817s 192.168.0.1     ens33:192.168.0.2        0     0     0
192.168.0.3       0 2-Way/DROther      3.518s 192.168.0.3     ens33:192.168.0.2        0     0     0
LVS1# show ip ospf  route
============ OSPF network routing table ============
N    192.168.0.0/24        [1] area: 0.0.0.0directly attached to ens33
N    192.168.0.100/32      [1] area: 0.0.0.0directly attached to lo
N    192.168.1.0/24        [2] area: 0.0.0.0via 192.168.0.1, ens33============ OSPF router routing table ========================= OSPF external routing table ===========

在Route上查看路由

router# show ip route
Codes: K - kernel route, C - connected, S - static, R - RIP,O - OSPF, I - IS-IS, B - BGP, A - Babel,> - selected route, * - FIB routeK>* 0.0.0.0/0 via 192.168.0.10, ens33
C>* 127.0.0.0/8 is directly connected, lo
O   192.168.0.0/24 [110/1] is directly connected, ens33, 00:11:04
C>* 192.168.0.0/24 is directly connected, ens33
O   192.168.0.100/32 [110/2] via 192.168.0.2, ens33 inactive, 00:00:11via 192.168.0.3, ens33 inactive, 00:00:11
O   192.168.1.0/24 [110/1] is directly connected, ens38, 00:12:19
C>* 192.168.1.0/24 is directly connected, ens38
//可以看到 到 192.168.0.100/32 的下一跳分配到 192.168.0.2、192.168.0.3

Client 测试

[root@Client ~]# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         192.168.1.1     0.0.0.0         UG    100    0        0 ens33
192.168.1.0     0.0.0.0         255.255.255.0   U     100    0        0 ens33
[root@Client ~]# traceroute 192.168.0.100
traceroute to 192.168.0.100 (192.168.0.100), 30 hops max, 60 byte packets1  192.168.1.1 (192.168.1.1)  0.575 ms  0.258 ms  0.478 ms2  192.168.0.100 (192.168.0.100)  1.901 ms  1.746 ms  1.370 ms
[root@Client ~]# curl 192.168.0.100
RealServer1 192.168.0.4

自动化

以上是最基础的ospf-Lvs集群配置，在实际生产环境中会有自动化部署、监控、告警等需求，例如quagga的初始化配置

quagga 配置

HOSTNAME=`hostname`
PASSWORD=shenyangchangkuan
#获取默认路由对应网卡
NIC=`/sbin/route -n|awk '$1=="default"||$1=="0.0.0.0"{print $NF}'|head -n 1`
if [ -z $NIC ];thenecho "get NIC err,NIC is null!"
fi
#获取该网卡IP
#centos7改为下面这行
#IP=`/sbin/ifconfig $NIC |grep "inet "|sed 's/:/ /'|awk '{print $2}'`
IP=`/sbin/ifconfig $NIC|grep "inet addr"|sed 's/:/ /'|awk '{print $3}'`
if  [ -z $IP ];thenecho "get IP err,IP is null!"exit 1
fi#将该网卡及子网卡IP信息写入tmp/lvs_network.tmp
echo "">/tmp/lvs_network.tmp
for cfg in ` ls -l  /etc/sysconfig/network-scripts/ifcfg-${NIC}*|awk '{print $NF}'`
docat $cfg |grep IPADDR=|awk -F "=" '{print $2}' |sed 's/"//g'>>/tmp/lvs_network.tmp
done
http://www.lryc.cn/news/144360.html

相关文章：

  • IDEA、git如何修改历史提交commit的邮箱
  • 字符设备驱动（内核态用户态内存交互）
  • Qt基础 线程池
  • Django(8)-静态资源引用CSS和图片
  • C++ list模拟实现
  • 中国建筑出版传媒许少辉博士八一新书乡村振兴战略下传统村落文化旅游设计日京东当当畅销榜自由营九三学
  • C语言(第三十二天)
  • arcgis+postgresql+postgis使用介绍
  • 机器视觉之开运算和闭运算
  • 【python爬虫】—URL管理器的实现
  • Oracle 19C RAC安装PSU oui-patch.xml权限错误
  • 华为数通方向HCIP-DataCom H12-821题库(单选题：161-180）
  • ResNet详解：网络结构解读与PyTorch实现教程
  • ChatGPT 随机动态可视化图表分析
  • 国标视频融合云平台EasyCVR视频汇聚平台的应用场景及其功能说明
  • 后端面试话术集锦第三篇：spring cloud 面试话术
  • React 18 选择 State 结构
  • LNMT与动静分离
  • 【java】LinkedList 和 ArrayList的简介与对比
  • 机器学习基础14-算法调参（基于印第安糖尿病Pima数据集）
  • ASUS华硕天选4笔记本电脑FA507XV原厂Windows11系统22H2
  • IET独立出版 | EI检索 | 2023年第三届机械、航空航天与汽车工程国际会议
  • 【Pytorch】CUDA error: no kernel image is available for execution on the device
  • dolphinschedule配置企微告警服务（WeChat群组）
  • Git中smart Checkout与force checkout
  • Java“牵手”1688商品跨境属性数据，1688API接口申请指南
  • Win解答 | 解决键盘中 字母+空格 导致的输入法弹窗导致的一系列问题
  • WPF读取dicom序列:实现上一帧、下一帧、自动播放、暂停
  • homeassistant ubuntu自启动 网络设置
  • 生成式AI背景下编程工作者的技术和高级软考理论的演进融合之路