当前位置： 首页 > news >正文

【k8s中安装rabbitmq】k8s中安装rabbitmq并搭建镜像集群-hostpath版

news 2025/7/22 23:59:59

文章目录

      • 简介
      • 一.条件及环境说明
      • 二.需求说明
      • 三.实现原理及说明
      • 四.详细步骤
        • 4.1.规划节点标签
        • 4.2.创建configmap配置
        • 4.3.创建三个statefulset和service headless配置
        • 4.4.创建service配置
      • 五.安装完后的配置
      • 六.安装说明

简介

k8s集群中搭建rabbitmq集群服务一般都会用到pvc，但是考虑到有些自建k8s环境下，搭建的共享存储可能会存在稳定性及性能问题，所以这次是通过采用节点亲和性和hostpath来实现，目前的operator的基本都是采用共享存储的方法。本文将根据现有环境及不同需求将rabbitmq镜像集群的搭建采用hostpath+亲和性的权重+多副本来实现数据持久化和高可用。

一.条件及环境说明

k8s版本k8s-1.29.4，环境搭建在电信机房,六个worker节点，每个节点有一块非系统盘的ssd盘挂载到/data/路径，不采用ceph或nas之类的共享存储，也未采用operator，目前rabbitmq-operator基本都是采用了共享存储。

二.需求说明

  • 搭建rabbitmq镜像模式集群。
  • 高可用：有一个k8s节点死掉或重启pod之后也不影响使用。
  • 高性能：读写本地磁盘实现高性能的io，共享存储如果资源及硬件性能不够的话，io将会是性能瓶颈。
  • 数据安全：在发生切换,节点故障以后，数据要尽可能的保证完整
  • 安装简单、管理维护容易

三.实现原理及说明

四.详细步骤

4.1.规划节点标签
节点名称角色标签
k8s-worker-120-81主storage-selector=node-a-master
k8s-worker-120-82主storage-selector=node-b-master
k8s-worker-120-83主storage-selector=node-c-master
k8s-worker-120-84备storage-selector=node-a-slve
k8s-worker-120-85备storage-selector=node-a-slve
k8s-worker-120-86备storage-selector=node-a-slve

注：这里的rabbitmq是三节点的镜像集群，每个redis的pod分别落在三个主的节点上，当主节点发生故障是，就通过权重分配切换到备节点，例如：名称为autotest-rabbitmq-a-0的pod，默认落在标签为storage-selector=node-a-master的节点上，当该节点发生故障是，就会根据权重匹配罗在storage-selector=node-a-slve的节点上。

4.2.创建configmap配置

将如下配置保存到autotest-rabbitmq-config.yaml，配置中配置默认的vhost和用户密码，并初始化好集群节点【安装前规划好节点名称】信息。

apiVersion: v1
kind: ConfigMap
metadata:name: autotest-rabbitmq-confignamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-config 
data:enabled_plugins: |[rabbitmq_management,rabbitmq_prometheus].rabbitmq.conf: |#listeners.tcp.default = 5672default_vhost = /default_user = admindefault_pass = admindefault_user_tags.administrator = truedefault_user_tags.management = truedefault_user_tags.custom_tag = truechannel_max = 1024tcp_listen_options.backlog = 2048cluster_partition_handling = autohealcluster_partition_handling.pause_if_all_down.recover = autohealcluster_formation.peer_discovery_backend = classic_configcluster_formation.classic_config.nodes.1 = rabbit@autotest-rabbitmq-a-0.autotest-rabbitmq-a.zx-app.svc.cluster.localcluster_formation.classic_config.nodes.2 = rabbit@autotest-rabbitmq-b-0.autotest-rabbitmq-b.zx-app.svc.cluster.localcluster_formation.classic_config.nodes.3 = rabbit@autotest-rabbitmq-c-0.autotest-rabbitmq-c.zx-app.svc.cluster.local# cluster_formation.node_type = disc# cluster_keepalive_interval = 10000log.file.level = errorlog.console = true
4.3.创建三个statefulset和service headless配置

1.autotest-a.yaml 节点a的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autotest-rabbitmq-a namespace: zx-app labels:appname: pcauto-zxapp: autotest-rabbitmq-aappgroup: autotest-rabbitmq
spec:serviceName: "autotest-rabbitmq-a"replicas: 1selector:matchLabels:app: autotest-rabbitmq-aappgroup: autotest-rabbitmqtemplate:metadata:labels:app: autotest-rabbitmq-aappgroup: autotest-rabbitmqspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-a-master- weight: 20preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-a-slavecontainers:- name: rabbitmq-serverimage: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management imagePullPolicy: IfNotPresentcommand: ["/bin/sh","-c"]args: - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-serverenv:- name: RABBITMQ_ERLANG_COOKIEvalue: "YZSDHWMFSMKEMBDHSGGZ"- name: K8S_SERVICE_NAMEvalue: autotest-rabbitmq-a- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: RABBITMQ_MNESIA_BASEvalue: /home/rabbitmq/$(POD_NAME)/mnesia- name: RABBITMQ_USE_LONGNAMEvalue: "true"- name: RABBITMQ_NODENAMEvalue: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.localports:- name: httpcontainerPort: 15672 - name: amqpcontainerPort: 5672 volumeMounts:- name: rbmq-datamountPath: /home/rabbitmq - name: rabbitmq-config-volumemountPath: /etc/rabbitmq/ restartPolicy: Always terminationGracePeriodSeconds: 30volumes:- name: rbmq-datahostPath:path: /data/rabbitmqtype: ""- name: rabbitmq-config-volumeconfigMap:name: autotest-rabbitmq-configtolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-anamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-a
spec:ports:- port: 5672clusterIP: Noneselector:app: autotest-rabbitmq-a

2.autotest-b.yaml 节点b的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autotest-rabbitmq-b namespace: zx-app labels:appname: pcauto-zxapp: autotest-rabbitmq-bappgroup: autotest-rabbitmq
spec:serviceName: "autotest-rabbitmq-b"replicas: 1selector:matchLabels:app: autotest-rabbitmq-bappgroup: autotest-rabbitmqtemplate:metadata:labels:app: autotest-rabbitmq-bappgroup: autotest-rabbitmqspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-b-master- weight: 20preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-b-slavecontainers:- name: rabbitmq-serverimage: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management imagePullPolicy: IfNotPresentcommand: ["/bin/sh","-c"]args: - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-serverenv:- name: RABBITMQ_ERLANG_COOKIEvalue: "YZSDHWMFSMKEMBDHSGGZ"- name: K8S_SERVICE_NAMEvalue: autotest-rabbitmq-b- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: RABBITMQ_MNESIA_BASEvalue: /home/rabbitmq/$(POD_NAME)/mnesia- name: RABBITMQ_USE_LONGNAMEvalue: "true"- name: RABBITMQ_NODENAMEvalue: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.localports:- name: httpcontainerPort: 15672 - name: amqpcontainerPort: 5672 volumeMounts:- name: rbmq-datamountPath: /home/rabbitmq - name: rabbitmq-config-volumemountPath: /etc/rabbitmq/ restartPolicy: Always terminationGracePeriodSeconds: 30volumes:- name: rbmq-datahostPath:path: /data/rabbitmqtype: ""- name: rabbitmq-config-volumeconfigMap:name: autotest-rabbitmq-configtolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-bnamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-b
spec:ports:- port: 5672clusterIP: Noneselector:app: autotest-rabbitmq-b

3.autotest-c.yaml 节点c的statuflset配置和无头服务配置

apiVersion: apps/v1
kind: StatefulSet
metadata:name: autotest-rabbitmq-c namespace: zx-app labels:appname: pcauto-zxapp: autotest-rabbitmq-cappgroup: autotest-rabbitmq
spec:serviceName: "autotest-rabbitmq-c"replicas: 1selector:matchLabels:app: autotest-rabbitmq-cappgroup: autotest-rabbitmqtemplate:metadata:labels:app: autotest-rabbitmq-cappgroup: autotest-rabbitmqspec:affinity:nodeAffinity:preferredDuringSchedulingIgnoredDuringExecution:- weight: 80preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-c-master- weight: 20preference:matchExpressions:- key: storage-selectoroperator: Invalues:- node-c-slavecontainers:- name: rabbitmq-serverimage: xxx-vpc.cn-shenzhen.cr.aliyuncs.com/public/rabbitmq:3.12.14-management imagePullPolicy: IfNotPresentcommand: ["/bin/sh","-c"]args: - mkdir -p /home/rabbitmq/${POD_NAME};chown rabbitmq.rabbitmq -R /home/rabbitmq/${POD_NAME};rabbitmq-serverenv:- name: RABBITMQ_ERLANG_COOKIEvalue: "YZSDHWMFSMKEMBDHSGGZ"- name: K8S_SERVICE_NAMEvalue: autotest-rabbitmq-c- name: POD_IPvalueFrom:fieldRef:fieldPath: status.podIP- name: POD_NAMEvalueFrom:fieldRef:fieldPath: metadata.name- name: POD_NAMESPACEvalueFrom:fieldRef:fieldPath: metadata.namespace- name: RABBITMQ_MNESIA_BASEvalue: /home/rabbitmq/$(POD_NAME)/mnesia- name: RABBITMQ_USE_LONGNAMEvalue: "true"- name: RABBITMQ_NODENAMEvalue: rabbit@$(POD_NAME).$(K8S_SERVICE_NAME).$(POD_NAMESPACE).svc.cluster.localports:- name: httpcontainerPort: 15672 - name: amqpcontainerPort: 5672 volumeMounts:- name: rbmq-datamountPath: /home/rabbitmq - name: rabbitmq-config-volumemountPath: /etc/rabbitmq/ restartPolicy: Always terminationGracePeriodSeconds: 30volumes:- name: rbmq-datahostPath:path: /data/rabbitmqtype: ""- name: rabbitmq-config-volumeconfigMap:name: autotest-rabbitmq-configtolerations:- key: node.kubernetes.io/not-readyoperator: Existseffect: NoExecutetolerationSeconds: 3600- key: node.kubernetes.io/unreachableoperator: Existseffect: NoExecutetolerationSeconds: 3600
---
apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-cnamespace: zx-applabels:appname: pcauto-zxapp: autotest-rabbitmq-c
spec:ports:- port: 5672clusterIP: Noneselector:app: autotest-rabbitmq-c
4.4.创建service配置

给amqp端口5672和管理端口15672配置一个service的代理服务，注意其中的selector是指向到appgroup,这是将三个statefulset定义了一个appgroup的标签。

apiVersion: v1
kind: Service
metadata:name: autotest-rabbitmq-managenamespace: zx-applabels:app: autotest-rabbitmq-manageappname: pcauto-zx
spec:ports:- port: 5672name: amqp- port: 15672name: httpselector:appgroup: autotest-rabbitmqsessionAffinity: ClientIPtype: LoadBalancer

五.安装完后的配置

将rabbitmq的集群设置成3节点的镜像集群，在4.4步骤操作完以后就可以通过loadbalancer的IP进行登录控制台了：http://ip:15672,用configmap中设置的 default_user和default_pass来登录。
在这里插入图片描述

镜像模式设置：
在这里插入图片描述

设置后：
在这里插入图片描述
在这里插入图片描述

镜像模式可以通过命令设置：
设置demo的vhost的镜像执行命令：
rabbitmqctl set_policy -p demo ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'
设置默认vhost /的镜像模式：
rabbitmqctl set_policy ha-all "^" '{"ha-mode":"all","ha-sync-mode":"automatic"}'

六.安装说明

  • 需要先在k8s的节点创建/data/rabbitmq的路径来存放数据。
  • 使用配置前将配置中的autotest替换成自己需要的命名,配置的命名空间为：zx-app，修改成自己的命名空间，appname为pcauto-zx，修改成自己的应用名称，该标签也可以删除。
  • 配置中的镜像地址采用了私有的镜像地址，镜像是dockerhub上下载的rabbitmq:3.12.14-management 镜像到私有镜像仓库的，如果能直通外网的k8s可以直接用rabbitmq:3.12.14-management 。否则配置一个可以通的镜像地址。
  • 默认的用户名和密码是admin。
  • 节点标签的命名规则若有更改需要在每个statefulset中的affinity: 中的标签名称进行修改。
http://www.lryc.cn/news/396667.html

相关文章：

  • (5) 深入探索Python-Pandas库的核心数据结构：Series详解
  • JAVA之开发神器——IntelliJ IDEA的下载与安装
  • 通过Umijs从0到1搭建一个React项目
  • Redis 数据过期及淘汰策略
  • vue vite+three在线编辑模型导入导出
  • 去水印小程序源码修复版-前端后端内置接口+第三方接口
  • 机器学习：预测评估8类指标
  • 【深度学习基础】MAC pycharm 专业版安装与激活
  • 排序相关算法--1.插入排序+冒泡排序回顾
  • 变阻器的故障排除方法有哪些？
  • 软考《信息系统运行管理员》-3.1信息系统设施运维的管理体系
  • Nginx重定向
  • 私有化地图离线部署方案之高程检索服务
  • PostgreSQL 中如何实现数据的增量更新和全量更新的平衡？
  • 数据结构--二叉树相关习题5（判断二叉树是否是完全二叉树 ）
  • Python 轻松生成多种条形码、二维码 （Code 128、EAN-13、QR code等）
  • Python: 分块读取文本文件
  • 服务攻防——中间件Jboss
  • 宏碁F5-572G-59K3笔记本笔记本电脑拆机清灰教程（详解）
  • 基于FPGA的LDPC编译码算法设计基础知识
  • 国际网课平台Udemy上的亚马逊云科技AWS免费高分课程和创建、维护EC2动手实践
  • 空中交通新动能！2024深圳eVTOL展动力电池展区核心内容抢先看！
  • 代码江湖：Python 中的进程与线程
  • 根据H在有限域GF(2^m)上求解生成矩阵G
  • Django 实现子模版继承父模板
  • 数据安全治理：从库级权限申请到表级权限申请
  • vue3源码（六）渲染原理-runtime-core
  • python拆分Excel数据，自动发邮箱
  • 2024年福州延安中学夏季拿云杯拔尖创新人才素养测试（小高组）
  • ES6 之 Promise 构造函数知识点总结 （四）