当前位置： 首页 > news >正文

Orchestrator源码解读2-故障失败发现

news 2025/7/21 2:12:16

目录

前言

核心流程函数调用路径

GetReplicationAnalysis

故障类型和对应的处理函数

​编辑

拓扑结构警告类型

核心流程总结

与MHA相比

前言

Orchestrator另外一个重要的功能是监控集群，发现故障。根据从复制拓扑本身获得的信息，它可以识别各种故障场景。

核心流程函数调用路径

ContinuousDiscovery
--> CheckAndRecover // 检查恢复的入口函数
--> GetReplicationAnalysis  // 查询SQL，根据实例的状态确定故障或者警告类型。检查复制问题  (dead master; unreachable master; 等)
--> executeCheckAndRecoverFunction  // 根据分析结果选择正确的检查和恢复函数。然后会同步执行该函数。
--> getCheckAndRecoverFunction // 根据分析结果选择正确的检查和恢复函数。然后会同步执行该函数。
--> runEmergentOperations // 
--> checkAndExecuteFailureDetectionProcesses //   尝试往数据库中插入这个故障发现记录，然后执行故障发现阶段所需的操作，执行 OnFailureDetectionProcesses （故障发现阶段的）钩子脚本
--> AttemptFailureDetectionRegistration //  尝试往数据库中插入故障发现记录 ，如果失败 意味着这个问题可能已经被发现了，
--> checkAndRecoverDeadMaster  // 根据故障情况 执行恢复，这里选择DeadMaster 故障类型举例 
--> AttemptRecoveryRegistration // 尝试往数据库中插入一条恢复记录；如果这一尝试失败，那么意味着恢复已经在进行中。
--> recoverDeadMaster //  用于恢复DeadMaster故障类型的函数，其中包含了完整的逻辑。会执行PreFailoverProcesses 钩子脚本

GetReplicationAnalysis

该函数将检查复制问题 (dead master; unreachable master; 等），根据实例的状态确定故障或者警告类型。

该函数会运行一个复杂SQL ,该SQL会每秒执行一次 ，执行间隔是由定时器 recoveryTick决定 ，SQL中已经进行了一部分逻辑处理，并将数据存储到 ReplicationAnalysis 结构体中，

SELECTmaster_instance.hostname,master_instance.port,master_instance.read_only AS read_only,MIN(master_instance.data_center) AS data_center,MIN(master_instance.region) AS region,MIN(master_instance.physical_environment) AS physical_environment,MIN(master_instance.master_host) AS master_host,MIN(master_instance.master_port) AS master_port,MIN(master_instance.cluster_name) AS cluster_name,MIN(master_instance.binary_log_file) AS binary_log_file,MIN(master_instance.binary_log_pos) AS binary_log_pos,MIN(IFNULL(master_instance.binary_log_file = database_instance_stale_binlog_coordinates.binary_log_fileAND master_instance.binary_log_pos = database_instance_stale_binlog_coordinates.binary_log_posAND database_instance_stale_binlog_coordinates.first_seen < NOW() - interval 10 second,0)) AS is_stale_binlog_coordinates,MIN(IFNULL(cluster_alias.alias,master_instance.cluster_name)) AS cluster_alias,MIN(IFNULL(cluster_domain_name.domain_name,master_instance.cluster_name)) AS cluster_domain,MIN(master_instance.last_checked <= master_instance.last_seenand master_instance.last_attempted_check <= master_instance.last_seen + interval 6 second) = 1 AS is_last_check_valid,/* To be considered a master, traditional async replication must not be present/valid AND the host should either *//* not be a replication group member OR be the primary of the replication group */MIN(master_instance.last_check_partial_success) as last_check_partial_success,MIN((master_instance.master_host IN ('', '_')OR master_instance.master_port = 0OR substr(master_instance.master_host, 1, 2) = '//')AND (master_instance.replication_group_name = ''OR master_instance.replication_group_member_role = 'PRIMARY')) AS is_master,MIN(master_instance.replication_group_name != ''AND master_instance.replication_group_member_state != 'OFFLINE') AS is_replication_group_member,MIN(master_instance.is_co_master) AS is_co_master,MIN(CONCAT(master_instance.hostname,':',master_instance.port) = master_instance.cluster_name) AS is_cluster_master,MIN(master_instance.gtid_mode) AS gtid_mode,COUNT(replica_instance.server_id) AS count_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seen),0) AS count_valid_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running != 0AND replica_instance.slave_sql_running != 0),0) AS count_valid_replicating_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running = 0AND replica_instance.last_io_error like '%error %connecting to master%'AND replica_instance.slave_sql_running = 1),0) AS count_replicas_failing_to_connect_to_master,MIN(master_instance.replication_depth) AS replication_depth,GROUP_CONCAT(concat(replica_instance.Hostname,':',replica_instance.Port)) as slave_hosts,MIN(master_instance.slave_sql_running = 1AND master_instance.slave_io_running = 0AND master_instance.last_io_error like '%error %connecting to master%') AS is_failing_to_connect_to_master,MIN(master_downtime.downtime_active is not nulland ifnull(master_downtime.end_timestamp, now()) > now()) AS is_downtimed,MIN(IFNULL(master_downtime.end_timestamp, '')) AS downtime_end_timestamp,MIN(IFNULL(unix_timestamp() - unix_timestamp(master_downtime.end_timestamp),0)) AS downtime_remaining_seconds,MIN(master_instance.binlog_server) AS is_binlog_server,MIN(master_instance.pseudo_gtid) AS is_pseudo_gtid,MIN(master_instance.supports_oracle_gtid) AS supports_oracle_gtid,MIN(master_instance.semi_sync_master_enabled) AS semi_sync_master_enabled,MIN(master_instance.semi_sync_master_wait_for_slave_count) AS semi_sync_master_wait_for_slave_count,MIN(master_instance.semi_sync_master_clients) AS semi_sync_master_clients,MIN(master_instance.semi_sync_master_status) AS semi_sync_master_status,SUM(replica_instance.is_co_master) AS count_co_master_replicas,SUM(replica_instance.oracle_gtid) AS count_oracle_gtid_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.oracle_gtid != 0),0) AS count_valid_oracle_gtid_replicas,SUM(replica_instance.binlog_server) AS count_binlog_server_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.binlog_server != 0),0) AS count_valid_binlog_server_replicas,SUM(replica_instance.semi_sync_replica_enabled) AS count_semi_sync_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.semi_sync_replica_enabled != 0),0) AS count_valid_semi_sync_replicas,MIN(master_instance.mariadb_gtid) AS is_mariadb_gtid,SUM(replica_instance.mariadb_gtid) AS count_mariadb_gtid_replicas,IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.mariadb_gtid != 0),0) AS count_valid_mariadb_gtid_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updates),0) AS count_logging_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updatesAND replica_instance.binlog_format = 'STATEMENT'),0) AS count_statement_based_logging_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updatesAND replica_instance.binlog_format = 'MIXED'),0) AS count_mixed_based_logging_replicas,IFNULL(SUM(replica_instance.log_binAND replica_instance.log_slave_updatesAND replica_instance.binlog_format = 'ROW'),0) AS count_row_based_logging_replicas,IFNULL(SUM(replica_instance.sql_delay > 0),0) AS count_delayed_replicas,IFNULL(SUM(replica_instance.slave_lag_seconds > 10),0) AS count_lagging_replicas,IFNULL(MIN(replica_instance.gtid_mode), '') AS min_replica_gtid_mode,IFNULL(MAX(replica_instance.gtid_mode), '') AS max_replica_gtid_mode,IFNULL(MAX(case when replica_downtime.downtime_active is not nulland ifnull(replica_downtime.end_timestamp, now()) > now() then '' else replica_instance.gtid_errant end),'') AS max_replica_gtid_errant,IFNULL(SUM(replica_downtime.downtime_active is not nulland ifnull(replica_downtime.end_timestamp, now()) > now()),0) AS count_downtimed_replicas,COUNT(DISTINCT case when replica_instance.log_binAND replica_instance.log_slave_updates then replica_instance.major_version else NULL end) AS count_distinct_logging_major_versionsFROMdatabase_instance master_instanceLEFT JOIN hostname_resolve ON (master_instance.hostname = hostname_resolve.hostname)LEFT JOIN database_instance replica_instance ON (COALESCE(hostname_resolve.resolved_hostname,master_instance.hostname) = replica_instance.master_hostAND master_instance.port = replica_instance.master_port)LEFT JOIN database_instance_maintenance ON (master_instance.hostname = database_instance_maintenance.hostnameAND master_instance.port = database_instance_maintenance.portAND database_instance_maintenance.maintenance_active = 1)LEFT JOIN database_instance_stale_binlog_coordinates ON (master_instance.hostname = database_instance_stale_binlog_coordinates.hostnameAND master_instance.port = database_instance_stale_binlog_coordinates.port)LEFT JOIN database_instance_downtime as master_downtime ON (master_instance.hostname = master_downtime.hostnameAND master_instance.port = master_downtime.portAND master_downtime.downtime_active = 1)LEFT JOIN database_instance_downtime as replica_downtime ON (replica_instance.hostname = replica_downtime.hostnameAND replica_instance.port = replica_downtime.portAND replica_downtime.downtime_active = 1)LEFT JOIN cluster_alias ON (cluster_alias.cluster_name = master_instance.cluster_name)LEFT JOIN cluster_domain_name ON (cluster_domain_name.cluster_name = master_instance.cluster_name)WHEREdatabase_instance_maintenance.database_instance_maintenance_id IS NULLAND '' IN ('', master_instance.cluster_name)GROUP BYmaster_instance.hostname,master_instance.portHAVING(MIN(master_instance.last_checked <= master_instance.last_seenand master_instance.last_attempted_check <= master_instance.last_seen + interval 6 second) = 1/* AS is_last_check_valid */) = 0OR (IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running = 0AND replica_instance.last_io_error like '%error %connecting to master%'AND replica_instance.slave_sql_running = 1),0)/* AS count_replicas_failing_to_connect_to_master */> 0)OR (IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seen),0)/* AS count_valid_replicas */< COUNT(replica_instance.server_id)/* AS count_replicas */)OR (IFNULL(SUM(replica_instance.last_checked <= replica_instance.last_seenAND replica_instance.slave_io_running != 0AND replica_instance.slave_sql_running != 0),0)/* AS count_valid_replicating_replicas */< COUNT(replica_instance.server_id)/* AS count_replicas */)OR (MIN(master_instance.slave_sql_running = 1AND master_instance.slave_io_running = 0AND master_instance.last_io_error like '%error %connecting to master%')/* AS is_failing_to_connect_to_master */)OR (COUNT(replica_instance.server_id)/* AS count_replicas */> 0)ORDER BYis_master DESC,is_cluster_master DESC,count_replicas DESC\G

故障类型和对应的处理函数

主要根据结构体 ReplicationAnalysis 中值判断故障类型。

故障类型处理函数
NoProblem没有
DeadMasterWithoutReplicas没有
DeadMastercheckAndRecoverDeadMaster
DeadMasterAndReplicascheckAndRecoverGenericProblem
DeadMasterAndSomeReplicascheckAndRecoverDeadMaster
UnreachableMasterWithLaggingReplicascheckAndRecoverGenericProblem
UnreachableMastercheckAndRecoverGenericProblem
MasterSingleReplicaNotReplicating
MasterSingleReplicaDead
AllMasterReplicasNotReplicatingcheckAndRecoverGenericProblem
AllMasterReplicasNotReplicatingOrDeadcheckAndRecoverGenericProblem
LockedSemiSyncMasterHypothesis
LockedSemiSyncMastercheckAndRecoverLockedSemiSyncMaster
MasterWithTooManySemiSyncReplicascheckAndRecoverMasterWithTooManySemiSyncReplicas
MasterWithoutReplicas
DeadCoMastercheckAndRecoverDeadCoMaster
DeadCoMasterAndSomeReplicascheckAndRecoverDeadCoMaster
UnreachableCoMaster
AllCoMasterReplicasNotReplicating
DeadIntermediateMastercheckAndRecoverDeadIntermediateMaster
DeadIntermediateMasterWithSingleReplicacheckAndRecoverDeadIntermediateMaster
DeadIntermediateMasterWithSingleReplicaFailingToConnectcheckAndRecoverDeadIntermediateMaster
DeadIntermediateMasterAndSomeReplicascheckAndRecoverDeadIntermediateMaster
DeadIntermediateMasterAndReplicascheckAndRecoverGenericProblem
UnreachableIntermediateMasterWithLaggingReplicascheckAndRecoverGenericProblem
UnreachableIntermediateMaster
AllIntermediateMasterReplicasFailingToConnectOrDeadcheckAndRecoverDeadIntermediateMaster
AllIntermediateMasterReplicasNotReplicating
FirstTierReplicaFailingToConnectToMaster
BinlogServerFailingToConnectToMaster
// Group replication problems
DeadReplicationGroupMemberWithReplicascheckAndRecoverDeadGroupMemberWithReplicas

 不知道为什么从表格中粘贴到博客中每一列的宽度不能调整，这里粘贴为图片，大家凑合看吧

拓扑结构警告类型

  • StatementAndMixedLoggingReplicasStructureWarning
  • StatementAndRowLoggingReplicasStructureWarning
  • MixedAndRowLoggingReplicasStructureWarning
  • MultipleMajorVersionsLoggingReplicasStructureWarning
  • NoLoggingReplicasStructureWarning
  • DifferentGTIDModesStructureWarning
  • ErrantGTIDStructureWarning
  • NoFailoverSupportStructureWarning
  • NoWriteableMasterStructureWarning
  • NotEnoughValidSemiSyncReplicasStructureWarning

核心流程总结

在实例发现阶段已经将实例的一些基础信息存储到了后台管理数据库的表中，故障发现阶段核心流程：

  1. 执行一个复杂SQL，查询实例和拓扑的状态，在SQL中已经进行了很多逻辑处理，状态对比。
  2. 将该SQL的查询结果存储到结构体ReplicationAnalysis中。
  3. 根据结构体ReplicationAnalysis中的字段进行条件判断，确定故障/失败类型或警告类型。
  4. 在恢复节点就会根据故障类型选择对应的故障处理函数

与MHA相比

MHA探活

核心功能主要有 MHA::HealthCheck::wait_until_unreachable 实现：

  1. 该函数通过一个死循环，检测 4 次，每次 sleep ping_interval 秒（这个值在配置文件指定,参数是 ping_interval），持续四次失败，就认为数据已经宕机；

  2. 如果有二路检测脚本，需要二路检测脚本检测主库宕机，才是真正的宕机，否则只是推出死循环，结束检测，不切换

  3. 通过添加锁来保护数据库的访问，防止脚本多次启动；

  4. 该函数可调用三种检测方法：ping_select、ping_insert、ping_connect

MHA探活流程图

MHAOC
故障类型

故障类型单一，

主要就是主库是否存活

故障类型丰富

有多种故障与警告类型

见上面的总结

探活节点

支持多个节点探活

即manager服务器与二次检查脚本中定义的服务器

多个OC节点 以及 配合该实例的从库复制状态是否正常进行检测
探活方式

ping_select、

ping_insert、

ping_connect

查询被管理的数据库，将数据写入到OC的后台管理数据库中，进行时间戳的比较或状态值比较等方式
探活间隔默认3秒默认每1秒
探活次数4次1次
故障探测速度慢，需要多次探活。快
探活理念多次多个节点避免网络抖动等配合从副本的主从状态以及OC节点的探活

http://www.lryc.cn/news/281138.html

相关文章：

  • REST2SQL是什么？它有什么功能和特性？它值不值得我们去学习？我们该如何去学习呢？
  • Android 实现获取集合中出现重复数据的值和数量
  • 【QT学习十一】QThread
  • Mybatis 39_使用MBG生成代码
  • Hudi metadata table(元数据表)
  • 提高iOS App开发效率的方法
  • MPU机制与实现详解
  • pom文件冲突引起的Excel无法下载
  • 【HarmonyOS4.0】第十篇-ArkUI布局容器组件(二)
  • PLECS如何下载第三方库并导入MOSFET 的xml文件，xml库路径添加方法及相关问题
  • 使用emu8086实现——子程序的设计
  • 快速排序、归并排序、希尔排序（2023-12-25）
  • Qt SDL2播放Wav音频
  • [ACM学习] 动态规划基础之一二三维dp
  • Qt点击按钮在其附近弹出一个窗口
  • Springboot注解@Configuration和@Bean注解作用，生命周期
  • 30天精通Nodejs--第十五天：Websocket
  • C++深入学习之STL：2、适配器、迭代器与算法部分
  • Tiktok/抖音旋转验证码识别
  • 【Java 设计模式】设计原则
  • Druid连接池工具公式化SQL附踩坑记录
  • Linux内核--网络协议栈(二)UDP数据包发送
  • 基于深度学习的时间序列算法总结
  • nginx中多个server块共用upstream会相互影响吗
  • 基于信号完整性的一些PCB设计建议
  • 《BackTrader量化交易图解》第8章：plot 绘制金融图
  • 什么是欧拉筛？？
  • 2023年全国职业院校技能大赛软件测试赛题—单元测试卷⑩
  • 使用WAF防御网络上的隐蔽威胁之SSRF攻击
  • Redis基础系列-哨兵模式