当前位置: 首页 > news >正文

【flink】Checkpoint expired before completing.

使用flink同步数据出现错误Checkpoint expired before completing.


11:32:34,455 WARN  org.apache.flink.runtime.checkpoint.CheckpointFailureManager [Checkpoint Timer]  - Failed to trigger or complete checkpoint 4 for job 1b1d41031ea45d15bdb3324004c2d749. (2 consecutive failed attempts so far)
org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint expired before completing.at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2143)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)at java.util.concurrent.FutureTask.run(FutureTask.java)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
11:32:34,459 INFO  org.jobslink.flink.sink.OperateMysqlDataSink                 [Source: CDC Sourceorg.jobslink.flink.TradeAndWorkTypeAndSkillsCDCJob -> (Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink jk_skills_base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink jk_trade_base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink jk_worktypes_base)) (1/1)#0]  - READ isExitSql is : [ SELECT count(1) count from jobslink_data_platform.src_skills_base where id= 1325753409319084034 ] 
11:32:34,468 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [flink-akka.actor.default-dispatcher-9]  - Trying to recover from a global failure.
org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold.at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.checkFailureAgainstCounter(CheckpointFailureManager.java:206)at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleJobLevelCheckpointException(CheckpointFailureManager.java:169)at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:122)at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2082)at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:2061)at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.access$600(CheckpointCoordinator.java:98)at org.apache.flink.runtime.checkpoint.CheckpointCoordinator$CheckpointCanceller.run(CheckpointCoordinator.java:2143)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)at java.util.concurrent.FutureTask.run(FutureTask.java)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)at java.lang.Thread.run(Thread.java:748)
11:32:34,470 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [flink-akka.actor.default-dispatcher-9]  - Job org.jobslink.flink.TradeAndWorkTypeAndSkillsCDCJob (1b1d41031ea45d15bdb3324004c2d749) switched from state RUNNING to RESTARTING.
11:32:34,471 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [flink-akka.actor.default-dispatcher-9]  - Source: CDC Sourceorg.jobslink.flink.TradeAndWorkTypeAndSkillsCDCJob -> (Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink base), Filter -> Flat Map -> Filter -> (Sink: Print to Std. Out, Sink: sink base)) (1/1) (3525ceb58f2dc3264812966ec8600a19) switched from RUNNING to CANCELING.

任务超时了:

重新把任务配置参数,配置如下:

//开启CK
env.getCheckpointConfig().setCheckpointTimeout(60000);
//设置定期安排检查点的时间间隔。
env.getCheckpointConfig().setCheckpointInterval(60000);
//设置可能同时进行的检查点尝试的最大次数
env.getCheckpointConfig().setMaxConcurrentCheckpoints(500);
//设置检查点尝试之间的最小暂停时间。
env.getCheckpointConfig().setMinPauseBetweenCheckpoints(500);

或者修改

flink的 配置文件flink-conf.yaml 

http://www.lryc.cn/news/118773.html

相关文章:

  • 【论文阅读】NoDoze:使用自动来源分类对抗威胁警报疲劳(NDSS-2019)
  • 【ARM64 常见汇编指令学习 16 -- ARM64 SMC 指令】
  • uprobe trace多线程mutex等待耗时
  • Linux 和 MacOS 中的 profile 文件详解(一)
  • 不用技术代码,如何制作成绩查询系统?
  • flinksql sink to sr often fail because of nullpoint
  • 达梦数据库:Error updating database. Cause: dm.jdbc.driver.DMException: 数据未找到
  • 电脑怎么查看连接过的WIFI密码(测试环境win11,win10也能用)
  • 处理数据部分必备代码
  • layui 集成 ztree异步加载
  • LeetCode面向运气之Javascript—第27题-移除元素-98.93%
  • 谷歌云 | 电子商务 | 如何更好地管理客户身份以支持最佳的用户体验
  • 行业追踪,2023-08-09
  • 视图、存储过程、函数、触发器
  • 数学建模学习(10):遗传算法
  • 私域流量整合:社群裂变的综合策略
  • Redis的RDB持久化
  • 三、MySql表的操作
  • 【模型加速部署】—— Pytorch自动混合精度训练
  • 【Qt】信号槽的三种连接方式
  • Jtti:Ubuntu静态IP地址怎么配置
  • iconfont 使用
  • 基于java冰雪旅游服务网设计与实现
  • django处理分页
  • CI+JUnit5并发单测机制创新实践
  • 蚁剑antSword-maste下载-安装-使用-一句话木马
  • [保研/考研机试] KY80 进制转换 北京大学复试上机题 C++实现
  • AP2915DC-DC降压恒流驱动IC LED电源驱动芯片 汽车摩托电动车灯
  • Android 实现无预览拍照功能
  • 第一章-数据结构绪论