当前位置: 首页 > news >正文

Flink1.20.1集成Paimon遇到的问题

flinkcdc mysql 到paimon

1:Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.data.Schema

可以参考这个文章 明确指出了flink-connector-mysql-cdc-3.4.0.jar存在这个包,但是flink-sql-connector-mysql-cdc-3.4.0.jar中没有这个包 可以在maven仓库中看到
在这里插入图片描述

https://developer.aliyun.com/ask/574255?spm=a2c6h.12873639.article-detail.8.6a82fe85SbDdj5

缺少 kafka的 connect-api-3.3.2.jar

Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/connect/data/Schema
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1575)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:79)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:508)
at java.io.ObjectStreamClass2.run(ObjectStreamClass.java:482)atjava.security.AccessController.doPrivileged(NativeMethod)atjava.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)atjava.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)atorg.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)atorg.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)atorg.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambda2.run(ObjectStreamClass.java:482)at java.security.AccessController.doPrivileged(Native Method)at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)at org.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambda2.run(ObjectStreamClass.java:482)atjava.security.AccessController.doPrivileged(NativeMethod)atjava.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)atjava.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)atorg.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)atorg.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)atorg.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambdacreateJobVertex24(StreamingJobGraphGenerator.java:1021)atjava.util.concurrent.CompletableFuture24(StreamingJobGraphGenerator.java:1021)at java.util.concurrent.CompletableFuture24(StreamingJobGraphGenerator.java:1021)atjava.util.concurrent.CompletableFutureAsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:624)...1moreCausedby:java.lang.ClassNotFoundException:org.apache.kafka.connect.data.Schemaatjava.net.URLClassLoader.findClass(URLClassLoader.java:381)atjava.lang.ClassLoader.loadClass(ClassLoader.java:424)atsun.misc.LauncherWorker.run(ThreadPoolExecutor.java:624)... 1 more Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.data.Schemaat java.net.URLClassLoader.findClass(URLClassLoader.java:381)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.LauncherWorker.run(ThreadPoolExecutor.java:624)...1moreCausedby:java.lang.ClassNotFoundException:org.apache.kafka.connect.data.Schemaatjava.net.URLClassLoader.findClass(URLClassLoader.java:381)atjava.lang.ClassLoader.loadClass(ClassLoader.java:424)atsun.misc.LauncherAppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 45 more

2:Caused by: java.io.InvalidClassException: org.apache.flink.cdc.connectors.shaded.com.fasterxml.jackson.databind.cfg.MapperConfig; incompatible types for field _mapperFeatures

不能同时存在
flink-connector-mysql-cdc-3.4.0.jar
flink-sql-connector-mysql-cdc-3.4.0.jar

3:cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.userFunction of type org.apache.flink.api.common.functions.Function in instance of org.apache.flink.streaming.api.operators.StreamFlatMap

dinky standalone 提mysqlCDC整库同步到paimon
将dinky-app的依赖丢进flink lib中就可以了

025-07-10 10:50:36,319 WARN  org.apache.flink.runtime.taskmanager.Task                    [] - Source: MySQL CDC Source -> PartitionByPrimaryKey -> Shunt -> FlatMapRow -> *anonymous_datastream_source$6*[22] -> Calc[23] -> ConstraintEnforcer[24] -> Map (1/2)#24 (2548d4605e02b73a83d284ff33f2e106_cbc357ccb763df2852fee8c4fc7d55f2_0_24) switched from INITIALIZING to FAILED with failure cause:
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot instantiate user function.at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:416) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:869) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:836) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:825) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:825) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:202) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.<init>(RegularOperatorChain.java:60) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:789) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939) [flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) [flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist-1.20.1.jar:1.20.1]at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.userFunction of type org.apache.flink.api.common.functions.Function in instance of org.apache.flink.streaming.api.operators.StreamFlatMapat java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233) ~[?:1.8.0_171]at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1405) ~[?:1.8.0_171]at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2291) ~[?:1.8.0_171]at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) ~[?:1.8.0_171]at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) ~[?:1.8.0_171]at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) ~[?:1.8.0_171]at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285) ~[?:1.8.0_171]at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) ~[?:1.8.0_171]at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) ~[?:1.8.0_171]at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) ~[?:1.8.0_171]at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) ~[?:1.8.0_171]at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:488) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:472) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:467) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:422) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:400) ~[flink-dist-1.20.1.jar:1.20.1]... 16 more
http://www.lryc.cn/news/583317.html

相关文章:

  • stm32Cubmax的配置
  • 微信小程序91~100
  • Pycharm 报错 Environment location directory is not empty 如何解决
  • 基于Spring Boot+Vue的巴彦淖尔旅游网站(AI问答、腾讯地图API、WebSocket及时通讯、支付宝沙盒支付)
  • Ragas的Prompt Object
  • NHibernate案例
  • SAP ERP与Oracle EBS对比,两个ERP系统有什么区别?
  • aichat-core简化 LLM 与 MCP 集成的前端核心库(TypeScript)
  • C#项目 在Vue/React前端项目中 使用使用wkeWebBrowser引用并且内部使用iframe网页外链 页面部分白屏
  • Spring IoC 如何实现条件化装配 Bean?
  • HUAWEI HiCar6.0的新变化
  • 一条Redis命令是如何执行的?
  • C++随机打乱函数:简化源码与原理深度剖析
  • 从零开始学前端html篇2
  • 微信小程序控制空调之微信小程序篇
  • 双esp8266-01s间TCP通讯
  • 图像硬解码和软解码
  • RAM带宽计算及分析
  • 区块链系统开发技术应用构建可信数字生态链
  • 以太坊智能合约核心技术解析与应用实践
  • Android 组件内核
  • NO.2数据结构线性表|线性表|顺序表|链表|单链表|双向链表|循环链表|静态链表|插入|删除
  • Qt-翻金币案例
  • 鸿蒙商城开发:ZKmall开源商城系统特性适配与性能优化
  • upload-labs靶场通关详解:第18关 条件竞争
  • CMake构建项目报错“No CUDA toolset found.”
  • android——热修复(补丁)
  • [Pytest][Part 4]多种测试运行方式
  • 三、Docker常用命令
  • 安装nvm管理node.js,详细安装使用教程和详细命令