Flink1.20.1集成Paimon遇到的问题
flinkcdc mysql 到paimon
1:Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.data.Schema
可以参考这个文章 明确指出了flink-connector-mysql-cdc-3.4.0.jar存在这个包,但是flink-sql-connector-mysql-cdc-3.4.0.jar中没有这个包 可以在maven仓库中看到
https://developer.aliyun.com/ask/574255?spm=a2c6h.12873639.article-detail.8.6a82fe85SbDdj5
缺少 kafka的 connect-api-3.3.2.jar
Caused by: java.lang.NoClassDefFoundError: org/apache/kafka/connect/data/Schema
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.getDeclaredMethod(Class.java:2128)
at java.io.ObjectStreamClass.getPrivateMethod(ObjectStreamClass.java:1575)
at java.io.ObjectStreamClass.access$1700(ObjectStreamClass.java:79)
at java.io.ObjectStreamClass$2.run(ObjectStreamClass.java:508)
at java.io.ObjectStreamClass2.run(ObjectStreamClass.java:482)atjava.security.AccessController.doPrivileged(NativeMethod)atjava.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)atjava.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)atorg.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)atorg.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)atorg.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambda2.run(ObjectStreamClass.java:482)at java.security.AccessController.doPrivileged(Native Method)at java.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)at java.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)at org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)at org.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)at org.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambda2.run(ObjectStreamClass.java:482)atjava.security.AccessController.doPrivileged(NativeMethod)atjava.io.ObjectStreamClass.<init>(ObjectStreamClass.java:482)atjava.io.ObjectStreamClass.lookup(ObjectStreamClass.java:379)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1134)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)atjava.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)atjava.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)atjava.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)atjava.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)atorg.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:502)atorg.apache.flink.util.SerializedValue.<init>(SerializedValue.java:62)atorg.apache.flink.streaming.api.graph.StreamingJobGraphGenerator.lambdacreateJobVertex24(StreamingJobGraphGenerator.java:1021)atjava.util.concurrent.CompletableFuture24(StreamingJobGraphGenerator.java:1021)at java.util.concurrent.CompletableFuture24(StreamingJobGraphGenerator.java:1021)atjava.util.concurrent.CompletableFutureAsyncSupply.run(CompletableFuture.java:1590)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutorWorker.run(ThreadPoolExecutor.java:624)...1moreCausedby:java.lang.ClassNotFoundException:org.apache.kafka.connect.data.Schemaatjava.net.URLClassLoader.findClass(URLClassLoader.java:381)atjava.lang.ClassLoader.loadClass(ClassLoader.java:424)atsun.misc.LauncherWorker.run(ThreadPoolExecutor.java:624)... 1 more
Caused by: java.lang.ClassNotFoundException: org.apache.kafka.connect.data.Schemaat java.net.URLClassLoader.findClass(URLClassLoader.java:381)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at sun.misc.LauncherWorker.run(ThreadPoolExecutor.java:624)...1moreCausedby:java.lang.ClassNotFoundException:org.apache.kafka.connect.data.Schemaatjava.net.URLClassLoader.findClass(URLClassLoader.java:381)atjava.lang.ClassLoader.loadClass(ClassLoader.java:424)atsun.misc.LauncherAppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
… 45 more
2:Caused by: java.io.InvalidClassException: org.apache.flink.cdc.connectors.shaded.com.fasterxml.jackson.databind.cfg.MapperConfig; incompatible types for field _mapperFeatures
不能同时存在
flink-connector-mysql-cdc-3.4.0.jar
flink-sql-connector-mysql-cdc-3.4.0.jar
3:cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.userFunction of type org.apache.flink.api.common.functions.Function in instance of org.apache.flink.streaming.api.operators.StreamFlatMap
dinky standalone 提mysqlCDC整库同步到paimon
将dinky-app的依赖丢进flink lib中就可以了
025-07-10 10:50:36,319 WARN org.apache.flink.runtime.taskmanager.Task [] - Source: MySQL CDC Source -> PartitionByPrimaryKey -> Shunt -> FlatMapRow -> *anonymous_datastream_source$6*[22] -> Calc[23] -> ConstraintEnforcer[24] -> Map (1/2)#24 (2548d4605e02b73a83d284ff33f2e106_cbc357ccb763df2852fee8c4fc7d55f2_0_24) switched from INITIALIZING to FAILED with failure cause:
org.apache.flink.streaming.runtime.tasks.StreamTaskException: Cannot instantiate user function.at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:416) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperator(OperatorChain.java:869) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:836) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:825) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOperatorChain(OperatorChain.java:825) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.createOutputCollector(OperatorChain.java:732) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.OperatorChain.<init>(OperatorChain.java:202) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.<init>(RegularOperatorChain.java:60) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:789) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939) [flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763) [flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575) [flink-dist-1.20.1.jar:1.20.1]at java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]
Caused by: java.lang.ClassCastException: cannot assign instance of java.lang.invoke.SerializedLambda to field org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.userFunction of type org.apache.flink.api.common.functions.Function in instance of org.apache.flink.streaming.api.operators.StreamFlatMapat java.io.ObjectStreamClass$FieldReflector.setObjFieldValues(ObjectStreamClass.java:2233) ~[?:1.8.0_171]at java.io.ObjectStreamClass.setObjFieldValues(ObjectStreamClass.java:1405) ~[?:1.8.0_171]at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2291) ~[?:1.8.0_171]at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) ~[?:1.8.0_171]at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) ~[?:1.8.0_171]at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) ~[?:1.8.0_171]at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2285) ~[?:1.8.0_171]at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2209) ~[?:1.8.0_171]at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2067) ~[?:1.8.0_171]at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1571) ~[?:1.8.0_171]at java.io.ObjectInputStream.readObject(ObjectInputStream.java:431) ~[?:1.8.0_171]at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:488) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:472) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.util.InstantiationUtil.deserializeObject(InstantiationUtil.java:467) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.util.InstantiationUtil.readObjectFromConfig(InstantiationUtil.java:422) ~[flink-dist-1.20.1.jar:1.20.1]at org.apache.flink.streaming.api.graph.StreamConfig.getStreamOperatorFactory(StreamConfig.java:400) ~[flink-dist-1.20.1.jar:1.20.1]... 16 more