当前位置： 首页 > news >正文

DataX工具部署与使用(PostgreSQL to Oracle)

news 2025/7/5 22:24:15

目录：

    • 一、准备环境
      • （1）安装JDK
      • （2）检查Python版本
      • （3）DataX 解压及测试
    • 二、同步测试
      • 1、配置清单
      • 2、操作示例
        • （1）同步测试环境
        • （2）准备测试表（postgresql）
        • （3）准备测试表（Oracle）
        • （4）创建并编辑json文件
        • （5）执行json配置文件

一、准备环境

1、创建用户组和目录

groupadd -g 1500 datax
useradd -g datax -u 1500 datax
mkdir /datax
chown datax:datax /datax

安装datax两端服务器都可以，但是要保证网络畅通

2、系统环境

  • Linux
  • JDK(1.8以上，推荐1.8)
  • Python(2或3都可以)
  • Apache Maven 3.x (Compile DataX)（本文使用datax工具包直接解压，未采用编译方式）

（1）安装JDK

安装JDK（1.8.0_361）

# rpm -ivh jdk-8u361-linux-x64.rpm
warning: jdk-8u361-linux-x64.rpm: Header V3 RSA/SHA256 Signature, key ID ec551f03: NOKEY
Preparing...                          ################################# [100%]
Updating / installing...1:jdk1.8-2000:1.8.0_361-fcs        ################################# [100%]
Unpacking JAR files...tools.jar...plugin.jar...javaws.jar...deploy.jar...rt.jar...jsse.jar...charsets.jar...localedata.jar...

检查JDK版本

# java -version
java version "1.8.0_361"
Java(TM) SE Runtime Environment (build 1.8.0_361-b09)
Java HotSpot(TM) 64-Bit Server VM (build 25.361-b09, mixed mode)

（2）检查Python版本

# python -V
Python 2.7.5

我这边使用的是linux7自带的python版本

（3）DataX 解压及测试

Download ：DataX下载地址，上传到/datax目录

解压缩：

# tar -zxvf datax.tar.gz

自检脚本：

python {YOUR_DATAX_HOME}/bin/datax.py {YOUR_DATAX_HOME}/job/job.json

示例：

# python /datax/datax/bin/datax.py /datax/datax/job/job.jsonDataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.2023-10-10 11:31:06.527 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-10-10 11:31:06.531 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-10-10 11:31:06.571 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-10-10 11:31:06.577 [main] INFO  Engine - the machine info  => osInfo:	Oracle Corporation 1.8 25.361-b09jvmInfo:	Linux amd64 5.4.17-2011.6.2.el7uek.x86_64cpu num:	4totalPhysicalMemory:	-0.00GfreePhysicalMemory:	-0.00GmaxFileDescriptorCount:	-1currentOpenFileDescriptorCount:	-1GC Names	[PS MarkSweep, PS Scavenge]MEMORY_NAME                    | allocation_size                | init_size                      PS Eden Space                  | 256.00MB                       | 256.00MB                       Code Cache                     | 240.00MB                       | 2.44MB                         Compressed Class Space         | 1,024.00MB                     | 0.00MB                         PS Survivor Space              | 42.50MB                        | 42.50MB                        PS Old Gen                     | 683.00MB                       | 683.00MB                       Metaspace                      | -0.00MB                        | 0.00MB                         2023-10-10 11:31:06.595 [main] INFO  Engine - 
{"content":[{"reader":{"name":"streamreader","parameter":{"column":[{"type":"string","value":"DataX"},{"type":"long","value":19890604},{"type":"date","value":"1989-06-04 00:00:00"},{"type":"bool","value":true},{"type":"bytes","value":"test"}],"sliceRecordCount":100000}},"writer":{"name":"streamwriter","parameter":{"encoding":"UTF-8","print":false}}}],"setting":{"errorLimit":{"percentage":0.02,"record":0},"speed":{"channel":1}}
}2023-10-10 11:31:06.615 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-10-10 11:31:06.617 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-10-10 11:31:06.617 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-10-10 11:31:06.620 [main] INFO  JobContainer - Set jobId = 0
2023-10-10 11:31:06.641 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-10-10 11:31:06.643 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do prepare work .
2023-10-10 11:31:06.643 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do prepare work .
2023-10-10 11:31:06.643 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-10-10 11:31:06.643 [job-0] INFO  JobContainer - Job set Channel-Number to 1 channels.
2023-10-10 11:31:06.644 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] splits to [1] tasks.
2023-10-10 11:31:06.645 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] splits to [1] tasks.
2023-10-10 11:31:06.667 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-10-10 11:31:06.674 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-10-10 11:31:06.677 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-10-10 11:31:06.685 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [1] channels for [1] tasks.
2023-10-10 11:31:06.688 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-10-10 11:31:06.688 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-10-10 11:31:06.701 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-10-10 11:31:07.003 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[304]ms
2023-10-10 11:31:07.004 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-10-10 11:31:16.696 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.050s |  All Task WaitReaderTime 0.060s | Percentage 100.00%
2023-10-10 11:31:16.696 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-10-10 11:31:16.696 [job-0] INFO  JobContainer - DataX Writer.Job [streamwriter] do post work.
2023-10-10 11:31:16.697 [job-0] INFO  JobContainer - DataX Reader.Job [streamreader] do post work.
2023-10-10 11:31:16.697 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-10-10 11:31:16.698 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /datax/datax/hook
2023-10-10 11:31:16.700 [job-0] INFO  JobContainer - [total cpu info] => averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    -1.00%                         | -1.00%                         | -1.00%[total gc info] => NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     PS MarkSweep         | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             PS Scavenge          | 0                  | 0                  | 0                  | 0.000s             | 0.000s             | 0.000s             2023-10-10 11:31:16.700 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-10-10 11:31:16.701 [job-0] INFO  StandAloneJobContainerCommunicator - Total 100000 records, 2600000 bytes | Speed 253.91KB/s, 10000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.050s |  All Task WaitReaderTime 0.060s | Percentage 100.00%
2023-10-10 11:31:16.705 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2023-10-10 11:31:06
任务结束时刻                    : 2023-10-10 11:31:16
任务总计耗时                    :                 10s
任务平均流量                    :          253.91KB/s
记录写入速度                    :          10000rec/s
读出记录总数                    :              100000
读写失败总数                    :                   0

自检脚本无法通过报错处理：https://developer.aliyun.com/ask/376622?spm=a2c6h.13148508.0.0.1d734f0eKYBCG7

二、同步测试

1、配置清单

在两个不同数据库之间使用datax数据迁移可以通过命令查看配置模板： python datax.py -r {Sourcedb_READER} -w {Targetdb_WRITER}

示例：查看PostgreSQL到Oracle数据传输json文件模板

# python /datax/datax/bin/datax.py -r postgresqlreader -w oraclewriterDataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.Please refer to the postgresqlreader document:https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md Please refer to the oraclewriter document:https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md Please save the following configuration as a json file and  usepython {DATAX_HOME}/bin/datax.py {JSON_FILE_NAME}.json 
to run the job.{"job": {"content": [{"reader": {"name": "postgresqlreader", "parameter": {"connection": [{"jdbcUrl": [], "table": []}], "password": "", "username": ""}}, "writer": {"name": "oraclewriter", "parameter": {"column": [], "connection": [{"jdbcUrl": "", "table": []}], "password": "", "preSql": [], "username": ""}}}], "setting": {"speed": {"channel": ""}}}
}

详情查看：

Please refer to the postgresqlreader document:
https://github.com/alibaba/DataX/blob/master/postgresqlreader/doc/postgresqlreader.md

Please refer to the oraclewriter document:
https://github.com/alibaba/DataX/blob/master/oraclewriter/doc/oraclewriter.md

2、操作示例

（1）同步测试环境
OS版本CentOS Linux release 7.6.1810 (Core)Ol7.9
DB版本PostgreSQL12.3Oracle19c (经典模式)
同步方向ReaderWriter
IP192.168.245.9192.168.245.10
端口54321521
数据库名htdborcl
用户/密码htuser/htuser123shuaige/shuaige123
表名httabhttab

两个虚拟机均使用OracleLinux7.9操作系统最小化安装，关闭防火墙和selinux

（2）准备测试表（postgresql）
-- 1、登录postgresql
------------------------------------------------------------------------------------
$ psql -h192.168.245.9 -p5432 -Uhtuser htdb
Password for user htuser: 
psql (12.3)
Type "help" for help.htdb=# \c
You are now connected to database "htdb" as user "htuser".
htdb=#-- 2、创建测试表 
------------------------------------------------------------------------------------
htdb=# create table httab(id serial primary key,name text,age int);
CREATE TABLE-- 3、检查新建表
------------------------------------------------------------------------------------
htdb=# \d httabTable "public.httab"Column |  Type   | Collation | Nullable |              Default              
--------+---------+-----------+----------+-----------------------------------id     | integer |           | not null | nextval('httab_id_seq'::regclass)name   | text    |           |          | age    | integer |           |          | 
Indexes:"httab_pkey" PRIMARY KEY, btree (id)htdb=#-- 4、插入测试数据
------------------------------------------------------------------------------------
htdb=# insert into  httab(name,age)  select  md5(random()::text),rn from  generate_series(1,10000) rn;
INSERT 0 10000-- 5、查询测试数据
------------------------------------------------------------------------------------
htdb=# select * from httab;id   |               name               |  age  
-------+----------------------------------+-------10001 | 4c943d9e2be750b836365166b8bbfdf1 |     110002 | ddbad0963e08c9642d1692e9961677f7 |     210003 | 40551c9e3f00b4e7ea7c81062a94b20b |     3
...
（3）准备测试表（Oracle）
-- 1、登录oracle
------------------------------------------------------------------------------------
$ sqlplus / as sysdbaSQL*Plus: Release 19.0.0.0.0 - Production on Tue Oct 10 16:06:40 2023
Version 19.3.0.0.0Copyright (c) 1982, 2019, Oracle.  All rights reserved.Connected to:
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production
Version 19.3.0.0.0SYS@orcl>-- 1、创建测试表
------------------------------------------------------------------------------------
SYS@orcl> create table shuaige.httab (id number(30) primary key,name VARCHAR2(1000) not null,age int);Table created.

注意更改用户名和表空间,不指定使用用户默认表空间

（4）创建并编辑json文件

在/datax/datax/job目录下用postgresqlreader to oraclewriter模板配置json文件

vi /datax/datax/job/postgresql2oracle_httab.json

编辑配置模板按照实际环境修改：

{"job": {"content": [{"reader": {"name": "postgresqlreader", "parameter": {"connection": [{"jdbcUrl": ["jdbc:postgresql://192.168.245.9:5432/htdb"], "table": ["httab"]}], "column": ["*"],"splitPk": "id","password": "htuser123", "username": "htuser"}}, "writer": {"name": "oraclewriter", "parameter": {"column": ["*"], "connection": [{"jdbcUrl": "jdbc:oracle:thin:@192.168.245.10:1521:orcl", "table": ["httab"]}], "password": "shuaige123", "preSql": ["truncate table httab"], "username": "shuaige"}}}], "setting": {"speed": {"channel": "4"}}}
}
（5）执行json配置文件
# python /datax/datax/bin/datax.py /datax/datax/job/postgresql2oracle_httab.jsonDataX (DATAX-OPENSOURCE-3.0), From Alibaba !
Copyright (C) 2010-2017, Alibaba Group. All Rights Reserved.2023-10-10 17:04:26.033 [main] INFO  MessageSource - JVM TimeZone: GMT+08:00, Locale: zh_CN
2023-10-10 17:04:26.035 [main] INFO  MessageSource - use Locale: zh_CN timeZone: sun.util.calendar.ZoneInfo[id="GMT+08:00",offset=28800000,dstSavings=0,useDaylight=false,transitions=0,lastRule=null]
2023-10-10 17:04:26.060 [main] INFO  VMInfo - VMInfo# operatingSystem class => sun.management.OperatingSystemImpl
2023-10-10 17:04:26.066 [main] INFO  Engine - the machine info  => osInfo:	Oracle Corporation 1.8 25.361-b09jvmInfo:	Linux amd64 5.4.17-2011.6.2.el7uek.x86_64cpu num:	4totalPhysicalMemory:	-0.00GfreePhysicalMemory:	-0.00GmaxFileDescriptorCount:	-1currentOpenFileDescriptorCount:	-1GC Names	[PS MarkSweep, PS Scavenge]MEMORY_NAME                    | allocation_size                | init_size                      PS Eden Space                  | 256.00MB                       | 256.00MB                       Code Cache                     | 240.00MB                       | 2.44MB                         Compressed Class Space         | 1,024.00MB                     | 0.00MB                         PS Survivor Space              | 42.50MB                        | 42.50MB                        PS Old Gen                     | 683.00MB                       | 683.00MB                       Metaspace                      | -0.00MB                        | 0.00MB                         2023-10-10 17:04:26.080 [main] INFO  Engine - 
{"content":[{"reader":{"name":"postgresqlreader","parameter":{"column":["*"],"connection":[{"jdbcUrl":["jdbc:postgresql://192.168.245.9:5432/htdb"],"table":["httab"]}],"password":"*********","splitPk":"id","username":"htuser"}},"writer":{"name":"oraclewriter","parameter":{"column":["*"],"connection":[{"jdbcUrl":"jdbc:oracle:thin:@192.168.245.10:1521:orcl","table":["httab"]}],"password":"**********","preSql":["truncate table httab"],"username":"shuaige"}}}],"setting":{"speed":{"channel":"4"}}
}2023-10-10 17:04:26.094 [main] WARN  Engine - prioriy set to 0, because NumberFormatException, the value is: null
2023-10-10 17:04:26.095 [main] INFO  PerfTrace - PerfTrace traceId=job_-1, isEnable=false, priority=0
2023-10-10 17:04:26.096 [main] INFO  JobContainer - DataX jobContainer starts job.
2023-10-10 17:04:26.097 [main] INFO  JobContainer - Set jobId = 0
2023-10-10 17:04:26.372 [job-0] INFO  OriginalConfPretreatmentUtil - Available jdbcUrl:jdbc:postgresql://192.168.245.9:5432/htdb.
2023-10-10 17:04:26.373 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置存在一定的风险. 因为您未配置读取数据库表的列，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-10-10 17:04:26.576 [job-0] INFO  OriginalConfPretreatmentUtil - table:[httab] all columns:[
ID,NAME,AGE
].
2023-10-10 17:04:26.576 [job-0] WARN  OriginalConfPretreatmentUtil - 您的配置文件中的列配置信息存在风险. 因为您配置的写入数据库表的列为*，当您的表字段个数、类型有变动时，可能影响任务正确性甚至会运行出错。请检查您的配置并作出修改.
2023-10-10 17:04:26.578 [job-0] INFO  OriginalConfPretreatmentUtil - Write data [
INSERT INTO %s (ID,NAME,AGE) VALUES(?,?,?)
], which jdbcUrl like:[jdbc:oracle:thin:@192.168.245.10:1521:orcl]
2023-10-10 17:04:26.578 [job-0] INFO  JobContainer - jobContainer starts to do prepare ...
2023-10-10 17:04:26.578 [job-0] INFO  JobContainer - DataX Reader.Job [postgresqlreader] do prepare work .
2023-10-10 17:04:26.578 [job-0] INFO  JobContainer - DataX Writer.Job [oraclewriter] do prepare work .
2023-10-10 17:04:26.627 [job-0] INFO  CommonRdbmsWriter$Job - Begin to execute preSqls:[truncate table httab]. context info:jdbc:oracle:thin:@192.168.245.10:1521:orcl.
2023-10-10 17:04:26.648 [job-0] INFO  JobContainer - jobContainer starts to do split ...
2023-10-10 17:04:26.648 [job-0] INFO  JobContainer - Job set Channel-Number to 4 channels.
2023-10-10 17:04:26.667 [job-0] INFO  SingleTableSplitUtil - split pk [sql=SELECT MIN(id),MAX(id) FROM httab] is running... 
2023-10-10 17:04:26.690 [job-0] INFO  SingleTableSplitUtil - After split(), allQuerySql=[
select * from httab  where  (10001 <= id AND id < 10501) 
select * from httab  where  (10501 <= id AND id < 11001) 
select * from httab  where  (11001 <= id AND id < 11501) 
select * from httab  where  (11501 <= id AND id < 12001) 
select * from httab  where  (12001 <= id AND id < 12501) 
select * from httab  where  (12501 <= id AND id < 13001) 
select * from httab  where  (13001 <= id AND id < 13501) 
select * from httab  where  (13501 <= id AND id < 14001) 
select * from httab  where  (14001 <= id AND id < 14501) 
select * from httab  where  (14501 <= id AND id < 15001) 
select * from httab  where  (15001 <= id AND id < 15501) 
select * from httab  where  (15501 <= id AND id < 16001) 
select * from httab  where  (16001 <= id AND id < 16501) 
select * from httab  where  (16501 <= id AND id < 17001) 
select * from httab  where  (17001 <= id AND id < 17501) 
select * from httab  where  (17501 <= id AND id < 18001) 
select * from httab  where  (18001 <= id AND id < 18501) 
select * from httab  where  (18501 <= id AND id < 19001) 
select * from httab  where  (19001 <= id AND id < 19501) 
select * from httab  where  (19501 <= id AND id <= 20000) 
select * from httab  where  id IS NULL
].
2023-10-10 17:04:26.691 [job-0] INFO  JobContainer - DataX Reader.Job [postgresqlreader] splits to [21] tasks.
2023-10-10 17:04:26.692 [job-0] INFO  JobContainer - DataX Writer.Job [oraclewriter] splits to [21] tasks.
2023-10-10 17:04:26.712 [job-0] INFO  JobContainer - jobContainer starts to do schedule ...
2023-10-10 17:04:26.756 [job-0] INFO  JobContainer - Scheduler starts [1] taskGroups.
2023-10-10 17:04:26.758 [job-0] INFO  JobContainer - Running by standalone Mode.
2023-10-10 17:04:26.771 [taskGroup-0] INFO  TaskGroupContainer - taskGroupId=[0] start [4] channels for [21] tasks.
2023-10-10 17:04:26.775 [taskGroup-0] INFO  Channel - Channel set byte_speed_limit to -1, No bps activated.
2023-10-10 17:04:26.776 [taskGroup-0] INFO  Channel - Channel set record_speed_limit to -1, No tps activated.
2023-10-10 17:04:26.782 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[14] attemptCount[1] is started
2023-10-10 17:04:26.785 [0-0-14-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (17001 <= id AND id < 17501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.786 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[3] attemptCount[1] is started
2023-10-10 17:04:26.787 [0-0-3-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (11501 <= id AND id < 12001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.801 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[13] attemptCount[1] is started
2023-10-10 17:04:26.802 [0-0-13-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (16501 <= id AND id < 17001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.807 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[2] attemptCount[1] is started
2023-10-10 17:04:26.807 [0-0-2-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (11001 <= id AND id < 11501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.831 [0-0-14-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (17001 <= id AND id < 17501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.836 [0-0-2-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (11001 <= id AND id < 11501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.851 [0-0-13-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (16501 <= id AND id < 17001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:26.865 [0-0-3-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (11501 <= id AND id < 12001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.110 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[13] is successed, used[323]ms
2023-10-10 17:04:27.157 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[4] attemptCount[1] is started
2023-10-10 17:04:27.157 [0-0-4-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (12001 <= id AND id < 12501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.180 [0-0-4-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (12001 <= id AND id < 12501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.258 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[2] is successed, used[456]ms
2023-10-10 17:04:27.258 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[14] is successed, used[477]ms
2023-10-10 17:04:27.259 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[18] attemptCount[1] is started
2023-10-10 17:04:27.261 [0-0-18-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (19001 <= id AND id < 19501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.262 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] attemptCount[1] is started
2023-10-10 17:04:27.263 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (10001 <= id AND id < 10501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.282 [0-0-18-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (19001 <= id AND id < 19501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.300 [0-0-0-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (10001 <= id AND id < 10501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.363 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[3] is successed, used[578]ms
2023-10-10 17:04:27.363 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[4] is successed, used[252]ms
2023-10-10 17:04:27.365 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[15] attemptCount[1] is started
2023-10-10 17:04:27.366 [0-0-15-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (17501 <= id AND id < 18001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.368 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[6] attemptCount[1] is started
2023-10-10 17:04:27.368 [0-0-6-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (13001 <= id AND id < 13501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.387 [0-0-15-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (17501 <= id AND id < 18001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.402 [0-0-6-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (13001 <= id AND id < 13501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.569 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[0] is successed, used[309]ms
2023-10-10 17:04:27.571 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[5] attemptCount[1] is started
2023-10-10 17:04:27.571 [0-0-5-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (12501 <= id AND id < 13001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.590 [0-0-5-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (12501 <= id AND id < 13001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.672 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[6] is successed, used[306]ms
2023-10-10 17:04:27.672 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[15] is successed, used[307]ms
2023-10-10 17:04:27.673 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[18] is successed, used[414]ms
2023-10-10 17:04:27.674 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[1] attemptCount[1] is started
2023-10-10 17:04:27.675 [0-0-1-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (10501 <= id AND id < 11001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.677 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[10] attemptCount[1] is started
2023-10-10 17:04:27.677 [0-0-10-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (15001 <= id AND id < 15501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.679 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[17] attemptCount[1] is started
2023-10-10 17:04:27.679 [0-0-17-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (18501 <= id AND id < 19001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.692 [0-0-1-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (10501 <= id AND id < 11001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.709 [0-0-17-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (18501 <= id AND id < 19001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.725 [0-0-10-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (15001 <= id AND id < 15501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.981 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[10] is successed, used[306]ms
2023-10-10 17:04:27.981 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[17] is successed, used[303]ms
2023-10-10 17:04:27.983 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[11] attemptCount[1] is started
2023-10-10 17:04:27.983 [0-0-11-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (15501 <= id AND id < 16001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:27.985 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[16] attemptCount[1] is started
2023-10-10 17:04:27.985 [0-0-16-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (18001 <= id AND id < 18501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.008 [0-0-11-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (15501 <= id AND id < 16001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.021 [0-0-16-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (18001 <= id AND id < 18501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.086 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[5] is successed, used[516]ms
2023-10-10 17:04:28.090 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[19] attemptCount[1] is started
2023-10-10 17:04:28.090 [0-0-19-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (19501 <= id AND id <= 20000) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.217 [0-0-19-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (19501 <= id AND id <= 20000) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.307 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[1] is successed, used[633]ms
2023-10-10 17:04:28.309 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[7] attemptCount[1] is started
2023-10-10 17:04:28.309 [0-0-7-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (13501 <= id AND id < 14001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.376 [0-0-7-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (13501 <= id AND id < 14001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.511 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[19] is successed, used[423]ms
2023-10-10 17:04:28.513 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[8] attemptCount[1] is started
2023-10-10 17:04:28.514 [0-0-8-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (14001 <= id AND id < 14501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.617 [0-0-8-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (14001 <= id AND id < 14501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.717 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[11] is successed, used[735]ms
2023-10-10 17:04:28.717 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[16] is successed, used[734]ms
2023-10-10 17:04:28.718 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[9] attemptCount[1] is started
2023-10-10 17:04:28.719 [0-0-9-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (14501 <= id AND id < 15001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.720 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[12] attemptCount[1] is started
2023-10-10 17:04:28.720 [0-0-12-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  (16001 <= id AND id < 16501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.814 [0-0-9-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (14501 <= id AND id < 15001) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:28.892 [0-0-12-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  (16001 <= id AND id < 16501) 
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.248 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[8] is successed, used[736]ms
2023-10-10 17:04:29.250 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[20] attemptCount[1] is started
2023-10-10 17:04:29.250 [0-0-20-reader] INFO  CommonRdbmsReader$Task - Begin to read record by Sql: [select * from httab  where  id IS NULL
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.269 [0-0-20-reader] INFO  CommonRdbmsReader$Task - Finished read record by Sql: [select * from httab  where  id IS NULL
] jdbcUrl:[jdbc:postgresql://192.168.245.9:5432/htdb].
2023-10-10 17:04:29.502 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[12] is successed, used[783]ms
2023-10-10 17:04:29.706 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[7] is successed, used[1398]ms
2023-10-10 17:04:29.806 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[9] is successed, used[1088]ms
2023-10-10 17:04:29.806 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] taskId[20] is successed, used[557]ms
2023-10-10 17:04:29.807 [taskGroup-0] INFO  TaskGroupContainer - taskGroup[0] completed it's tasks.
2023-10-10 17:04:36.773 [job-0] INFO  StandAloneJobContainerCommunicator - Total 10000 records, 408894 bytes | Speed 39.93KB/s, 1000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.010s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-10-10 17:04:36.773 [job-0] INFO  AbstractScheduler - Scheduler accomplished all tasks.
2023-10-10 17:04:36.774 [job-0] INFO  JobContainer - DataX Writer.Job [oraclewriter] do post work.
2023-10-10 17:04:36.774 [job-0] INFO  JobContainer - DataX Reader.Job [postgresqlreader] do post work.
2023-10-10 17:04:36.774 [job-0] INFO  JobContainer - DataX jobId [0] completed successfully.
2023-10-10 17:04:36.775 [job-0] INFO  HookInvoker - No hook invoked, because base dir not exists or is a file: /datax/datax/hook
2023-10-10 17:04:36.775 [job-0] INFO  JobContainer - [total cpu info] => averageCpu                     | maxDeltaCpu                    | minDeltaCpu                    -1.00%                         | -1.00%                         | -1.00%[total gc info] => NAME                 | totalGCCount       | maxDeltaGCCount    | minDeltaGCCount    | totalGCTime        | maxDeltaGCTime     | minDeltaGCTime     PS MarkSweep         | 1                  | 1                  | 1                  | 0.018s             | 0.018s             | 0.018s             PS Scavenge          | 1                  | 1                  | 1                  | 0.009s             | 0.009s             | 0.009s             2023-10-10 17:04:36.776 [job-0] INFO  JobContainer - PerfTrace not enable!
2023-10-10 17:04:36.776 [job-0] INFO  StandAloneJobContainerCommunicator - Total 10000 records, 408894 bytes | Speed 39.93KB/s, 1000 records/s | Error 0 records, 0 bytes |  All Task WaitWriterTime 0.010s |  All Task WaitReaderTime 0.000s | Percentage 100.00%
2023-10-10 17:04:36.777 [job-0] INFO  JobContainer - 
任务启动时刻                    : 2023-10-10 17:04:26
任务结束时刻                    : 2023-10-10 17:04:36
任务总计耗时                    :                 10s
任务平均流量                    :           39.93KB/s
记录写入速度                    :           1000rec/s
读出记录总数                    :               10000
读写失败总数                    :                   0

执行成功，检查Oracle数据库查看数据是否同步成功

SYS@orcl> col name for a50
SYS@orcl> select * from shuaige.httab where rownum < 10;ID NAME 						     			AGE
---------- -------------------------------------------- ----------16809 5f19fd4c4f65352c79318089634ee459			    680916810 039773606c114ae52af27fd6bb2603e7			    681016811 9dd5a56b63e6192e6f091c3dfd600f50			    681116812 564147edcc95fe83af7b5bf2cc42268a			    681216813 1dc07f8e48bd6baf258c0a341ad8753a			    681316814 7569553c59834aff4e0c78c8a5292a9f			    681416815 b93d01952acc42d345e06a062166693f			    681516816 d60332e3fef6292da8c98d963435ec9e			    681616817 9f3271cd20b8e473ae9e98c4b258f45d			    68179 rows selected.

数据传输成功，如果在执行json文件报错，一定要先看好json文件格式，用户密码大小写，[ ]，" “等，“channel"参数至少为“1”，“preSql”:[””] SQL条件可以不加，但是有主键冲突的行会报错传输失败。

http://www.lryc.cn/news/192788.html

相关文章：

  • 【PyTorch2 之027】在 PyTorch 中的R-CNN、Fast R-CNN和 Faster R-CNN
  • C++学习——C++函数的编译、成员函数的调用、this指针详解
  • Pulsar Manager和dashboard部署和启用认证
  • K8S环境搭建
  • 常用的软件项目管理工具一览
  • 关于网络协议的若干问题(五)
  • TensorFlow入门(十七、神经元的拟合原理)
  • VSCODE配置C和C++
  • 位于同一子网下的ip在子网掩码配置错误的情况下如何进行通信（wireshrak抓包分析）
  • Dockerfile镜像实战
  • 企业如何选择安全又稳定的文件传输协议
  • Linux Kernel 4.13 RC6发布：正式版9月3日发布
  • C++学习——C++中const的新花样
  • 【Linux环境搭建】五、Linux(CentOS7)编译源码安装Subversion
  • 微信小程序入门讲解【超详细】
  • AtCoder ABC239G 最小割集
  • Simple RPC - 01 框架原理及总体架构初探
  • VScode运行C/C++
  • #智能车项目（三）串口初始化
  • 网络通信错误代码列表 HTTP 、FTP
  • 最新开源ThinkPHP6框架云梦卡社区系统源码/亲测可用（全新开发）
  • [ROS2系列] ubuntu 20.04测试rtabmap
  • 【Java学习之道】JavaFx 框架与组件介绍
  • Windows bat 脚本设计-开机自启动服务的方法、bat 调用另外的 bat 脚本 -没有java环境也能运行jar，在不安装jdk下如何运行jar包
  • zabbix触发器与动作
  • 华纳云：Nginx服务器可视化配置问题怎么解决
  • C指针与一维二维数组、数组指针与指针数组、函数指针_数组的理解使用
  • 安装运行vue-element-admin的报错问题-解决办法
  • 高数笔记03：几何、物理应用
  • js + selenium 获取chatgpt的accessToken