当前位置：首页 > news >正文

ORA-00600: internal error code, arguments: [krse_arc_source_init.1], [4], [2]

news 2025/7/22 8:42:24

有套3节点的rac，数据库版本11204，15号凌晨0点 3个实例同时宕机，手动启动节点二，节点三上实例，正常启动，手动启动节点一实例，告警如下:

节点一

SYS@orcl1> startup;
ORACLE instance started.

Total System Global Area 1.2827E+10 bytes
Fixed Size 2265224 bytes
Variable Size 5133832056 bytes
Database Buffers 7650410496 bytes
Redo Buffers 40861696 bytes
Database mounted.
ORA-00600: internal error code, arguments: [krse_arc_source_init.1], [4], [2],[144971], [1], [0], [0], [0], [], [], [], []

查看3个节点的日志，如下

Tue Jul 15 00:00:28 2025
Archived Log entry 469783 added for thread 1 sequence 196976 ID 0xceb9aaf7 dest 1:
Tue Jul 15 00:05:22 2025
Dumping diagnostic data in directory=[cdmp_20250715000522], requested by (instance=2, osid=1205 (CKPT)), summary=[abnormal instance termination].
Tue Jul 15 00:05:28 2025
Reconfiguration started (old inc 38, new inc 40)
List of instances:
1 3 (myinst: 1)
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Tue Jul 15 00:05:28 2025
* domain 0 not valid according to instance 3
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Jul 15 00:05:28 2025
Tue Jul 15 00:05:28 2025
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 2: 2 GCS shadows cancelled, 2 closed, 0 Xw survived
Tue Jul 15 00:05:28 2025
LMS 0: 1 GCS shadows cancelled, 1 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Tue Jul 15 00:05:32 2025
Setting Resource Manager plan SCHEDULER[0x32D9]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Tue Jul 15 00:05:35 2025
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Tue Jul 15 00:05:35 2025
minact-scn: master found reconf/inst-rec before recscn scan old-inc#:40 new-inc#:40
minact-scn: master continuing after IR
minact-scn: Master considers inst:2 dead
Tue Jul 15 00:05:45 2025
Reconfiguration started (old inc 40, new inc 42)
List of instances:
1 2 3 (myinst: 1)
Global Resource Directory frozen
Communication channels reestablished
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Jul 15 00:05:45 2025
Tue Jul 15 00:05:45 2025
LMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Tue Jul 15 00:05:45 2025
LMS 2: 1 GCS shadows cancelled, 0 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete
Tue Jul 15 00:05:47 2025
minact-scn: Master returning as live inst:2 has inc# mismatch instinc:0 cur:42 errcnt:0
Tue Jul 15 00:06:03 2025
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Tue Jul 15 00:06:04 2025
********************* ATTENTION: ********************
The controlfile header block returned by the OS
has a sequence number that is too old.
The controlfile might be corrupted.
PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE
without following the steps below.
RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE
TO THE DATABASE, if the controlfile is truly corrupted.
In order to re-start the instance safely,
please do the following:
(1) Save all copies of the controlfile for later
analysis and contact your OS vendor and Oracle support.
(2) Mount the instance and issue:
ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
(3) Unmount the instance.
(4) Use the script in the trace file to
RE-CREATE THE CONTROLFILE and open the database.
*****************************************************
USER (ospid: 33777): terminating the instance
Tue Jul 15 00:20:35 2025
System state dump requested by (instance=1, osid=33777), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_diag_8249_20250715002035.trc
Tue Jul 15 00:20:36 2025
ORA-1092 : opitsk aborting process
Tue Jul 15 00:20:36 2025
License high water mark = 181
Instance terminated by USER, pid = 33777
USER (ospid: 559): terminating the instance
Instance terminated by USER, pid = 559
Tue Jul 15 00:20:42 2025
Starting ORACLE instance (normal)

节点二

Tue Jul 15 00:00:27 2025
Archived Log entry 469782 added for thread 2 sequence 144968 ID 0xceb9aaf7 dest 1:
Tue Jul 15 00:05:22 2025
Error: Controlfile sequence number in file header is different from the one in memory
Please check that the correct mount options are used if controlfile is located on NFS
CKPT (ospid: 1205): terminating the instance
Tue Jul 15 00:05:22 2025
System state dump requested by (instance=2, osid=1205 (CKPT)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_diag_1135_20250715000522.trc
Tue Jul 15 00:05:22 2025
ORA-1092 : opitsk aborting process
Instance terminated by CKPT, pid = 1205
Tue Jul 15 00:05:34 2025
Starting ORACLE instance (normal)

节点三：

Tue Jul 15 00:00:28 2025
Thread 3 advanced to log sequence 127839 (LGWR switch)
Current log# 9 seq# 127839 mem# 0: +DATA/orcl/onlinelog/group_9.308.1092668279
Current log# 9 seq# 127839 mem# 1: +FRA/orcl/onlinelog/group_9.290.1092668281
Tue Jul 15 00:00:28 2025
Archived Log entry 469784 added for thread 3 sequence 127838 ID 0xceb9aaf7 dest 1:
Tue Jul 15 00:05:22 2025
Dumping diagnostic data in directory=[cdmp_20250715000522], requested by (instance=2, osid=1205 (CKPT)), summary=[abnormal instance termination].
Tue Jul 15 00:05:28 2025
Reconfiguration started (old inc 38, new inc 40)
List of instances:
1 3 (myinst: 3)
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
Tue Jul 15 00:05:28 2025
* domain 0 valid = 0 according to instance 1
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
Tue Jul 15 00:05:28 2025
LMS 1: 0 GCS shadows cancelled, 0 closed, 0 Xw survived
Tue Jul 15 00:05:28 2025
LMS 0: 2 GCS shadows cancelled, 1 closed, 0 Xw survived
Tue Jul 15 00:05:28 2025
LMS 2: 9 GCS shadows cancelled, 1 closed, 0 Xw survived
Set master node info
Submitted all remote-enqueue requests

....................

Tue Jul 15 00:20:46 2025
Setting Resource Manager plan SCHEDULER[0x32D9]:DEFAULT_MAINTENANCE_PLAN via scheduler window
Setting Resource Manager plan DEFAULT_MAINTENANCE_PLAN via parameter
Completed redo application of 3.02MB
Completed instance recovery at
Thread 1: logseq 196977, block 243843, scn 19534278073
509 data blocks read, 584 data blocks written, 8301 redo k-bytes read
Thread 1 advanced to log sequence 196978 (thread recovery)
Redo thread 1 internally disabled at seq 196978 (SMON)
minact-scn: master continuing after IR
Tue Jul 15 00:20:48 2025

Tue Jul 15 01:02:31 2025
Error: Controlfile sequence number in file header is different from the one in memory
Please check that the correct mount options are used if controlfile is located on NFS
Tue Jul 15 01:02:31 2025
System state dump requested by (instance=2, osid=11887 (SMON)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl2/trace/orcl2_diag_11821_20250715010231.trc
SMON (ospid: 11887): terminating the instance
Instance terminated by SMON, pid = 11887
Tue Jul 15 01:02:41 2025
Starting ORACLE instance (normal)
********************* ATTENTION: ********************
The controlfile header block returned by the OS
has a sequence number that is too old.
The controlfile might be corrupted.
PLEASE DO NOT ATTEMPT TO START UP THE INSTANCE
without following the steps below.
RE-STARTING THE INSTANCE CAN CAUSE SERIOUS DAMAGE
TO THE DATABASE, if the controlfile is truly corrupted.
In order to re-start the instance safely,
please do the following:
(1) Save all copies of the controlfile for later
analysis and contact your OS vendor and Oracle support.
(2) Mount the instance and issue:
ALTER DATABASE BACKUP CONTROLFILE TO TRACE;
(3) Unmount the instance.
(4) Use the script in the trace file to
RE-CREATE THE CONTROLFILE and open the database.
*****************************************************
ARC0 (ospid: 30214): terminating the instance
Tue Jul 15 00:20:50 2025
System state dump requested by (instance=3, osid=30214 (ARC0)), summary=[abnormal instance termination].
System State dumped to trace file /u01/app/oracle/diag/rdbms/orcl/orcl3/trace/orcl3_diag_30079_20250715002050.trc
Tue Jul 15 00:20:50 2025
ORA-1092 : opitsk aborting process
Instance terminated by ARC0, pid = 30214
Tue Jul 15 00:21:01 2025
Starting ORACLE instance (normal)