DB2 - Problem description
Problem IT40335 | Status: Closed |
HADR STANDBY IS SHUTDOWN DUE TO NOT FINDING A LOG FILE RETRIEVED FROM ARCHIVE | |
product: | |
DB2 FOR LUW / DB2FORLUW / B50 - DB2 | |
Problem description: | |
This error only exists in 11.5.6 and 11.5.7. On HADR standby startup, standby scans log files in active log path to see if log files needed for recovery exist. However, this scan may delete log file that was previously retrieved from archive, which may cause subsequent standby replay fail to find it. In that case, replay cannot continue as the log file is needed for recovery. Eventually, standby will be forced to shut down due to replay error. If this were to happen, on the standby one may see messages in the db2diag.log as follows: 1. Standby retrieved log file (S0000002.LOG) from archive: 2022-02-17-00.27.47.956835-300 I739230E579 LEVEL: Info PID : 26608 TID : 139992252802816 PROC : db2sysc INSTANCE: myinstance NODE : 000 DB : MYDB HOSTNAME: myhost EDUID : 82 EDUNAME: db2logmgr (MYDB) FUNCTION: DB2 UDB, data protection services, sqpLogMgrEdu::sqlpgRetrieveLogFile, probe:4148 DATA #1 : Completed retrieve for log file S0000001.LOG on chain 1 to /home/myinstance/myinstance/NODE0000/SQL00001/LOGSTREAM0000/LOGS TREAM0000/. The newly retrieved log will be verified. 2. Standby replay cannot find log file S0000002.LOG: 2022-02-17-00.28.37.253042-300 I1629509E537 LEVEL: Info PID : 4946 TID : 140631208879872 PROC : db2sysc INSTANCE: myinstance NODE : 000 DB : MYDB HOSTNAME: myhost EDUID : 28 EDUNAME: db2lfr.0 (MYDB) FUNCTION: DB2 UDB, data protection services, sqlpgOpenLogExtent, probe:40 MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found." DIA8411C A file "" could not be found. DATA #1 : Error opening database log extent 2. 2022-02-17-00.28.37.332256-300 I1634260E596 LEVEL: Error PID : 4946 TID : 140631208879872 PROC : db2sysc INSTANCE: myinstance NODE : 000 DB : MYDB HOSTNAME: myhost EDUID : 28 EDUNAME: db2lfr.0 (MYDB) FUNCTION: DB2 UDB, recovery manager, sqlplfrDoScanNext, probe:1100 MESSAGE : hit EOL for hadr DATA #1 : SQLPLFR_SCAN_ID, PD_TYPE_SQLPLFR_SCAN_ID, 8 bytes LFR Scan Num = 5 LFR Scan Caller's EDUID = 79 DATA #2 : Hex integer, 8 bytes 0x0000000004031008 DATA #3 : unsigned integer, 4 bytes 2249129994 3. Standby replay failed due to missing log file: 2022-02-17-00.28.37.601962-300 I1669869E1100 LEVEL: Info PID : 4946 TID : 140631234045696 PROC : db2sysc INSTANCE: myinstance NODE : 000 DB : MYDB APPHDL : 0-8 APPID: *LOCAL.DB2.220217052837 HOSTNAME: myhost EDUID : 22 EDUNAME: db2agent (MYDB) FUNCTION: DB2 UDB, recovery manager, sqlpReplayMaster, probe:2460 MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found." DIA8411C A file "" could not be found. DATA #1 : String, 88 bytes Replay master encoutered some error. Try to restart standby to handle recoverable error. DATA #2 : unsigned integer, 8 bytes 0 DATA #3 : SQLCA, PD_DB2_TYPE_SQLCA, 136 bytes sqlcaid : SQLCA sqlcabc: 136 sqlcode: -1042 sqlerrml: 0 sqlerrmc: sqlerrp : sqlpRepl sqlerrd : (1) 0x00000000 (2) 0x00000000 (3) 0x00000000 (4) 0x00000000 (5) 0x00000000 (6) 0x00000000 sqlwarn : (1) (2) (3) (4) (5) (6) (7) (8) (9) (10) (11) sqlstate: 4. Standby get forced shut down due to replay failure: 2022-02-17-00.28.44.160556-300 I2122852E622 LEVEL: Severe PID : 4946 TID : 140628885235456 PROC : db2sysc INSTANCE: myinstance NODE : 000 DB : MYDB APPHDL : 0-14 APPID: *LOCAL.DB2.220217052843 HOSTNAME: myhost EDUID : 163 EDUNAME: db2agent (MYDB) FUNCTION: DB2 UDB, recovery manager, sqlpReplayMaster, probe:2500 MESSAGE : ZRC=0x860F000A=-2045837302=SQLO_FNEX "File not found." DIA8411C A file "" could not be found. DATA #1 : Fatal error during HADR replay. Forcing the database to shut down. 2022-02-17-00.28.44.161403-300 I2123475E1339 LEVEL: Info PID : 4946 TID : 140628889429760 PROC : db2sysc INSTANCE: myinstance NODE : 000 DB : MYDB APPHDL : 0-15 APPID: *LOCAL.DB2.220217052844 HOSTNAME: myhost EDUID : 204 EDUNAME: db2agent (MYDB) FUNCTION: DB2 UDB, base sys utilities, sqeLocalDatabase::ForceDBShutdown, probe:17516 MESSAGE : Regular agent EDU doing ForceDBShutdown. Force DB shutdown agent ID is: DATA #1 : APPHDL, PD_TYPE_SQLZ_APPHDL, 4 bytes 0-15 DATA #2 : sqeApplication_acbInfo, PD_TYPE_sqeApplication_acbInfo, 4 bytes x0 CALLSTCK: (Static functions may not be resolved correctly, as they are resolved to the nearest symbol) [0] 0x00007FE7592FE7BF _ZN16sqeLocalDatabase15ForceDBShutdownEi + 0x18F [1] 0x00007FE752FEAB02 _ZN21HDR_FORCE_DB_SHUTDOWN16hdrExecOperationEP8sqeAgent + 0x12 [2] 0x00007FE752FEB27C _ZN18HDR_ICOORD_COMMAND19hdrICoordCmdExecuteEP8sqeAgent + 0xDC [3] 0x00007FE75914B9EF _Z26sqleIndCoordProcessRequestP8sqeAgent + 0x1BDF [4] 0x00007FE75917DC78 _ZN8sqeAgent6RunEDUEv + 0x538 [5] 0x00007FE75D445D94 _ZN9sqzEDUObj9EDUDriverEv + 0x1A4 [6] 0x00007FE75B6D05D9 sqloEDUEntry + 0x2A9 [7] 0x00007FE762317EA5 /lib64/libpthread.so.0 + 0x7EA5 [8] 0x00007FE74FA518DD clone + 0x6D | |
Problem Summary: | |
**************************************************************** * USERS AFFECTED: * * All * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Upgrade to Db2 v11.5.7.1 * **************************************************************** | |
Local Fix: | |
Solution | |
Workaround | |
**************************************************************** * USERS AFFECTED: * * All * **************************************************************** * PROBLEM DESCRIPTION: * * See Error Description * **************************************************************** * RECOMMENDATION: * * Upgrade to Db2 v11.5.7.1 * **************************************************************** | |
Comment | |
Fix included in Db2 v11.5.7.1 | |
Timestamps | |
Date - problem reported : Date - problem closed : Date - last modified : | 18.03.2022 05.04.2022 05.04.2022 |
Problem solved at the following versions (IBM BugInfos) | |
Problem solved according to the fixlist(s) of the following version(s) |