|
Problem IC82262
|
Status: Closed |
ENOMEM ERROR ON DB2ACD PROCESS COULD LEAD TO NON-FATAL ASSERTION FAILED |
| product: |
DB2 FOR LUW / DB2FORLUW / 950 - DB2 |
| Problem description: |
If db2acd process hit a ENOMEM issue, NON-FATAL ASSERTION
FAILED messages might be dumped in db2diag.log
Symptoms in the db2diag.log file :
1/ the ENOMEM error on db2acd:
011-11-27-17.29.24.858867+060 I18856096A2007 LEVEL: Error
(OS)
PID : 27000988 TID : 1926 PROC : db2acd 0
INSTANCE: db2xx NODE : 000 APPID
:*LOCAL.db2xx.111127162823
EDUID : 1926 EDUNAME: db2acd 0
FUNCTION: DB2 Common, OSSe, ossErrorMemoryAnalysis, probe:100
CALLED : OS, -, malloc
OSERR : ENOMEM (12) "Not enough space"
DATA #1 : String, 72 bytes
A total of 5 analysis will be performed :
- ulimit info
- Memory info
DATA #2 : String, 381 bytes
Current process limits (unit in bytes except for nofiles) :
mem (S/H) = unlimited / unlimited
core (S/H) = 1073741312 / unlimited
cpu (S/H) = unlimited / unlimited
data (S/H) = 134217728 / unlimited
fsize (S/H) = 1073741312 / 1073741312
nofiles (S/H) = unlimited / unlimited
stack (S/H) = 33554432 / 4294967296
rss (S/H) = 33554432 / unlimited
...
2/ the NON-FATAL ASSERTION FAILED messages:
2011-11-30-06.50.00.856051+060 I1437A1052 LEVEL: Severe
PID : 28836048 TID : 1993 PROC : db2acd
0
INSTANCE: db2pxx NODE : 000 APPID :
*LOCAL.db2pxx.111129063131
EDUID : 1993 EDUNAME: db2acd 0
FUNCTION: DB2 UDB, global services, sqlzAssertFailed, probe:10
DATA #1 : String, 196 bytes
NON-FATAL ASSERTION FAILED!!!
ASSERTION EXPRESSION: Freeing freed memory found at:
SOURCE FILENAME:
/view/db2_v97fp3_aix64_special_25384_special_25491/vbs/engn/atm/
inc/atmStats.h
LINE NUMBER: 181
CALLSTCK:
[0] 0x09000000026646F4 pdLog + 0xA8
[1] 0x09000000029424B4 sqlzAssertFailedValist + 0x1AC
[2] 0x090000000631CB18 ossAssertFailureAction + 0x138
[3] 0x090000000633DF2C _ossMemAllocCheck + 0x68C
[4] 0x090000000633E098 _ossMemFree + 0x58
[5] 0x0900000010EE7CE8 __dt__8AtmTableFv + 0x108
[6] 0x0900000010EE383C process_tables__3AtmFv + 0x10DC
[7] 0x0900000010EE140C iter__3AtmFv + 0x2AC
[8] 0x0900000010EDF45C run__3AtmFPCc + 0x1BC
[9] 0x0900000010EDE6EC db2HmonEvalStats + 0x42C
2011-11-30-06.50.00.858207+060 I2490A1070 LEVEL: Severe
PID : 28836048 TID : 1993 PROC : db2acd
0
INSTANCE: db2pxx NODE : 000
APPID : *LOCAL.db2pxx.111129063131
EDUID : 1993 EDUNAME: db2acd 0
FUNCTION: DB2 UDB, global services, sqlzAssertFailed, probe:10
DATA #1 : String, 214 bytes
NON-FATAL ASSERTION FAILED!!!
ASSERTION EXPRESSION: Invalid block eye-catcher (0xDEAD055E)
found at:
SOURCE FILENAME:
/view/db2_v97fp3_aix64_special_25384_special_25491/vbs/engn/atm/
inc/atmStats.h
LINE NUMBER: 181
CALLSTCK:
[0] 0x09000000026646F4 pdLog + 0xA8
[1] 0x09000000029424B4 sqlzAssertFailedValist + 0x1AC
[2] 0x090000000631CB18 ossAssertFailureAction + 0x138
[3] 0x090000000633DFB8 _ossMemAllocCheck + 0x718
[4] 0x090000000633E098 _ossMemFree + 0x58
[5] 0x0900000010EE7D94 __dt__8AtmTableFv + 0x1B4
[6] 0x0900000010EE383C process_tables__3AtmFv + 0x10DC
[7] 0x0900000010EE140C iter__3AtmFv + 0x2AC
[8] 0x0900000010EDF45C run__3AtmFPCc + 0x1BC
[9] 0x0900000010EDE6EC db2HmonEvalStats + 0x42C
There can be multiple reasons for the out of memory condition.
One common reason is that the db2acd memory requirement exceeds
the data ulimit of the userid which performed the db2start
operation. In the example above, the ossErrorMemoryAnalysis
probe indicates the db2acd process was running with a data
ulimit of 128MB. It is likely that the out of memory condition
experienced by db2acd will not occur if the instance is
restarted by an userid with sufficient data ulimit.
Depending on the operating system involved, the diaglog messages
above might differ, and the facilities to control resource
limits might also differ. |
| Problem Summary: |
****************************************************************
* USERS AFFECTED: *
* ALL *
****************************************************************
* PROBLEM DESCRIPTION: *
* If db2acd process hit a ENOMEM issue, NON-FATAL ASSERTION *
* FAILED messages might be dumped in db2diag.log *
* *
* Symptoms in the db2diag.log file : *
* *
* 1/ the ENOMEM error on db2acd: *
* *
* 011-11-27-17.29.24.858867+060 I18856096A2007 LEVEL: *
* Error *
* (OS) *
* PID : 27000988 TID : 1926 PROC : *
* db2acd 0 *
* INSTANCE: db2xx NODE : 000 APPID *
* :*LOCAL.db2xx.111127162823 *
* EDUID : 1926 EDUNAME: db2acd 0 *
* FUNCTION: DB2 Common, OSSe, ossErrorMemoryAnalysis, *
* probe:100 *
* CALLED : OS, -, malloc *
* OSERR : ENOMEM (12) "Not enough space" *
* DATA #1 : String, 72 bytes *
* A total of 5 analysis will be performed : *
* - ulimit info *
* - Memory info *
* DATA #2 : String, 381 bytes *
* Current process limits (unit in bytes except for nofiles) : *
* mem (S/H) = unlimited / unlimited *
* core (S/H) = 1073741312 / unlimited *
* cpu (S/H) = unlimited / unlimited *
* data (S/H) = 134217728 / unlimited *
* fsize (S/H) = 1073741312 / 1073741312 *
* nofiles (S/H) = unlimited / unlimited *
* stack (S/H) = 33554432 / 4294967296 *
* rss (S/H) = 33554432 / unlimited *
* ... *
* *
* 2/ the NON-FATAL ASSERTION FAILED messages: *
* *
* 2011-11-30-06.50.00.856051+060 I1437A1052 LEVEL: *
* Severe *
* PID : 28836048 TID : 1993 PROC : *
* db2acd *
* 0 *
* INSTANCE: db2pxx NODE : 000 APPID : *
* *LOCAL.db2pxx.111129063131 *
* EDUID : 1993 EDUNAME: db2acd 0 *
* FUNCTION: DB2 UDB, global services, sqlzAssertFailed, *
* probe:10 *
* DATA #1 : String, 196 bytes *
* NON-FATAL ASSERTION FAILED!!! *
* ASSERTION EXPRESSION: Freeing freed memory found at: *
* SOURCE FILENAME: *
* /view/db2_v97fp3_aix64_special_25384_special_25491/vbs/engn/ *
* atm/ *
* inc/atmStats.h *
* LINE NUMBER: 181 *
* CALLSTCK: *
* [0] 0x09000000026646F4 pdLog + 0xA8 *
* [1] 0x09000000029424B4 sqlzAssertFailedValist + 0x1AC *
* [2] 0x090000000631CB18 ossAssertFailureAction + 0x138 *
* [3] 0x090000000633DF2C _ossMemAllocCheck + 0x68C *
* [4] 0x090000000633E098 _ossMemFree + 0x58 *
* [5] 0x0900000010EE7CE8 __dt__8AtmTableFv + 0x108 *
* [6] 0x0900000010EE383C process_tables__3AtmFv + 0x10DC *
* [7] 0x0900000010EE140C iter__3AtmFv + 0x2AC *
* [8] 0x0900000010EDF45C run__3AtmFPCc + 0x1BC *
* [9] 0x0900000010EDE6EC db2HmonEvalStats + 0x42C *
* *
* 2011-11-30-06.50.00.858207+060 I2490A1070 LEVEL: *
* Severe *
* PID : 28836048 TID : 1993 PROC : *
* db2acd *
* 0 *
* INSTANCE: db2pxx NODE : 000 *
* APPID : *LOCAL.db2pxx.111129063131 *
* EDUID : 1993 EDUNAME: db2acd 0 *
* FUNCTION: DB2 UDB, global services, sqlzAssertFailed, *
* probe:10 *
* DATA #1 : String, 214 bytes *
* NON-FATAL ASSERTION FAILED!!! *
* ASSERTION EXPRESSION: Invalid block eye-catcher (0xDEAD055E) *
* found at: *
* SOURCE FILENAME: *
* /view/db2_v97fp3_aix64_special_25384_special_25491/vbs/engn/ *
* atm/ *
* inc/atmStats.h *
* LINE NUMBER: 181 *
* CALLSTCK: *
* [0] 0x09000000026646F4 pdLog + 0xA8 *
* [1] 0x09000000029424B4 sqlzAssertFailedValist + 0x1AC *
* [2] 0x090000000631CB18 ossAssertFailureAction + 0x138 *
* [3] 0x090000000633DFB8 _ossMemAllocCheck + 0x718 *
* [4] 0x090000000633E098 _ossMemFree + 0x58 *
* [5] 0x0900000010EE7D94 __dt__8AtmTableFv + 0x1B4 *
* [6] 0x0900000010EE383C process_tables__3AtmFv + 0x10DC *
* [7] 0x0900000010EE140C iter__3AtmFv + 0x2AC *
* [8] 0x0900000010EDF45C run__3AtmFPCc + 0x1BC *
* [9] 0x0900000010EDE6EC db2HmonEvalStats + 0x42C *
* *
* There can be multiple reasons for the out of memory *
* condition. *
* One common reason is that the db2acd memory requirement *
* exceeds *
* the data ulimit of the userid which performed the db2start *
* operation. In the example above, the ossErrorMemoryAnalysis *
* probe indicates the db2acd process was running with a data *
* ulimit of 128MB. It is likely that the out of memory *
* condition *
* experienced by db2acd will not occur if the instance is *
* restarted by an userid with sufficient data ulimit. *
* *
* Depending on the operating system involved, the diaglog *
* messages *
* above might differ, and the facilities to control resource *
* limits might also differ. *
****************************************************************
* RECOMMENDATION: *
* Upgrade to DB2 UDB version 9.5 fixpack 10. *
**************************************************************** |
| Local Fix: |
Examine the DB2 diagnostic log for messages associated with
ENOMEM errors logged by the db2acd process. On operating
systems which support user resource limits, restart the DB2
instance from a user account whose data ulimit is sufficient,
such as unlimited. |
| Solution |
Problem was first fixed in DB2 UDB Version 9.5 Fix Pack 10. |
| Workaround |
not known / see Local fix |
| Timestamps |
Date - problem reported :
Date - problem closed :
Date - last modified :
| 26.03.2012
20.10.2012
20.10.2012 |
| Problem solved at the following versions (IBM BugInfos) |
9.5.FP10
|
| Problem solved according to the fixlist(s) of the following version(s) |
| 9.5.0.10
|
 |
