DB2 may report out of memory (OOM) errors even if there is
sufficient cached database memory available.
OOMs could probably have been avoided altogether if the request
for Instance Memory (requestMemory) tried a bit harder to
reclaim cached memory in the other consumers.
For example, db2diag.log shows:
2017-07-17-10.12.13.088021+000 I929503368E1120 LEVEL:
Warning
PID : 10598 TID : 46970232104704 PROC :
db2sysc 21
INSTANCE: db2inst1 NODE : 021 DB :
BLUDB
APPHDL : 0-27126 APPID:
10.113.34.46.62106.170717100710
AUTHID : ELTUSER HOSTNAME: host
EDUID : 369804 EDUNAME: db2agntcol (BLUDB) 21
FUNCTION: DB2 UDB, SQO Memory Management,
SqloMemController::requestMemory, probe:50
MESSAGE : ZRC=0x8B0F0000=-1961951232=SQLO_NOMEM "No Memory
Available"
DIA8300C A memory heap error has occurred.
DATA #1 : String, 36 bytes
OOM - Instance memory request failed
DATA #2 : String, 35 bytes
Logging disabled until next success
DATA #3 : unsigned integer, 8 bytes
34406400
DATA #4 : unsigned integer, 8 bytes
10
DATA #5 : String, 13 bytes
APPL-BLUDB
DATA #6 : unsigned integer, 8 bytes
1439105024
DATA #7 : unsigned integer, 8 bytes
0
DATA #8 : unsigned integer, 8 bytes
32124362752
DATA #9 : unsigned integer, 8 bytes
32131727360
DATA #10: unsigned integer, 8 bytes
3226992640
...
APPL-BLUDB - Current size : 1405376 KB, HWM : 2827136 KB,
Cached : 0 KB
DBMS-db2inst1 - Current size : 156096 KB, HWM : 162880 KB,
Cached : 0 KB
FMP_RESOURCES - Current size : 768 KB, HWM : 22528 KB, Cached :
0 KB
PRIVATE - Current size : 167040 KB, HWM : 340608 KB, Cached : 0
KB
FCM_RESOURCES - Current size : 104312 KB, HWM : 197376 KB,
Cached : 0 KB
DB-BLUDB - Current size : 29537856 KB, HWM : 30081408 KB,
Cached : 3184960 KB
The typical db2diag.log entry will show "10" for the
retryCounter value, which is DATA#4:
The request size is DATA #3, 34MB
APPL memory size is DATA #6, 1.4GB, not unreasonable. DATA #7
is its cached amount, 0.
Limit is DATA #9, 32GB, DATA #8 shows usage we are at the limit
But DATA #10 shows 3.2GB Cached, plenty to satisfy a 32MB
request, and from subsequent entries we could see it´s all in
Database Memory.
So we can conclude we were hitting contention trying to reduce
(the cached/overflow amount) in database memory set.
It must be busy due to sort memory growth and/or decrease across
the various sort pools. |
