When a storage path is added to an automatic storage database,
and an alter tablespace issued to rebalance the conatiners, DB2
may erroneously conclude that a new container is going to fit
into a new path to be used for rebalancing. However, in reality
the path does not have enough disk space to accommodate the new
container.
Following is the sequence of steps, that may lead to this issue:
db2 "ALTER DATABASE SAMPLE ADD STORAGE ON '/db2/data/sample/a3'"
db2 "ALTER DATABASE SAMPLE ADD STORAGE ON '/db2/data/sample/a3'"
db2 connect to SAMPLE
db2 "ALTER TABLESPACE TBSP1 REBALANCE"
Isuuing the alter database command mentioned above twice, causes
DB2 to show 2 identical automatic storage paths, as can be seen
in the snapshot below:
Number of automatic storage paths = 4
Automatic storage path = /db2/data/sample/a1
Node number = 0
State = In Use
Automatic storage path = /db2/data/sample/a2
Node number = 0
State = In Use
Automatic storage path = /db2/data/sample/a3
Node number = 0
State = Not In Use
Automatic storage path = /db2/data/sample/a3
Node number = 0
State = Not In Use
After the last ALTER TABLEPSACE command is issued, since there
is not enough space on the storage path just added to accomodate
the container, the retry logic in DB2 will keep on attempting to
create the new container, during which time the table space will
be unaccessible to the end user, who may perceive this situation
as a hang. Eventually, SQL2094W reason code 2 will be returned,
but this may happen a long time (e.g. tens of minutes) after
issuing the ALTER TABLE ... REBALANCE command. During this time,
the following error messages might be dumped in the db2diag.log:
2010-10-19-22.08.57.906941+120 I230832A2923 LEVEL: Error
(OS)
PID : 10293 TID : 81 PROC : db2sysc
0
INSTANCE: db2usrp2 NODE : 000 DB : SAMPLE
APPHDL : 0-10932 APPID:
*LOCAL.db2usrp2.101019193340
AUTHID : DB2USRP2
EDUID : 81 EDUNAME: db2agent (ARNP) 0
FUNCTION: DB2 Common, OSSe, ossErrorIOAnalysis, probe:100
CALLED : OS, -, write
OSERR : ENOSPC (28) "No space left on device"
DATA #1 : String, 146 bytes
A total of 5 analysis will be performed :
- User info
- ulimit info
- Target file info
- I/O attempt
- File system
Target file handle = 58
DATA #2 : String, 184 bytes
Real user ID of current process = 212
Effective user ID of current process = 212
Real group ID of current process = 212
Effective group ID of current process = 212
DATA #3 : String, 353 bytes
Current process limits (unit in bytes except for nofiles) :
mem (S/H) = unlimited / unlimited
core (S/H) = unlimited / unlimited
cpu (S/H) = unlimited / unlimited
data (S/H) = unlimited / unlimited
fsize (S/H) = unlimited / unlimited
nofiles (S/H) = 65536 / 65536
stack (S/H) = 8388608 / unlimited
rss (S/H) = 0 / 0
DATA #4 : String, 263 bytes
Target File Information :
Size = 4185948160
Link = No
Reference path = N/A
Type = 0x8000
Permissions = rw-------
UID = 212
GID = 212
Last modified time = 1287518937
DATA #5 : String, 33 bytes
I/O attempt not implemented yet.
DATA #6 : String, 397 bytes
File System Information of the target file :
Block size = 1024 bytes
Total size = 26430790656 bytes
Free size = 0 bytes
Total # of inodes = 3110016
FS name = /dev/md/db2p2/dsk/d6
Mount point = /db2/data/arnp/a3
FSID = 22298630
FS type name = ufs
DIO/CIO mount opt = None
Device type = N/A
FS type = 0x9
CALLSTCK:
[0] 0xFFFFFFFF7EDBB198 ossLogSysRC + 0x3EC
[1] 0xFFFFFFFF7EDACCE4 ossErrorNameMapSystem + 0x1AFC
[2] 0xFFFFFFFF791DF6D4 sqloSystemErrorHandler + 0x9DC
[3] 0xFFFFFFFF7924971C sqloSetFileSize + 0x1C2C
[4] 0xFFFFFFFF7924A650 sqloSetFileSize + 0x2B60
[5] 0xFFFFFFFF79248A64 sqloSetFileSize + 0xF74
[6] 0xFFFFFFFF777A5EF8
__1cbHsqlbServiceAddDMSContainerRequest6FpnOSQdDLB_pfParIoCb_pnM
SQdDLB_P
OOL_CB_pnQSQdDLB_POOLCONT_CB_pnMSQdDLB_GLOBALS__i_ + 0x5E0
[7] 0xFFFFFFFF778F129C
__1cbAsqlbDMSAddContainerRequest6FpnMSQdDLB_POOL_CB_pnQSQdDLB_PO
OLCONT_C
B_pnMSQdDLB_GLOBALS_pnOSQdDLB_pfParIoCb_bL_i_ + 0x17D4
[8] 0xFFFFFFFF7791578C
__1cRsqlbDMSCreatePool6FpnMSQdDLB_POOL_CB_ipnQSQdDLB_POOLCONT_CB
_pnMSQdD
LB_GLOBALS__i_ + 0x2024
[9] 0xFFFFFFFF7791740C
__1cbHsqlbPerformDMSContainerOperations6FpnMSQdDLB_POOL_CB_1LpnQ
SQdDLB_P
OOLCONT_CB_bpnbASQdDLB_AS_CONT_AND_PATH_INFO_pnMSQdDLB_ + 0x108C
The problem only happens for containers whose new size will be
larger than the maximum value of the unsigned integer C type,
i.e. for containers larger than 4,294,967,295 bytes. This is a
integer overflow problem, where DB2 miscaulculates the amount of
size required for the creation of the new container. |
