Trouble recovering MySQL InnoDB database after server crash
- by Andy Shinn
I had a server crash due to a broken iSCSI link (the filesystem went into read-only mode). After repairing the link and rebooting the machine (CentOS 5 / MySQL 5.1), the MySQL server would not start and gave the following error:
100603 19:11:46 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql
InnoDB: The log sequence number in ibdata files does not match
InnoDB: the log sequence number in the ib_logfiles!
100603 19:11:46 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Warning: database page corruption or a failed
InnoDB: file read of page 112541.
InnoDB: Trying to recover it from the doublewrite buffer.
InnoDB: Dump of the page:
100603 19:11:46 InnoDB: Page dump in ascii and hex (16384 bytes):
lots of binary data
100603 19:11:46 InnoDB: Page checksum 953720272, prior-to-4.0.14-form checksum 2641912043
InnoDB: stored checksum 617821918, prior-to-4.0.14-form stored checksum 2080617765
InnoDB: Page lsn 115 2632899642, low 4 bytes of lsn at page end 2641594600
InnoDB: Page number (if stored to page already) 112541,
InnoDB: space id (if created with = MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 0 616
InnoDB: Dump of corresponding page in doublewrite buffer:
100603 19:11:46 InnoDB: Page dump in ascii and hex (16384 bytes):
more binary data
100603 19:11:46 InnoDB: Page checksum 908374788, prior-to-4.0.14-form checksum 824841363
InnoDB: stored checksum 912869634, prior-to-4.0.14-form stored checksum 2210927931
InnoDB: Page lsn 115 2635312169, low 4 bytes of lsn at page end 2633173354
InnoDB: Page number (if stored to page already) 112541,
InnoDB: space id (if created with = MySQL-4.1.1 and stored already) 0
InnoDB: Page may be an index page where index id is 0 616
InnoDB: Also the page in the doublewrite buffer is corrupt.
InnoDB: Cannot continue operation.
InnoDB: You can try to recover the database with the my.cnf
InnoDB: option:
InnoDB: innodb_force_recovery=6
100603 19:11:46 mysqld_safe mysqld from pid file /var/run/mysqld/mysqld.pid ended
Per the error message, I have tried setting set-variable=innodb_force_recovery=6 in the my.cnf to get to the data. This allows the MySQL server to start. But when I try to do a mysqldump of the database or a SELECT * INTO OUTFILE "filename" FROM broken_table; it seems to endlessly just export the same line over and over again.
I have also tried http://code.google.com/p/innodb-tools/. But this tool fails with an error that 'blob' type is not supported.
If I try to access the data using the PHP application it crashes MySQL:
`100603 19:19:19 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=8384512
read_buffer_size=131072
max_used_connections=2
max_threads=151
threads_connected=2
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338317 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
thd: 0x15d33f0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x453aff00 thread_stack 0x40000
/usr/libexec/mysqld(my_print_stacktrace+0x24) [0x874364]
/usr/libexec/mysqld(handle_segfault+0x346) [0x5c9166]
/lib64/libpthread.so.0 [0x3a6e40eb10]
/usr/libexec/mysqld(rec_get_offsets_func+0x30) [0x7cc310]
/usr/libexec/mysqld [0x7674d8]
/usr/libexec/mysqld(btr_search_info_update_slow+0x638) [0x768d48]
/usr/libexec/mysqld(btr_cur_search_to_nth_level+0xc7d) [0x75f86d]
/usr/libexec/mysqld [0x7dd1c1]
/usr/libexec/mysqld(row_search_for_mysql+0x18b0) [0x7e03d0]
/usr/libexec/mysqld(ha_innobase::general_fetch(unsigned char*, unsigned int, unsigned int)+0x7c) [0x7526fc]
/usr/libexec/mysqld(handler::read_multi_range_next(st_key_multi_range**)+0x29) [0x6aed09]
/usr/libexec/mysqld(QUICK_RANGE_SELECT::get_next()+0x194) [0x69a964]
/usr/libexec/mysqld [0x6aafe9]
/usr/libexec/mysqld(sub_select(JOIN*, st_join_table*, bool)+0x56) [0x635196]
/usr/libexec/mysqld [0x63f9cd]
/usr/libexec/mysqld(JOIN::exec()+0x950) [0x6497c0]
/usr/libexec/mysqld(mysql_select(THD*, Item*, TABLE_LIST*, unsigned int, List&, Item*, unsigned int, st_order*, st_order*, Item*, st_order*, unsign
ed long long, select_result*, st_select_lex_unit*, st_select_lex*)+0x17b) [0x64b34b]
/usr/libexec/mysqld(handle_select(THD*, st_lex*, select_result*, unsigned long)+0x169) [0x64bc79]
/usr/libexec/mysqld [0x5d34b6]
/usr/libexec/mysqld(mysql_execute_command(THD*)+0x4e5) [0x5d6b45]
/usr/libexec/mysqld(mysql_parse(THD*, char const*, unsigned int, char const)+0x211) [0x5dc321]
/usr/libexec/mysqld(dispatch_command(enum_server_command, THD*, char*, unsigned int)+0x10b8) [0x5dd3f8]
/usr/libexec/mysqld(do_command(THD*)+0xe6) [0x5dd9e6]
/usr/libexec/mysqld(handle_one_connection+0x73d) [0x5d036d]
/lib64/libpthread.so.0 [0x3a6e40673d]
/lib64/libc.so.6(clone+0x6d) [0x3a6d4d3d1d]
Trying to get some variables.
Some pointers may be invalid and cause the dump to abort...
thd-query at 0x15de5e0 = SELECT DISTINCT count(DISTINCT i.itemid) as rowscount,i.hostid FROM items i WHERE ((i.itemid BETWEEN 000000000000000 AND 0999999
99999999)) AND i.type<9 AND (i.hostid IN (10017,10047,10050,10054,10056,10059,10062,10063,10064,10065,10066,10067,10068,10069,10070,10071,10072,10073,100
74,10075,10076,10077,10078,10079,10080,10081,10082,10084,10088,10089,10090,10091,10092,10093,10094,10095,10096,10097,10098,10099,10100,10101,10102,10103,10
104,10105,10106,10107,10108,10109)) GROUP BY i.hostid
thd-thread_id=3
thd-killed=NOT_KILLED
The manual page at contains
information that should help you find out what is causing the crash.
100603 19:19:19 mysqld_safe Number of processes running now: 0
100603 19:19:19 mysqld_safe mysqld restarted`
Before recovering form an older backup as a last resort I am looking for anymore suggestions. Thanks!
