Coolstack MySQL Crash Unable to Restart
- by rayblasdel
Environment: Solaris 10
This mysql server has been up and running for 6 months now. Today all of a sudden it crashed. When typing 'mysql' as user it gives the error
MYSQL" Error 2002 (HY000): Can't Connect to Local MySQL server though socket '/tmp/mysql.sock'
The server try to open mysql, it stays open for 9-10 seconds and restarts the process. Below are the application logs.
Application-database-mysql_mysql-csk.log
[ May 30 22:37:52 Enabled. ]
[ May 30 22:37:58 Rereading configuration. ]
[ May 30 22:37:59 Executing start method ("/opt/coolstack/lib/svc/method/svc-cskmysql start") ]
/opt/coolstack/mysql/bin/mysqld_safe --user=mysql --datadir=/dbpool1/data --pid-file=/dbpool1/data/database.soliaonline.com.pid
[ May 30 22:37:59 Method "start" exited with status 0 ]
[ May 30 22:38:13 Stopping because all processes in service exited. ]
[ May 30 22:38:13 Executing stop method ("/opt/coolstack/lib/svc/method/svc-cskmysql stop") ]
[ May 30 22:38:13 Method "stop" exited with status 0 ]
[ May 30 22:38:13 Executing start method ("/opt/coolstack/lib/svc/method/svc-cskmysql start") ]
/opt/coolstack/mysql/bin/mysqld_safe --user=mysql --datadir=/dbpool1/data --pid-file=/dbpool1/data/database.soliaonline.com.pid
[ May 30 22:38:13 Method "start" exited with status 0 ]
[ May 30 22:38:25 Stopping because all processes in service exited. ]
[ May 30 22:38:25 Executing stop method ("/opt/coolstack/lib/svc/method/svc-cskmysql stop") ]
[ May 30 22:38:25 Method "stop" exited with status 0 ]
I am hoping someone might have run into this before and might know how to fix it.
The following is an excerpt from the MySQL Error log
100530 22:44:03 mysqld_safe mysqld from pid file /dbpool1/data/database.soliaonline.com.pid ended
100530 22:44:04 mysqld_safe Starting mysqld daemon with databases from /dbpool1/data
InnoDB: Log scan progressed past the checkpoint lsn 32 727170612
100530 22:44:13 InnoDB: Database was not shut down normally!
InnoDB: Starting crash recovery.
InnoDB: Reading tablespace information from the .ibd files...
InnoDB: Restoring possible half-written data pages from the doublewrite
InnoDB: buffer...
InnoDB: Doing recovery: scanned up to log sequence number 32 727200901
100530 22:44:14 InnoDB: Starting an apply batch of log records to the database...
InnoDB: Progress in percents: 100530 22:44:14 - mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
We will try our best to scrape up some info that will hopefully help diagnose
the problem, but since we have already crashed, something is definitely wrong
and this may fail.
key_buffer_size=209715200
read_buffer_size=1048576
max_used_connections=0
max_threads=10000
threads_connected=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 31024253 K
bytes of memory
Hope that's ok; if not, decrease some variables in the equation.
enter code here
