I'm having an issue with a linux server. Once a week the running mysql instance hangs and there is no way to fully stop it. If I kill it, it remains in zombie status and init does not reap its pid.

The server is used for staging deployments and some internal tools, so it's not under heavy load. The only process constantly used id mysql and for this I think that it's the only process which suffer of this issue.

I've searched system logs for errors and the only thing I found is this error (repeated a couple of times) in dmesg output:

[706560.640085] INFO: task mysqld:31965 blocked for more than 120 seconds.
[706560.640198] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[706560.640312] mysqld          D ffff88032fd93f40     0 31965      1 0x00000000
[706560.640317]  ffff880242a27d18 0000000000000086 ffff88031a50dd00 ffff880242a27fd8
[706560.640321]  ffff880242a27fd8 ffff880242a27fd8 ffff88031e549740 ffff88031a50dd00
[706560.640325]  ffff88031a50dd00 ffff88032fd947f8 0000000000000002 ffffffff8112f250
[706560.640328] Call Trace:
[706560.640338]  [<ffffffff8112f250>] ? __lock_page+0x70/0x70
[706560.640344]  [<ffffffff816cb1b9>] schedule+0x29/0x70
[706560.640347]  [<ffffffff816cb28f>] io_schedule+0x8f/0xd0
[706560.640350]  [<ffffffff8112f25e>] sleep_on_page+0xe/0x20
[706560.640353]  [<ffffffff816c9900>] __wait_on_bit+0x60/0x90
[706560.640356]  [<ffffffff8112f390>] wait_on_page_bit+0x80/0x90
[706560.640360]  [<ffffffff8107dce0>] ? autoremove_wake_function+0x40/0x40
[706560.640363]  [<ffffffff8112f891>] filemap_fdatawait_range+0x101/0x190
[706560.640366]  [<ffffffff81130975>] filemap_write_and_wait_range+0x65/0x70
[706560.640371]  [<ffffffff8122e441>] ext4_sync_file+0x71/0x320
[706560.640376]  [<ffffffff811c3e6d>] do_fsync+0x5d/0x90
[706560.640379]  [<ffffffff811c40d0>] sys_fsync+0x10/0x20
[706560.640383]  [<ffffffff816d495d>] system_call_fastpath+0x1a/0x1f

When this happens the only way to make everything working again is a full reboot, but in order to do that I'm forced to use this command after I've manually stopped all running processes

echo b > /proc/sysrq-trigger

otherwise normal reboot process hangs forever. I've tracked reboots script and I've found out that also the reboot process hangs on a sync call, this one in /etc/init.d/sendsigs (I'm on ubuntu)

# Flush the kernel I/O buffer before we start to kill
# processes, to make sure the IO of already stopped services to
# not slow down the remaining processes to a point where they
# are accidentily killed with SIGKILL because they did not
# manage to shut down in time.
sync

I'm almost sure that the cause of this is an hardware issue (the RAID controller???) also because I've other two machines with the same hardware and software configuration and they don't suffer of this, but I can't find any hint in syslog or dmesg. I've also installed smartmontools and mcelog packages but none of them did report any issue.

What can I do to track the cause of this issue?

Today is happened again, here is the status of system after triggering a reboot

init---console-kit-dae---64*[{console-kit-dae}]
     +-dbus-daemon
     +-mcelog
     +-mysqld---{mysqld}
     +-newrelic-daemon---newrelic-daemon---11*[{newrelic-daemon}]
     +-ntpd
     +-polkitd---{polkitd}
     +-python3
     +-rpc.idmapd
     +-rpc.statd
     +-rpcbind
     +-sh---rc---S20sendsigs---sync
     +-smartd
     +-snmpd
     +-sshd---sshd---zsh---sudo---zsh---pstree
     +-sshd---sshd---zsh---sudo---zsh

And here is the status of sync process

# ps aux | grep sync
root      3637  0.1  0.0   4352   372 ?        D    05:53   0:00 sync

i.e. Uninterruptible sleep...

Hardware specs as reported by lshw

I think the raid controller is a fake raid. I usually don't deal with hardware (and for the record I don't have physical access to it)

description: Computer
product: X7DBP ()
vendor: Supermicro
version: 0123456789
serial: 0123456789
width: 64 bits
capabilities: smbios-2.4 dmi-2.4 vsyscall32
configuration: administrator_password=disabled boot=normal frontpanel_password=unknown keyboard_password=unknown power-on_password=disabled uuid=53D19F64-D663-A017-8922-0030487C1FEE
*-core
   description: Motherboard
   product: X7DBP
   vendor: Supermicro
   physical id: 0
   version: PCB Version
   serial: 0123456789
 *-firmware
      description: BIOS
      vendor: Phoenix Technologies LTD
      physical id: 0
      version: 6.00
      date: 05/29/2007
      size: 106KiB
      capacity: 960KiB
      capabilities: pci pnp upgrade shadowing escd cdboot bootselect edd int13floppy2880 acpi usb ls120boot zipboot biosbootspecification

  *-storage
         description: RAID bus controller
         product: 631xESB/632xESB SATA RAID Controller
         vendor: Intel Corporation
         physical id: 1f.2
         bus info: pci@0000:00:1f.2
         version: 09
         width: 32 bits
         clock: 66MHz
         capabilities: storage pm bus_master cap_list
         configuration: driver=ahci latency=0
         resources: irq:19 ioport:18a0(size=8) ioport:1874(size=4) ioport:1878(size=8) ioport:1870(size=4) ioport:1880(size=32) memory:d8500400-d85007ff