We gave access to a contractor to install a firewall and somehow while he was doing it he fracked something up. Everything went off-line about 24 hours ago and we are effectively out of business until I solve this and the person who messed up the thing is not returning calls.

I found a few errors. First, I'm not a server guy - I can look at log files and normally everything runs fine. All 'services' are running according to 1and1 server monitoring and mail is being delivered just fine. The whole thing was off-line until I (probably stupidly) updated the kernel from 6.2 to 6.3 this morning and I got everything back except the http access.

All the domains (~200 of them) are returning a 403 error and nothing is recorded in the access log.

On every restart I see this error in the messages log file:

init: Failed to spawn ttyS0 main process: unable to execute: No such file or directory

and a little later these:

kernel: WARNING: at kernel/sched.c:5914 thread_return+0x232/0x79d() (Not tainted)
kernel: Hardware name: X9SCL/X9SCM
kernel: Modules linked in: xt_iprange iptable_filter ip_tables ip6t_REJECT nf_conntrack_ipv6 nf_defrag_ipv6 xt_state nf_conntrack ip6table_filter ip6_tables ipv6 ext4 jbd2 serio_raw i2c_i801 i2c_core sg iTCO_wdt iTCO_vendor_support e1000e ext3 jbd mbcache raid1 sd_mod crc_t10dif ahci dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]
kernel: Pid: 367, comm: md3_raid1 Not tainted 2.6.32-220.2.1.el6.x86_64 #1
kernel: Call Trace:
kernel: [<ffffffff81069997>] ? warn_slowpath_common+0x87/0xc0
kernel: [<ffffffff810699ea>] ? warn_slowpath_null+0x1a/0x20
kernel: [<ffffffff814eccc5>] ? thread_return+0x232/0x79d
kernel: [<ffffffff8126a4d9>] ? cpumask_next_and+0x29/0x50
kernel: [<ffffffff813e9c05>] ? md_super_wait+0x55/0x90
kernel: [<ffffffff81090a10>] ? autoremove_wake_function+0x0/0x40
kernel: [<ffffffff813ebf46>] ? md_update_sb+0x206/0x3f0
kernel: [<ffffffff813ee922>] ? md_check_recovery+0x3f2/0x6d0
kernel: [<ffffffffa005b129>] ? raid1d+0x49/0x1050 [raid1]
kernel: [<ffffffff814ed985>] ? schedule_timeout+0x215/0x2e0
kernel: [<ffffffff814ef447>] ? _spin_unlock_irqrestore+0x17/0x20
kernel: [<ffffffff813eb336>] ? md_thread+0x116/0x150
kernel: [<ffffffff81090a10>] ? autoremove_wake_function+0x0/0x40
kernel: [<ffffffff813eb220>] ? md_thread+0x0/0x150
kernel: [<ffffffff810906a6>] ? kthread+0x96/0xa0
kernel: [<ffffffff8100c14a>] ? child_rip+0xa/0x20
kernel: [<ffffffff81090610>] ? kthread+0x0/0xa0
kernel: [<ffffffff8100c140>] ? child_rip+0x0/0x20

And something is wrong with the Named/BIND resulting in the same error for all domains:

zone DOMAINEXAMPLE.com/IN: loading from master file DOMAINEXAMPLE.com failed: file not found
zone DOMAINEXAMPLE.com/IN: not loaded due to errors.
_default/DOMAINEXAMPLE.com/IN: file not found

I'm pretty sure this is not enough information to solve the problem, but I'm willing to engage someone who can work this out for me.

Any help would be greatly appreciated.