We have a production server that seems to Seg Fault a few times every day. The fault is picked up by Apache and logged in the error log - but there seems to be no traffic around the time. If it's a request generating the fault then it looks like it happens before any other logging is made so I can't see how it's happening so it's very hard to debug.

Our setup is Linux 64 bit Centos 5.3 Apache is loaded with the following modules apachectl -t -D DUMP_MODULES | more

Loaded Modules:
 core_module (static)
 mpm_prefork_module (static)
 http_module (static)
 so_module (static)
 auth_basic_module (shared)
 auth_digest_module (shared)
 authn_file_module (shared)
 authn_alias_module (shared)
 authn_anon_module (shared)
 authn_dbm_module (shared)
 authn_default_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 authz_owner_module (shared)
 authz_groupfile_module (shared)
 authz_dbm_module (shared)
 authz_default_module (shared)
 ldap_module (shared)
 authnz_ldap_module (shared)
 include_module (shared)
 log_config_module (shared)
 logio_module (shared)
 env_module (shared)
 ext_filter_module (shared)
 mime_magic_module (shared)
 expires_module (shared)
 deflate_module (shared)
 headers_module (shared)
 usertrack_module (shared)
 setenvif_module (shared)
 mime_module (shared)
 dav_module (shared)
 status_module (shared)
 autoindex_module (shared)
 info_module (shared)
 dav_fs_module (shared)
 vhost_alias_module (shared)
 negotiation_module (shared)
 dir_module (shared)
 actions_module (shared)
 speling_module (shared)
 userdir_module (shared)
 alias_module (shared)
 rewrite_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_ftp_module (shared)
 proxy_http_module (shared)
 proxy_connect_module (shared)
 cache_module (shared)
 suexec_module (shared)
 disk_cache_module (shared)
 file_cache_module (shared)
 mem_cache_module (shared)
 cgi_module (shared)
 version_module (shared)
 security2_module (shared)
 unique_id_module (shared)
 fcgid_module (shared)
 php5_module (shared)
 proxy_ajp_module (shared)
 ssl_module (shared)

Here's an exert from the Apache error log:

[Mon Mar 15 06:39:25 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 07:41:31 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 08:24:16 2010] [error] [client 67.19.250.146] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 08:43:46 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 08:54:02 2010] [error] [client 74.208.123.71] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 09:09:51 2010] [notice] child pid 2138 exit signal Segmentation fault (11), possible coredump in /tmp
[Mon Mar 15 09:45:27 2010] [error] [client 213.246.222.74] client sent HTTP/1.1 request without hostname (see RFC2616 section 14.23): /w00tw00t.at.ISC.SANS.DFind:)
[Mon Mar 15 09:49:05 2010] [error] [client 190.12.113.196] File does not exist: /var/www/vhosts/default/htdocs/phpMyAdmin
[Mon Mar 15 09:49:06 2010] [error] [client 190.12.113.196] File does not exist: /var/www/vhosts/default/htdocs/PMA

And the Access log around the same time (09:09:51):

213.246.222.74 - - [15/Mar/2010:08:43:46 +0000] "GET /" 400 561 "-" "-"
208.80.193.28 - - [15/Mar/2010:08:52:20 +0000] "GET / HTTP/1.0" 301 313 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0; SU 2.009)"
74.208.123.71 - - [15/Mar/2010:08:54:02 +0000] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 298 "-" "-"
81.149.146.231 - - [15/Mar/2010:09:15:18 +0000] "GET /zabbix/ HTTP/1.1" 200 3565 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_6_2; en-us) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"
81.158.71.196 - - [15/Mar/2010:09:16:06 +0000] "GET / HTTP/1.1" 301 313 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10.4; en-US; rv:1.9.0.18) Gecko/2010020219 Firefox/3.0.18"
213.246.222.74 - - [15/Mar/2010:09:45:27 +0000] "GET /" 400 561 "-" "-"
213.246.222.74 - - [15/Mar/2010:09:45:27 +0000] "GET /w00tw00t.at.ISC.SANS.DFind:) HTTP/1.1" 400 298 "-" "-"
190.12.113.196 - - [15/Mar/2010:09:49:05 +0000] "GET /phpMyAdmin/main.php HTTP/1.0" 404 295 "-" "-"

So As you can see, there's no access logged around the time of the fault!! How annoying :s

I enabled core dumps and here is the backtrace:

#0  0x00007f9c8c8a858b in memcpy () from /lib64/libc.so.6
No symbol table info available.
#1  0x00007f9c8cfb066d in apr_pstrcat (a=<value optimized out>) at strings/apr_strings.c:165
        cp = 0x1fa6b "\205¦H\211¦t`¦\003"
        argp = 0x7f9c9ad790e8 "Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Referer, Re"...
        res = 0x0
        saved_lengths = {129643, 2, 43, 140310399395576, 0, 140310394592712}
        nargs = <value optimized out>
        len = <value optimized out>
        adummy = {{gp_offset = 16, fp_offset = 32668, overflow_arg_area = 0x7fff968a0ec0, reg_save_area = 0x7fff968a0de0}}
#2  0x00007f9c8cfb1bf9 in apr_table_merge (t=0x7f9c8f83b148, key=0x7f9c85a465fe "Vary", val=0x7f9c9ad99070 "Referer, Referer, Referer, Referer, Referer") at tables/apr_tables.c:688
        next_elt = (apr_table_entry_t *) 0x7f9c8f83b270
        end_elt = (apr_table_entry_t *) 0x7f9c8f83b270
        checksum = <value optimized out>
        hash = 22
#3  0x00007f9c85a42cfa in ?? () from /etc/httpd/modules/mod_rewrite.so
No symbol table info available.
#4  0x00007f9c85a44022 in ?? () from /etc/httpd/modules/mod_rewrite.so
No symbol table info available.
#5  0x00007f9c8e87bd1a in ap_run_fixups () from /usr/sbin/httpd
No symbol table info available.
#6  0x00007f9c8e88e8f8 in ap_process_request () from /usr/sbin/httpd
No symbol table info available.
#7  0x00007f9c8e88bb40 in ?? () from /usr/sbin/httpd
No symbol table info available.
#8  0x00007f9c8e887ca2 in ap_run_process_connection () from /usr/sbin/httpd
No symbol table info available.
#9  0x00007f9c8e892849 in ?? () from /usr/sbin/httpd
No symbol table info available.
#10 0x00007f9c8e892ada in ?? () from /usr/sbin/httpd
No symbol table info available.
#11 0x00007f9c8e892b90 in ?? () from /usr/sbin/httpd
No symbol table info available.
#12 0x00007f9c8e89387b in ap_mpm_run () from /usr/sbin/httpd
No symbol table info available.
#13 0x00007f9c8e86de48 in main () from /usr/sbin/httpd
No symbol table info available.

Can anyone shed any light on how to move forward with this? I can confirm that the server is operational and doesn't appear to be misbehaving - the failures are so infrequent that I haven't seen it do one while making a request myself.

Really appreciate any help! Cheers!