I'm getting Segmentation fault error when trying to access the "Hello World" pyramid app. This error only occurs when running against CentOS 5.7 setup, but no problem whatsoever when tested against OSX and Arch Linux. Could it be a CentOS specific issue?

[error] [client 10.211.55.2] Premature end of script headers: pyramid.wsgi
[notice] child pid 31212 exit signal Segmentation fault (11)

I have tried to follow the troubleshooting guides posted here http://code.google.com/p/modwsgi/wiki/InstallationIssues which suggests that it might caused by missing Shared Library. A quick check reveals that shared library is not the issue.

[centos57@localhost modules]$ ldd mod_wsgi.so 
    linux-gate.so.1 =>  (0x00e6a000)
    libpython2.7.so.1.0 => /home/python/lib/libpython2.7.so.1.0 (0x0024c000)
    libpthread.so.0 => /lib/libpthread.so.0 (0x00da8000)
    libdl.so.2 => /lib/libdl.so.2 (0x00cd6000)
    libutil.so.1 => /lib/libutil.so.1 (0x00110000)
    libm.so.6 => /lib/libm.so.6 (0x0085c000)
    libc.so.6 => /lib/libc.so.6 (0x00682000)
    /lib/ld-linux.so.2 (0x0012b000)

Then I found another clue that might be able to solve my problem. Unfortunately libexpat is not the source of the problem. http://code.google.com/p/modwsgi/wiki/IssuesWithExpatLibrary

[centos57@localhost bin]$ ldd ~/httpd/bin/httpd | grep expat
libexpat.so.1 => /usr/local/lib/libexpat.so.1 (0x00b00000)

[centos57@localhost bin]$ strings /usr/local/lib/libexpat.so.1 | grep expat
   libexpat.so.1
   expat_2.0.1

[centos57@localhost bin]$ python
  Python 2.7.2 (default, Nov 26 2011, 08:08:44) 
  [GCC 4.1.2 20080704 (Red Hat 4.1.2-51)] on linux2
  Type "help", "copyright", "credits" or "license" for more information.
  >>> import pyexpat
  >>> pyexpat.version_info
  (2, 0, 0)
  >>> 

I've been pulling my hair out trying to figure out what I'm missing in my setup. Why the problem only occurs with CentOS?

Here is the detailed setup:

Apache 2.2.19
Python 2.7.2
mod_wsgi-3.3

/home/httpd/conf/extra/pyramid.wsgi

from pyramid.paster import get_app
application = get_app('/home/homecamera/hcadmin/root/production.ini', 'main')

/home/httpd/conf/extra/modwsgi.conf

LoadModule wsgi_module modules/mod_wsgi.so
WSGIScriptAlias /myapp /home/root/test.wsgi
<Directory /home/root>
  WSGIProcessGroup pyramid
  Order allow,deny
  Allow from all
</Directory>

# Use only 1 Python sub-interpreter.  Multiple sub-interpreters
# play badly with C extensions.

WSGIApplicationGroup %{GLOBAL}
WSGIPassAuthorization On
WSGIDaemonProcess pyramid user=daemon group=daemon processes=1 \
  threads=4 \
  python-path=/home/python/lib/python2.7/site-packages
WSGIScriptAlias /hello /home/httpd/conf/extra/pyramid.wsgi

<Directory /home/httpd/conf/extra>
  WSGIProcessGroup pyramid
  Order allow,deny
  Allow from all
</Directory>

Again this same setup works on OSX and Arch Linux but not on CentOS 5.7. Could someone out there point me to the right direction before I ran out of my hair.

====================================================================================

When apache started with gdb, I got a couple of warnings

Reading symbols from /home/httpd/bin/httpd...done.
Attaching to program: /home/httpd/bin/httpd, process 1821

warning: .dynamic section for "/lib/libcrypt.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

warning: .dynamic section for "/lib/libutil.so.1" is not at the expected address

warning: difference appears to be caused by prelink, adjusting expectations

gdb output. After hitting refresh button, to load pyramid.

(gdb) cont
Continuing.
warning: .dynamic section for "/usr/lib/libgssapi_krb5.so.2" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/usr/lib/libkrb5.so.3" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations
warning: .dynamic section for "/lib/libresolv.so.2" is not at the expected address
warning: difference appears to be caused by prelink, adjusting expectations

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x8edbb90 (LWP 1824)]
0x0814c120 in EVP_PKEY_CTX_dup ()

apache_error_log

[info] mod_wsgi (pid=1821): Starting process 'pyramid' with threads=1.
[info] mod_wsgi (pid=1821): Initializing Python.
[info] mod_wsgi (pid=1821): Attach interpreter ''.
[info] mod_wsgi (pid=1821): Create interpreter 'web.domain.com:20000|/hcadmin'.
[info] [client 10.211.55.2] mod_wsgi (pid=1821, process='pyramid', application='web.domain.com:20000|/hcadmin'): Loading WSGI script '/home/httpd/conf/extra/pyramid.wsgi'.
[error] hello 1