We are running a pfsense firewall / load balancer with varnish as service, In front of Fedora linux webservers running apache.
We are getting intermittent 503 guru mediation errors. We are a bit stuck scratching our heads because it is not easily repeatable. The timeouts are set to 30s (connect and first byte) but yet the 503 page will show instantly, not after 30s. Then if you refresh immediately it may very well work instantly and sometimes for a 100 refreshes.
The load average on the web servers is < 1, the DB server is < 3 (all servers (web, db, pfsense/varnish) are physical rather than VM.
I would have thought if the timeouts were being hit then the 503 page would only appear after 30s am I mistaken?
Also when an error happens there does not appear to be any corresponding error in apache's log files.
This seems to affect pages as well as images, so it is possible to have the page load fine, and for 9/10 images on the page to be fine but 1 not work
An example of the varnish debug is below. It says no backend connection but I can't figure out why, if the load was high on apache I could understand it being flaky
The machines are on the same gig ethernet lan
21 ReqStart c *IP-REMOVED* 33418 1274368062
21 RxRequest c GET
21 RxURL c /fashion/
21 RxProtocol c HTTP/1.1
21 RxHeader c User-Agent: Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.9.0.5) Gecko/2008121622 Fedora/3.0.5-1.fc10 Firefox/3.0.5
21 RxHeader c Host: *ourdomain.com*
21 RxHeader c Accept: */*
21 RxHeader c Accept-Encoding: deflate, gzip
21 VCL_call c recv lookup
21 VCL_call c hash
21 Hash c /fashion/
21 Hash c *ourdomain.com*
21 VCL_return c hash
21 VCL_call c miss fetch
21 FetchError c no backend connection
21 VCL_call c error restart
21 VCL_call c recv lookup
21 VCL_call c hash
21 Hash c /fashion/
21 Hash c *ourdomain.com*
21 VCL_return c hash
21 VCL_call c miss fetch
21 FetchError c no backend connection
21 VCL_call c error restart
21 VCL_call c recv lookup
21 VCL_call c hash
21 Hash c /fashion/
21 Hash c *ourdomain.com*
21 VCL_return c hash
21 VCL_call c miss fetch
21 FetchError c no backend connection
21 VCL_call c error deliver
21 VCL_call c deliver deliver
21 TxProtocol c HTTP/1.1
21 TxStatus c 503
21 TxResponse c Service Unavailable
21 TxHeader c Server: Varnish
21 TxHeader c Content-Type: text/html; charset=utf-8
21 TxHeader c Content-Length: 384
21 TxHeader c Accept-Ranges: bytes
21 TxHeader c Date: Wed, 11 Apr 2012 10:36:17 GMT
21 TxHeader c X-Varnish: 1274368062
21 TxHeader c Age: 0
21 TxHeader c Via: 1.1 varnish
21 TxHeader c Connection: close
21 TxHeader c X-Cache: MISS
21 Length c 384
21 ReqEnd c 1274368062 1334140577.449995041 1334140577.450334787 1.794108152 0.000282764 0.000056982
