Issue with apache mod proxy, it stops proxying requests after a graceful restart but not all the time. It seems to happen only on a Sunday when a graceful restart is triggered by logrotate.

    [Sun Sep 9 05:25:06 2012] [notice] SIGUSR1 received.  Doing graceful restart
    [Sun Sep 9 05:25:06 2012] [notice] Apache/2.2.22 (Ubuntu) Phusion_Passenger/3.0.11 configured -- resuming normal operations
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(492) failed in child 26153 for worker proxy:reverse
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(486) failed in child 26153 for worker http://api.myservice.org/api
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(487) failed in child 26153 for worker http://api.myservice.org/editor/$1
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(489) failed in child 26153 for worker http://api.myservice.org/build
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(490) failed in child 26153 for worker http://api.myservice.org/help
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(491) failed in child 26153 for worker http://api.myservice.org/motd.html
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(480) failed in child 26153 for worker http://api.myservice.org/api
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(481) failed in child 26153 for worker http://api.myservice.org/editor/$1
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(483) failed in child 26153 for worker http://api.myservice.org/build
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(484) failed in child 26153 for worker http://api.myservice.org/help
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(485) failed in child 26153 for worker http://api.myservice.org/motd.html
    [Sun Sep 9 05:25:06 2012] [error] proxy: ap_get_scoreboard_lb(479) failed in child 26153 for worker http://api.myservice.org/motd.html

After these lines, the logs are flooded with 404's because the requests are not being proxied. It's worth noting that the destination is just another vhost on the same apache instance, but the vhost (http://api.myservice.org) is serving passenger (mod_rails)

I was thinking that maybe there's some startup issues with the passenger workers not being ready during a graceful restart?

After a full restart resolves it and everything returns to normal.

//Edit Here's the vhost config, thanks :)

 <VirtualHost *:80>
      UseCanonicalName Off

      LogFormat "%V %h %l %u %t \"%r\" %s %b" vcommon

      <Directory /var/www/vhosts>
              RewriteEngine on
              AllowOverride All
      </Directory>

      RewriteEngine on
      RewriteCond /var/www/vhosts/%{SERVER_NAME} !-d
      RewriteCond /var/www/vhosts/%{SERVER_NAME} !-l
      RewriteRule ^ http://sitenotfound.myservice.org/ [R=302,L]

      VirtualDocumentRoot /var/www/vhosts/%0/current

      # Rewrite requests to /assets to map to the /var/file-store/<SERVER_NAME>/
      RewriteMap lowercase int:tolower
      RewriteCond %{REQUEST_URI} ^/assets/
      RewriteRule  ^/assets/(.*)$ /var/file-store/${lowercase:%{SERVER_NAME}}/$1

      # Map /login to /editor.html as it's far friendlier.
      RewriteCond %{REQUEST_URI} ^/login
      RewriteRule .* /editor.html [PT]

      # Forward some requests to the API
      ProxyPass /api http://api.myservice.org/api
      ProxyPass /site.json http://api.myservice.org/api/editor/site
      ProxyPassMatch ^/editor/(.*)$ http://api.myservice.org/editor/$1
      ProxyPassMatch ^/api/(.*) http://api.myservice.org/api/$1
      ProxyPass /build http://api.myservice.org/build
      ProxyPass /help http://api.myservice.org/help
      ProxyPass /motd.html http://api.myservice.org/motd.html

      <Proxy *>
              Order allow,deny
              Allow from all
      </Proxy>

      # TODO generate slightly more specific Error Documents for 401/403/500's, 
      # but for now the 404 page is good enough
      ErrorDocument 401 /404.html
      ErrorDocument 403 /404.html
      ErrorDocument 404 /404.html
      ErrorDocument 500 /404.html
 </VirtualHost>