I have a FreeBSD machine with jails -- two in particular, one that runs nginx and another that runs a Java program that accepts requests via Jetty (embedded mode)

Jetty receives upwards of 500 requests/sec constantly and there has been an issue lately where I will constantly have over 60,000 connections in the LAST_ACK state between nginx and jetty.

Distribution of all connections (includes some other services, particularly php-fpm)

root@host:/root # netstat -an > conns.txt
root@host:/root # cat conns.txt | awk '{print $6}' | sort | uniq -c | sort -n
18 LISTEN
112 CLOSING
485 ESTABLISHED
650 FIN_WAIT_2
1425 FIN_WAIT_1
3301 TIME_WAIT
64215 LAST_ACK

Distribution of nginx -> jetty connections

root@host:/root # cat conns.txt | grep '10.10.1.57' | awk '{print $6}' | sort | uniq -c | sort -n
1 
3 CLOSE_WAIT
3 LISTEN
18 FIN_WAIT_2
125 ESTABLISHED
64193 LAST_ACK

I'd prefer every request to fully close the connection. Clients requests are about 10 minutes apart from each other so connections must be closed.

Some of the connections,

tcp4       0      0 10.10.1.50.46809       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46805       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46797       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46794       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46790       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46789       10.10.1.57.9050        LAST_ACK
tcp4       0      0 10.10.1.50.46771       10.10.1.57.9050        LAST_ACK
etc..

• On Jetty's end I've set maxIdleTime to 2000 -- before this all connections were in ESTABLISHED but they are now LAST_ACK
• On Jetty's end I've set Connection: close (i.e response.setHeader(HttpHeaders.CONNECTION, HttpHeaderValues.CLOSE);)
• Jetty never reports a lot of open connections -- always very few.
• PF/IPFW is not currently being used
• nginx - reset_timedout_connection is on

I cannot figure out how to get nginx or jetty to forcibly close the connection, is this simply something that needs to be fixed in Jetty so that it fully closes the socket after the request finishes?

Thanks a lot in advance

EDIT: forgot my nginx config for the proxy setup-

    proxy_pass              http://10.10.1.57:9050;
    proxy_set_header        HTTP_X_GEOIP $http_x_geoip;
    proxy_set_header        GEOIP_COUNTRY_CODE $geoip_country_code;
    proxy_set_header        X-Real-IP $remote_addr;
    proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header        Host $http_host;
    proxy_set_header        Connection "";
    proxy_http_version      1.1;

EDIT2: Forcing Jetty to close the connection via request.getConnection().getEndPoint().close() does nothing -- it's obvious the connection IS being closed (as it's in LAST_ACK) but why isn't it getting past this? Is Nginx keeping the connection open to the backend for some reason?