My best theory is that some kernel limit is being hit preventing outbound connections. We have tried basically everything from tcpdumps to provisioning an entirely new virtual server (we do not have this problem on any other virtuals), however the problem somehow carried over, even with new postfix build (working).

Emails work, and outbound connections work, so long as postfix does not have too much going on. /proc/user_beancounters shows no limits being hit (show below). Nevertheless, pings fail even to IP addresses. TCP stack appears healthy. Load is low. No iowait. Flushed iptables already. Has anyone experienced anything like this?

   uid  resource                     held              maxheld              barrier                   limit              failcnt
    3:  kmemsize                166216365            170262528  9223372036854775807  9223372036854775807                    0
        lockedpages                     0                    0  9223372036854775807  9223372036854775807                    0
        privvmpages                285727               351885  9223372036854775807  9223372036854775807                    0
        shmpages                    16933                17605  9223372036854775807  9223372036854775807                    0
        dummy                           0                    0                    0                    0                    0
        numproc                       150                  303  9223372036854775807  9223372036854775807                    0
        physpages                  314156               326191                    0              1280000                    0
        vmguarpages                     0                    0  9223372036854775807  9223372036854775807                    0
        oomguarpages               165355               165355  9223372036854775807  9223372036854775807                    0
        numtcpsock                     89                  172  9223372036854775807  9223372036854775807                    0
        numflock                       22                   76  9223372036854775807  9223372036854775807                    0
        numpty                          1                    2  9223372036854775807  9223372036854775807                    0
        numsiginfo                      0                   75  9223372036854775807  9223372036854775807                    0
        tcpsndbuf                 2733472              4371752  9223372036854775807  9223372036854775807                    0
        tcprcvbuf                 1798336              5427296  9223372036854775807  9223372036854775807                    0
        othersockbuf               491120              1000760  9223372036854775807  9223372036854775807                    0
        dgramrcvbuf                     0               238728  9223372036854775807  9223372036854775807                    0
        numothersock                  361                  505  9223372036854775807  9223372036854775807                    0
        dcachesize              135941831            136114679  9223372036854775807  9223372036854775807                    0
        numfile                      2905                 4990  9223372036854775807  9223372036854775807                    0
        dummy                           0                    0                    0                    0                    0
        dummy                           0                    0                    0                    0                    0
        dummy                           0                    0                    0                    0                    0
        numiptent                       8                    9  9223372036854775807  9223372036854775807                    0

[root@bni /]# ping 4.2.2.1 PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data.

--- 4.2.2.1 ping statistics --- 9 packets transmitted, 0 received, 100% packet loss, time 8493ms

[root@bni /]# service postfix stop

[root@bni /]# ping 4.2.2.1 PING 4.2.2.1 (4.2.2.1) 56(84) bytes of data. 64 bytes from 4.2.2.1: icmp_seq=1 ttl=53 time=8.63 ms 64 bytes from 4.2.2.1: icmp_seq=2 ttl=53 time=8.62 ms 64 bytes from 4.2.2.1: icmp_seq=3 ttl=53 time=8.63 ms 64 bytes from 4.2.2.1: icmp_seq=4 ttl=53 time=8.66 ms

Outbound connections of all sorts fail when postfix is running.