Previously I described how to observe congestion control strategies during transmission, and here I talked about TCP's sliding window approach for handling flow control on the receive side. A neat trick would now be to put the pieces together and ask the following question - how often is TCP transmission blocked by congestion control (send-side flow control) versus a zero-sized send window (which is the receiver saying it cannot process any more data)? So in effect we are asking whether the size of the receive window of the peer or the congestion control strategy may be sub-optimal. The result of such a problem would be that we have TCP data that we could be transmitting but we are not, potentially effecting throughput.

So flow control is in effect:

• when the congestion window is less than or equal to the amount of bytes outstanding on the connection. We can derive this from args[3]->tcps_snxt - args[3]->tcps_suna, i.e. the difference between the next sequence number to send and the lowest unacknowledged sequence number; and
• when the window in the TCP segment received is advertised as 0

#!/usr/sbin/dtrace -s

#pragma D option quiet


tcp:::send
/ (args[3]->tcps_snxt - args[3]->tcps_suna) >= args[3]->tcps_cwnd /
{
        cwndclosed[args[1]->cs_cid] = timestamp;
        cwndsnxt[args[1]->cs_cid] = args[3]->tcps_snxt;
        @numclosed["cwnd", args[2]->ip_daddr, args[4]->tcp_dport] = count();
}

tcp:::send
/ cwndclosed[args[1]->cs_cid] && args[4]->tcp_seq >= cwndsnxt[args[1]->cs_cid] /
{
        @meantimeclosed["cwnd", args[2]->ip_daddr, args[4]->tcp_dport] =
            avg(timestamp - cwndclosed[args[1]->cs_cid]);
        @stddevtimeclosed["cwnd", args[2]->ip_daddr, args[4]->tcp_dport] =
            stddev(timestamp - cwndclosed[args[1]->cs_cid]);
        @numclosed["cwnd", args[2]->ip_daddr, args[4]->tcp_dport] = count();
        cwndclosed[args[1]->cs_cid] = 0;
        cwndsnxt[args[1]->cs_cid] = 0;
}

tcp:::receive
/ args[4]->tcp_window == 0 &&
  (args[4]->tcp_flags & (TH_SYN|TH_RST|TH_FIN)) == 0 /
{
        swndclosed[args[1]->cs_cid] = timestamp;
        swndsnxt[args[1]->cs_cid] = args[3]->tcps_snxt;
        @numclosed["swnd", args[2]->ip_saddr, args[4]->tcp_dport] = count();
}

tcp:::send
/ swndclosed[args[1]->cs_cid] && args[4]->tcp_seq >= swndsnxt[args[1]->cs_cid] /
{
        @meantimeclosed["swnd", args[2]->ip_daddr, args[4]->tcp_sport] =
            avg(timestamp - swndclosed[args[1]->cs_cid]);
        @stddevtimeclosed["swnd", args[2]->ip_daddr, args[4]->tcp_sport] =
            stddev(timestamp - swndclosed[args[1]->cs_cid]);
        swndclosed[args[1]->cs_cid] = 0;
        swndsnxt[args[1]->cs_cid] = 0;
}


END
{
        printf("%-6s %-20s %-8s %-25s %-8s %-8s\n", "Window",
            "Remote host", "Port", "TCP Avg WndClosed(ns)", "StdDev",
            "Num");
        printa("%-6s %-20s %-8d %@-25d %@-8d %@-8d\n", @meantimeclosed,
            @stddevtimeclosed, @numclosed);
}

So this script will show us whether the peer's receive window size is preventing flow ("swnd" events) or whether congestion control is limiting flow ("cwnd" events). As an example I traced on a server with a large file transfer in progress via a webserver and with an active ssh connection running "find / -depth -print". Here is the output:

^C
Window Remote host          Port     TCP Avg WndClosed(ns)     StdDev   Num     
cwnd   10.175.96.92         80       86064329                  77311705 125     
cwnd   10.175.96.92         22       122068522                 151039669 81  

So we see in this case, the congestion window closes 125 times for port 80 connections and 81 times for ssh. The average time the window is closed is 0.086sec for port 80 and 0.12sec for port 22.

So if you wish to change congestion control algorithm in Oracle Solaris 11, a useful step may be to see if congestion really is an issue on your network. Scripts like the one posted above can help assess this, but it's worth reiterating that if congestion control is occuring, that's not necessarily a problem that needs fixing. Recall that congestion control is about controlling flow to prevent large-scale drops, so looking at congestion events in isolation doesn't tell us the whole story. For example, are we seeing more congestion events with one control algorithm, but more drops/retransmission with another? As always, it's best to start with measures of throughput and latency before arriving at a specific hypothesis such as "my congestion control algorithm is sub-optimal".