FreeBSD's ng_nat stopping pass the packets periodically

Posted by Korjavin Ivan on Server Fault See other posts from Server Fault or by Korjavin Ivan
Published on 2013-05-08T14:03:22Z Indexed on 2013/11/12 21:57 UTC
Read the original article Hit count: 299

Filed under:

freebsd

|

nat

|

netgraph

I have FreeBSD router:

#uname
9.1-STABLE FreeBSD 9.1-STABLE #0: Fri Jan 18 16:20:47 YEKT 2013

It's a powerful computer with a lot of memory

#top -S
last pid: 45076;  load averages:  1.54,  1.46,  1.29                                      up 0+21:13:28  19:23:46
84 processes:  2 running, 81 sleeping, 1 waiting
CPU:  3.1% user,  0.0% nice, 32.1% system,  5.3% interrupt, 59.5% idle
Mem: 390M Active, 1441M Inact, 785M Wired, 799M Buf, 5008M Free
Swap: 8192M Total, 8192M Free

  PID USERNAME    THR PRI NICE   SIZE    RES STATE   C   TIME   WCPU COMMAND
   11 root          4 155 ki31     0K    64K RUN     3  71.4H 254.83% idle
   13 root          4 -16    -     0K    64K sleep   0 101:52 103.03% ng_queue
    0 root         14 -92    0     0K   224K -       2 229:44 16.55% kernel
   12 root         17 -84    -     0K   272K WAIT    0 213:32 15.67% intr
40228 root          1  22    0 51060K 25084K select  0  20:27  1.66% snmpd
15052 root          1  52    0   104M 22204K select  2   4:36  0.98% mpd5
   19 root          1  16    -     0K    16K syncer  1   0:48  0.20% syncer

Its tasks are: NAT via ng_nat and PPPoE server via mpd5.

Traffic through - about 300Mbit/s, about 40kpps at peak. Pppoe sessions created - 350 max.

ng_nat is configured by by the script:

 /usr/sbin/ngctl -f- <<-EOF                                            

             mkpeer ipfw: nat %s out                                                                               
             name ipfw:%s %s                                                                                       
             connect ipfw: %s: %s in                                                                               
             msg %s: setaliasaddr 1.1.%s

There are 20 such ng_nat nodes, with about 150 clients.

Sometimes, the traffic via nat stops. When this happens vmstat reports a lot of FAIL counts

vmstat -z | grep -i netgraph
ITEM                   SIZE  LIMIT     USED     FREE      REQ FAIL SLEEP
NetGraph items:          72,  10266,       1,     376,39178965,   0,   0
NetGraph data items:     72,  10266,       9,   10257,2327948820,2131611,4033

I was tried increase

net.graph.maxdata=10240                                                                                           
net.graph.maxalloc=10240

but this doesn't work.

It's a new problem (1-2 week). The configuration had been working well for about 5 months and no configuration changes were made leading up to the problems starting.

In the last few weeks we have slightly increased traffic (from 270 to 300 mbits) and little more pppoe sessions (300->350).

Help me please, how to find and solve my problem?

Developer IT