I wanted to share knowledge of tuning FreeBSD via sysctls, so i'm posting them with comments. Based on Igor Sysoev (author of nginx) presentation about FreeBSD tuning up to 100,000-200,000 active connections.

Sysctls are for 7.x FreeBSD. Since 7.2 amd64 some of them are tuned well by default. Prior 7.0 some of them are boot only (set via /boot/loader.conf) or does not exist at all.

Highload web server sysctls:

# Max. backlog size
kern.ipc.somaxconn=4096

# Shared memory // 7.2+ can use shared memory > 2Gb
kern.ipc.shmmax=2147483648

# Sockets
kern.ipc.maxsockets=204800
# Do not use lager sockbufs on 8.0 
# ( http://old.nabble.com/Significant-performance-regression-for-increased-maxsockbuf-on-8.0-RELEASE-tt26745981.html#a26745981 )
kern.ipc.maxsockbuf=262144 

# Recive clusters (on amd64 7.2+ 65k is default)
# For such high value vm.kmem_size must be increased to 3G
#kern.ipc.nmbclusters=229376

# Jumbo pagesize(4k/8k) clusters
# Used as general packet storage for jumbo frames
# can be monitored via `netstat -m`
#kern.ipc.nmbjumbop=192000

# Jumbo 9k/16k clusters
# If you are using them
#kern.ipc.nmbjumbo9=24000
#kern.ipc.nmbjumbo16=10240

# Every socket is a file, so increase them
kern.maxfiles=204800
kern.maxfilesperproc=200000
kern.maxvnodes=200000

# Turn off receive autotuning
#net.inet.tcp.recvbuf_auto=0

# Small receive space, only usable on http-server, on file server this 
# should be increased to 65535 or even more
#net.inet.tcp.recvspace=8192

# Small send space is useful for http servers that serve small files 
# Autotuned since 7.x
net.inet.tcp.sendspace=16384

# This should be enabled if you going to use big spaces (>64k)
#net.inet.tcp.rfc1323=1
# Turn this off on highspeed, lossless connections (LAN 1Gbit+)
#net.inet.tcp.delayed_ack=0

# This feature is useful if you are serving data over modems, Gigabit Ethernet, 
# or even high speed WAN links (or any other link with a high bandwidth delay product), 
# especially if you are also using window scaling or have configured a large send window.
# You can try setting it to 0 on fileserver with 1GBit+ interfaces
# Automatically disables on small RTT ( http://www.freebsd.org/cgi/cvsweb.cgi/src/sys/netinet/tcp_subr.c?#rev1.237 )
#net.inet.tcp.inflight.enable=0

# Disable randomizing of ports to avoid false RST
# Before usage check SA here www.bsdcan.org/2006/papers/ImprovingTCPIP.pdf
# (it's also says that port randomization auto-disables at some conn.rates, but I didn't tested it thou)
#net.inet.ip.portrange.randomized=0

# Increase portrange
# For outgoing connections only. Good for seed-boxes and ftp servers.
net.inet.ip.portrange.first=1024
net.inet.ip.portrange.last=65535

# Security
net.inet.ip.redirect=0
net.inet.ip.sourceroute=0
net.inet.ip.accept_sourceroute=0
net.inet.icmp.maskrepl=0
net.inet.icmp.log_redirect=0
net.inet.icmp.drop_redirect=1
net.inet.tcp.drop_synfin=1

# Security
net.inet.udp.blackhole=1
net.inet.tcp.blackhole=2

# Increases default TTL, sometimes useful
# Default is 64
net.inet.ip.ttl=128

# Lessen max segment life to conserve resources
# ACK waiting time in miliseconds (default: 30000 from RFC)
net.inet.tcp.msl=5000

# Max bumber of timewait sockets
net.inet.tcp.maxtcptw=40960
# Don't use tw on local connections
# As of 15 Apr 2009. Igor Sysoev says that nolocaltimewait has some buggy realization.
# So disable it or now till get fixed
#net.inet.tcp.nolocaltimewait=1

# FIN_WAIT_2 state fast recycle
net.inet.tcp.fast_finwait2_recycle=1

# Time before tcp keepalive probe is sent
# default is 2 hours (7200000)
#net.inet.tcp.keepidle=60000

# Should be increased until net.inet.ip.intr_queue_drops is zero
net.inet.ip.intr_queue_maxlen=4096

# Interrupt handling via multiple CPU, but with context switch.
# You can play with it. Default is 1;
#net.isr.direct=0

# This is for routers only
#net.inet.ip.forwarding=1
#net.inet.ip.fastforwarding=1

# This speed ups dummynet when channel isn't saturated
net.inet.ip.dummynet.io_fast=1
# Increase dummynet(4) hash
#net.inet.ip.dummynet.hash_size=2048
#net.inet.ip.dummynet.max_chain_len

# Should be increased when you have A LOT of files on server 
# (Increase until vfs.ufs.dirhash_mem becames lower)
vfs.ufs.dirhash_maxmem=67108864

# Explicit Congestion Notification (see http://en.wikipedia.org/wiki/Explicit_Congestion_Notification)
net.inet.tcp.ecn.enable=1

# Flowtable - flow caching mechanism
# Useful for routers
#net.inet.flowtable.enable=1
#net.inet.flowtable.nmbflows=65535

# Extreme polling tuning
#kern.polling.burst_max=1000
#kern.polling.each_burst=1000
#kern.polling.reg_frac=100
#kern.polling.user_frac=1
#kern.polling.idle_poll=0

# IPFW dynamic rules and timeouts tuning
# Increase dyn_buckets till net.inet.ip.fw.curr_dyn_buckets is lower
net.inet.ip.fw.dyn_buckets=65536
net.inet.ip.fw.dyn_max=65536
net.inet.ip.fw.dyn_ack_lifetime=120
net.inet.ip.fw.dyn_syn_lifetime=10
net.inet.ip.fw.dyn_fin_lifetime=2
net.inet.ip.fw.dyn_short_lifetime=10
# Make packets pass firewall only once when using dummynet
# i.e. packets going thru pipe are passing out from firewall with accept
#net.inet.ip.fw.one_pass=1

# shm_use_phys Wires all shared pages, making them unswappable
# Use this to lessen Virtual Memory Manager's work when using Shared Mem.
# Useful for databases
#kern.ipc.shm_use_phys=1

/boot/loader.conf:

# Accept filters for data, http and DNS requests
# Usefull when your software uses select() instead of kevent/kqueue or when you under DDoS
# DNS accf available on 8.0+
accf_data_load="YES" 
accf_http_load="YES"
accf_dns_load="YES"

# Async IO system calls
aio_load="YES"

# Adds NCQ support in FreeBSD
# WARNING! all ad[0-9]+ devices will be renamed to ada[0-9]+
# 8.0+ only
#ahci_load=
#siis_load=

# Increase kernel memory size to 3G. 
#
# Use ONLY if you have KVA_PAGES in kernel configuration, and you have more than 3G RAM 
# Otherwise panic will happen on next reboot!
#
# It's required for high buffer sizes: kern.ipc.nmbjumbop, kern.ipc.nmbclusters, etc
# Useful on highload stateful firewalls, proxies or ZFS fileservers
# (FreeBSD 7.2+ amd64 users: Check that current value is lower!)
#vm.kmem_size="3G"

# Older versions of FreeBSD can't tune maxfiles on the fly
#kern.maxfiles="200000"

# Useful for databases 
# Sets maximum data size to 1G
# (FreeBSD 7.2+ amd64 users: Check that current value is lower!)
#kern.maxdsiz="1G"

# Maximum buffer size(vfs.maxbufspace)
# You can check current one via vfs.bufspace
# Should be lowered/upped depending on server's load-type
# Usually decreased to preserve kmem
# (default is 200M)
#kern.maxbcache="512M"

# Sendfile buffers
# For i386 only
#kern.ipc.nsfbufs=10240

# syncache Hash table tuning
net.inet.tcp.syncache.hashsize=1024
net.inet.tcp.syncache.bucketlimit=100

# Incresed hostcache
net.inet.tcp.hostcache.hashsize="16384"
net.inet.tcp.hostcache.bucketlimit="100"

# TCP control-block Hash table tuning
net.inet.tcp.tcbhashsize=4096

# Enable superpages, for 7.2+ only
# Also read http://lists.freebsd.org/pipermail/freebsd-hackers/2009-November/030094.html
vm.pmap.pg_ps_enabled=1

# Usefull if you are using Intel-Gigabit NIC
#hw.em.rxd=4096
#hw.em.txd=4096
#hw.em.rx_process_limit="-1"
# Also if you have ALOT interrupts on NIC - play with following parameters
# NOTE: You should set them for every NIC
#dev.em.0.rx_int_delay: 250
#dev.em.0.tx_int_delay: 250
#dev.em.0.rx_abs_int_delay: 250
#dev.em.0.tx_abs_int_delay: 250
# There is also multithreaded version of em drivers can be found here:
# http://people.yandex-team.ru/~wawa/
#
# for additional em monitoring and statistics use 
# `sysctl dev.em.0.stats=1 ; dmesg`
#
#Same tunings for igb
#hw.igb.rxd=4096
#hw.igb.txd=4096
#hw.igb.rx_process_limit=100

# Some useful netisr tunables. See sysctl net.isr
#net.isr.defaultqlimit=4096
#net.isr.maxqlimit: 10240
# Bind netisr threads to CPUs
#net.isr.bindthreads=1

# Nicer boot logo =)
loader_logo="beastie"

And finally here is my additions to GENERIC kernel

# Just some of them, see also
# cat /sys/{i386,amd64,}/conf/NOTES

# This one useful only on i386
#options         KVA_PAGES=512

# You can play with HZ in environments with high interrupt rate (default is 1000) 
# 100 is for my notebook to prolong it's battery life
#options         HZ=100
# Polling is goot on network loads with high packet rates and low-end NICs
# NB! Do not enable it if you want more than one netisr thread
#options         DEVICE_POLLING

# Eliminate datacopy on socket read-write
# To take advantage with zero copy sockets you should have an MTU of 8K(amd64) 
# (4k for i386). This req. is only for receiving data.
# Read more in man zero_copy_sockets
#options         ZERO_COPY_SOCKETS

# Support TCP sign. Used for IPSec
options         TCP_SIGNATURE
options         IPSEC

# This ones can be loaded as modules. They described in loader.conf section     
#options         ACCEPT_FILTER_DATA
#options         ACCEPT_FILTER_HTTP

# Adding ipfw, also can be loaded as modules
options         IPFIREWALL
options         IPFIREWALL_VERBOSE
options         IPFIREWALL_VERBOSE_LIMIT=10
options         IPFIREWALL_DEFAULT_TO_ACCEPT
options         IPFIREWALL_FORWARD
# Adding kernel NAT
options         IPFIREWALL_NAT
options         LIBALIAS
# Traffic shaping
options         DUMMYNET          
# Divert, i.e. for userspace NAT
options         IPDIVERT

# This is for OpenBSD's pf firewall
device          pf
device          pflog
# pf's QoS - ALTQ
options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
options         ALTQ_NOPCC      # Required for SMP build

# Pretty console 
# Manual can be found here http://forums.freebsd.org/showthread.php?t=6134
#options         VESA
#options         SC_PIXEL_MODE

# Disable reboot on Ctrl Alt Del
#options         SC_DISABLE_REBOOT
# Change normal|kernel messages color
options         SC_NORM_ATTR=(FG_GREEN|BG_BLACK)
options         SC_KERNEL_CONS_ATTR=(FG_YELLOW|BG_BLACK)
# More scroll space
options         SC_HISTORY_SIZE=8192

# Adding hardware crypto device
device          crypto
device          cryptodev

# Useful network interfaces
device          vlan
device          tap                     #Virtual Ethernet driver
device          gre                     #IP over IP tunneling
device          if_bridge               #Bridge interface
device          pfsync                  #synchronization interface for PF
device          carp                    #Common Address Redundancy Protocol
device          enc                     #IPsec interface
device          lagg                    #Link aggregation interface
device          stf                     #IPv4-IPv6 port

# Also for my notebook, but may be used with Opteron
#device         amdtemp

# Support for ECMP. More than one route for destination
# Works even with default route so one can use it as LB for two ISP
# For now code is unstable and panics (panic: rtfree 2) on route deletions.
#options         RADIX_MPATH

# Multicast routing
#options         MROUTING
#options         PIM

# DTrace
options         KDTRACE_HOOKS        # all architectures - enable general DTrace hooks
options         DDB_CTF              # all architectures - kernel ELF linker loads CTF data
#options         KDTRACE_FRAME        # amd64-only

# Adaptive spining in lockmgr (8.x+)
# See http://www.mail-archive.com/[email protected]/msg10782.html
options         ADAPTIVE_LOCKMGRS

# UTF-8 in console (9.x+) 
#options         TEKEN_UTF8
#options         TEKEN_XTERM

# NCQ support
# WARNING! all ad[0-9]+ devices will be renamed to ada[0-9]+
#options         ATA_CAM

# FreeBSD 9+
# Deadlock resolver thread 
# For additional information see http://www.mail-archive.com/[email protected]/msg18124.html 
#options         DEADLKRES

PS. Also most of FreeBSD's limits can be monitored by

# vmstat -z

and

# limits  

PPS. variety of network counters can be monitored via

# netstat -s

In FreeBSD-9 netstat's -Q option appeared, try following command to display netisr stats

# netstat -Q

PPPS. also see

# man 7 tuning

PPPPS. I wanted to thank FreeBSD community, especially author of nginx - Igor Sysoev, nginx-ru@ and FreeBSD-performance@ mailing lists for providing useful information about FreeBSD tuning.

So here is the question:
What tunings are you using on yours FreeBSD servers?

You can also post your /etc/sysctl.conf, /boot/loader.conf, kernel options, etc with description of its' meaning (do not copy-paste from sysctl -d). Don't forget to specify server type (web, smb, gateway, etc)

Let's share experience!