First I should say that my Linux knowledge is minimal; just enough to set up some servers (Apache, Tomcat, Couch, etc). I built a MiniITX server to host some simple sites, act as an SSH tunnel while I'm away, and act as a torrent server. It was not properly secured for a long time (iptables was empty, all ports open, no firewall) though my router did not have much port forwarding set up beyond HTTP, FTP, and SSH.

A week or two ago my bandwidth at home dropped from around 27Mbps to 2Mbps and my upload went from 7Mbps to 0.06Mbps. When I unplug the server from the LAN, by bandwidth shoots back up.

I threw up a restrictive iptables, removed most of the port forwarding, and checked my router logs to see if there were any open connections from the server (malware?) but there were none.

What would you do? What are the first things you'd check?

I can of course reinstall everything from scratch, but I'd like to find the root cause.