So I have a box running a Windows Server 2008 R2 in my LAN, which is running, among other things, a voip server used by other people. I myself am in the same subnet with the server, while the other users are not.

Now, maybe on average twice a day, all the other users get disconnected from the server (and if I take a remote desktop connection to it, I'm not able to open any web pages either). These disconnections usually last only about 5 seconds, but sometimes they may last up to a minute. What could possibly cause this?

I have all the updates installed from windows update and the motherboard of the server is asus p5b deluxe (if that is of any interest).

edit: Forgot to mention that my own internet connectivity stays intact while the server's does not, and both computers are behind the same router.