For some reasons I cant access any of my accounts on my dedicated server via FTP. It simply times out when it tried to display the directories.

Heres a log from FileZila...

Status: Resolving address of testdomain.com
Status: Connecting to 64.237.58.43:21...
Status: Connection established, waiting for welcome message...
Response:   220---------- Welcome to Pure-FTPd [TLS] ----------
Response:   220-You are user number 3 of 50 allowed.
Response:   220-Local time is now 19:39. Server port: 21.
Response:   220-This is a private system - No anonymous login
Response:   220-IPv6 connections are also welcome on this server.
Response:   220 You will be disconnected after 15 minutes of inactivity.
Command:    USER testaccount
Response:   331 User testaccount OK. Password required
Command:    PASS ********
Response:   230-User testaccount has group access to:  testaccount
Response:   230 OK. Current restricted directory is /
Command:    SYST
Response:   215 UNIX Type: L8
Command:    FEAT
Response:   211-Extensions supported:
Response:    EPRT
Response:    IDLE
Response:    MDTM
Response:    SIZE
Response:    REST STREAM
Response:    MLST type*;size*;sizd*;modify*;UNIX.mode*;UNIX.uid*;UNIX.gid*;unique*;
Response:    MLSD
Response:    ESTP
Response:    PASV
Response:    EPSV
Response:    SPSV
Response:    ESTA
Response:    AUTH TLS
Response:    PBSZ
Response:    PROT
Response:   211 End.
Status: Connected
Status: Retrieving directory listing...
Command:    PWD
Response:   257 "/" is your current location
Command:    TYPE I
Response:   200 TYPE is now 8-bit binary
Command:    PASV
Response:   227 Entering Passive Mode (64,237,58,43,145,153)
Command:    MLSD
Response:   150 Accepted data connection
Response:   226-ASCII
Response:   226-Options: -a -l 
Response:   226 18 matches total
Error:  Connection timed out
Error:  Failed to retrieve directory listing

I have restarted the FTP service serveral times but still It doesnt loads. I only have this problem when my server is reaching it peak usage which is still only 1.0 (4 cores), 40% of 4GB ram.

The ftp connections isnt maxed out because only me and my colleague have access to FTP on the server.