The remote server requires a private key and passphrase for authentication. I tried putting username@server in the "Server" field and leaving "User name" and "Password" fields blank, putting the passphrase in the "Password" field, but it still says "Permission denied" and it doesn't ask for the private key at any point.

I also tried ssh-add path/to/privatekey, but it says "Could not open a connection to your authentication agent.", however I'm not sure if ssh-add is even relevant.

I can ssh into the server from the terminal just fine with the key/passphrase.