I am currently running a file sharing website where all files are stored remotely, they are hot linked on the download buttons (on the main site). The current .htaccess force downloads all files, images etc.

<FilesMatch "\.(?i:doc|odf|pdf|rtf|txt|png|jpg|jpeg|mp3|mp4|wav|wmv|gif|bmp|avi|mts)$">
  Header set Content-Disposition attachment
</FilesMatch>

What i am trying to do is make sure people cannot hot link the files. So i was thinking, is there a was i can block all other referrers to the domain that stores the files (it's an IP) apart from the main website (a domain).

Thanks!