How to transfer data between two netowks efficiently
- by Tono Nam
I will like to transfer files between two places over the internet. Right now I have a VPN and I am able to browse, download and transfer files. So my question is not really how to transfer the files; Instead, I will like to use the most efficient approach because the two places constantly share a lot of data.
The reason why I want to get rid of the vpn is because it is two slow. Having high upload speed is very expensive/impossible on residential places so I will like to use a different approach.
I was thinking about using programs such as http://www.dropbox.com . The problem with dropbox is it only enables 2 GB of storage in order for it to be free. I think the deals they offer are ok and I might be willing to pay to get that increase in speed. But I am concerned with the speed of transferring data. Dropbox will upload the file to their server then send it from the server to the other location. I will like it even faster lol.
Anyways I was thinking why not create a program my self. This is the algorithm that I was thinking let me know if it sounds to crazy.
(remember my goal is to transfer files as fastest as possible)
Things that I will use in this algorithm:
Server on the internet called S ( has fast download and upload speed. I pay to host a website and some services in there. I want to take advantage of it)
Client A on location 1
Client B on location 2
So lets say on location 1 20 large files are created and need to be transferred to location 2.
Client A compresses the files with the highest compression ratio possible.
Client A starts sending data via UDP to client B.
Because I am using UDP I will include the sequence number on each package.
Have server S help speed up things. For example every time a package is lost we can use Server S to inform client A that it needs to resend a package.
Anyways I think this approach will increase the transfer rate. I do not know if it is possible to start sending data meanwhile it is being compressed. Also if it is possible to start decompressing data even if we are not done receiving all the info. Maybe it will be faster to start sending the files right away without compressing. If I knew that I will always be sending large text files then I will obviously use the compression. I need this as a general algorithm.
So i guess my question is should using UDP over TCP could increase performance by using an extra server to keep track of lost packages? and How should I compress files before sending? compressing a 1 GB file with the highest compression ration takes about 1 hour! I will like to take advantage of that time by sending it meanwhile it is compressed.
