FTP transfers can seem to slow down and grind to a halt and you never reach the ‘line speed’ of you connection. This can be due to latency or bandwidth throttling. Latency is the amount of time it takes a packet to get from source to destination and the further the distance, the greater the latency. During a file transfer, ‘messages’ are sent back and forth between source and destination, and while the file transfer can often be good, the confirmation takes time and slows down the process. Therefore there will be slack in your use of bandwidth.
To mitigate this, a file can be transferred in several segments simultaneously. Or several files can be transferred at the same time, filling the slack.
In this way you can use lftp as a download accelerator.
lftp has pget command.
It allows you download files in parallel. lftp pget command syntax:
lftp -e ‘pget -n 5 ftp://path/to/file’
For example, download http://kernel.org/pub/linux/kernel/v2.6/linux-220.127.116.11.tar.bz2 file using pget in 5 parallel parts:
$ cd /tmp
$ lftp -e ‘pget -n 5 http://kernel.org/pub/linux/kernel/v2.6/linux-18.104.22.168.tar.bz2′
We created a bespoke DR facility for one of our customers using LFTP. Database and application data was replicated across a pan-European WAN. Replication throughput was more than doubled over the previous system. I’ll document that solution in more detail another time.