Transferring data with On-The-Fly Compression Using SSH and SCP
March 9th, 2012
If you need to transfer large size files over very slow network, and if these files are not yet compressed, the following techniques effectively save your time and the bandwidth needed.
1. SCP Compress Option
SCP‘s -C option enables on-the-fly compression. Transfering an achive.log(/var/tmp/archive.log) using scp with compress option from a remote server to local:
2. On-the-fly compression using SSH and TAR
Compressing ‘archive.log’ with tar, sending its output to STDOUT, and then piping it from remote server to STDIN of local:
Speed Comparison
I tested them with and without on-the-fly compression and measured their real time. I transferred a large size file, archive.log which takes up about 1.1G disk usage from a remote server named “sakura” to local machine.
transfer without on-the-fly compression
executed: scp -p sakura:/var/tmp/archive.log /var/tmp
elapsed time: 9min 23sec
[case2]
transfer with on-the-fly compression using SCP’s -C option
executed: scp -C -p sakura:/var/tmp/archive.log /var/tmp
elapsed timed: 43 sec
[case3]
transfer with on-the-fly compression using SSH and TAR
executed: ssh sakura “tar zcpf – -C /var/tmp archive.log” | tar zxpf – -C /var/tmp
elapsed time: 36 sec
Do compress and make the most of your CPU power, then you can finish it In a shorter period of time!
No related posts.
Posted in: Random Topics
