Friday, June 28, 2013

Fast directory transfer on Unix machines

Here is a little trick to transfer a big folder from one unix machine to another in 2 variations.

In this variation netcat is in listen mode on the target (execute in the given order):

on target> nc -l 19001 | lzop -d | tar x on source> tar c [directory to copy] | lzop | nc [target] 19001

In the second variation netcat is in listen mode on the source system (again, execute in the given order):

on source> tar c [directory to copy] | lzop | nc -l 19001 on target> nc [source] 19001 | lzop -d | tar x

Make sure you have a decent network connection, 1 Gbit/s is fine.

Update 2015-11-18: I experimented with cpio and found that it is a lot faster then tar. I also added pipe viewer (pv) to get some sense of when a transfer is done.

This is using cpio with netcat in listen mode on the target and send mode on the source:

on target> nc -l 19001 | lzop -d | cpio -idm on source> cd [directory to copy]; find . -depth -print0 | cpio -o0 \ | pv -s $(du -ks . | awk '{print $1}')k \ | lzop | nc [target] 19001

This is what it looks like on the source side:

No comments:

Post a Comment