Tuesday, September 13, 2016

Fastest way to copy data from one directory to another

Using tar(1) or cpio(1) is by and large the fastest way of copying large amounts of data from one place to another, especially when the data is unstructured (many small files as opposed to a few large ones). Setting the block size is key, and here's what generally works the best in my experience:
# cd parent_directory
# tar Ecbf 256 - source_data | (cd /target_directory/ ; tar xf - );
In this example, we're copying /parent_directory/source_data to /target_directory/source_data, where source_data is usually a directory.

Notice that this can easily be converted to a for statement that iterates parent_directory and starts a tar session for each of its subdirectories.

No comments:

Post a Comment