We often see DBAs talking that the robocopy data transfer is very sluggish.
Recently I was involved in a project where 14 TB data was being transferred. My DBA informed me that the data transfer rate is just 325 KB/sec. With this rate of transfer it would take 1.4 years to transfer 14 TB of data. There was an absolute need fix either n\w design or robocopy. I chose to review robocopy command.
My DBA was using command with /Z option and without any /MT multi threading. The original command looked like as follows:
robocopy \\SourceServer\Sourcefolder \\DestinationServer\Destinationfolder /Z
I recommended following changes and confirm the speed
robocopy \\SourceServer\Sourcefolder \\DestinationServer\Destinationfolder /MT:64
/Z actually slows the speed a lot. This is absolutely great option to avoid any data discrepancy but I opted not to use this option and in case of any failures, the copy will be retried.
The speed was 595 MB/sec after this change. The copy time decreased from 1.4 years to whopping 6.8 hours. I am completely ok to retry this if any bit transfer is flipped because of n\w issue and causes corruption.
Recently I was involved in a project where 14 TB data was being transferred. My DBA informed me that the data transfer rate is just 325 KB/sec. With this rate of transfer it would take 1.4 years to transfer 14 TB of data. There was an absolute need fix either n\w design or robocopy. I chose to review robocopy command.
My DBA was using command with /Z option and without any /MT multi threading. The original command looked like as follows:
robocopy \\SourceServer\Sourcefolder \\DestinationServer\Destinationfolder /Z
I recommended following changes and confirm the speed
robocopy \\SourceServer\Sourcefolder \\DestinationServer\Destinationfolder /MT:64
/Z actually slows the speed a lot. This is absolutely great option to avoid any data discrepancy but I opted not to use this option and in case of any failures, the copy will be retried.
The speed was 595 MB/sec after this change. The copy time decreased from 1.4 years to whopping 6.8 hours. I am completely ok to retry this if any bit transfer is flipped because of n\w issue and causes corruption.
No comments:
Post a Comment