Limiting Disk IO on Linux with ionice
One solution to this is ionice which lets you set the IO priority of processes. It's kind of like regular nice but for IO specifically. I found that the follow command, which sets the scheduling class to be 'Idle', let my desktop remain completely responsive while the transfer was taking place.
ionice -c 3 cp $src_file $dst_directory