A new, simple, tool for faster transport of large data sets

Harro Verkouter (JIVE, the Netherlands), with help from Stuart Weston (Auckland University of Technology, New Zealand) - ASTERICS Work Package 5

The ESFRI facilities in astronomy and astro(particle) physics will generate large amounts of data. SKA, for example, will produce an amount of data similar to five times the current global internet traffic. This data needs to be transferred over large distances from the facility sites to research institutes. The current data channel protocol Transmission Control Protocol (TCP) was developed in 1981 and is not fast enough for the transfer of large amounts of data. Therefore, software was developed that allowed the use of User Datagram Protocol (UPD)-based Data Transfer (UDT) protocol, which has a higher data transfer speed than TCP. Tests were done to transfer a large data amounts (20 GB and 80 GB) from the Netherlands to New Zealand using the newly developed software. The transfer rate became three times faster: from 0.9 GB per second using TCP (at 8:53 in the figure) to 2.8 GB per second using UTP (at 9:28 in the figure).