Research and Education (R&E) networks form the underlying “conduit system” for research data movement. Unlike plumbing that directs water, the act of sharing data sets with remote collaborators often involves a deep understanding of the proper hardware, software, and configuration parameters that must be used in order to ensure the operation performs well. This complexity can be frustrating for network users, resulting in lost productivity due to inadequate performance or may cause a mistrust of the available technology to perform routine tasks.
Data mobility is a critical component of the process of science. Being able to predictably and efficiently move data between experimental source, processing facilities, long-term storage and collaborators, is a common use case that transcends the boundaries of research disciplines.
The Energy Sciences Network (ESnet), in collaboration with Indiana University (IU) via the Engagement and Performance Operations Center (EPOC) are working to alleviate this problem.
By considering the full end-to-end data movement pipeline, ESnet and partners are able to uniquely support collaborative science, allowing researchers to make the most effective use of networks to facilitate sharing research data in order to accelerate the scientific discovery process. This program, entitled the “Data Mobility Exhibition,” or DME, is an outcome-oriented approach to measuring and improving the experience of data sharing.
The core question the DME program attempts to answer is: how long does it take a site to transfer 1TB of data? This is answered through performing a set of well-defined tests from each site’s data movement capabilities to a set of supported endpoints at partner R&E sites. The DME goals are flexible, but have set a baseline of having all 10 Gbps capable devices reaching a minimum performance measurement of 1TB/hour performance (e.g., 2.22Gbps, when measured across the wide area network and involving the disk systems of each participating site).
Since the program began in 2019, we have worked with approximately 50 sites to validate upload performance and over 100 sites to evaluate download performance; this testing has determined a community wide average of 3.5 Gbps on uploads and 1.8 Gbps for downloads. Some larger facilities are capable of delivering nearly 70 Gbps routinely and for those that have requested assistance, we have often achieved performance increases of an order of magnitude (e.g., from initial speeds below 1 Gbps to near 10 Gbps).
ESnet and our partners are continuing to work with The Quilt to expand this program to all NSF CC* awardees, campus networks of all sizes, and regional networking exchanges.
For further information, contact [engage at es dot net] or visit us at http://fasterdata.es.net.
The Energy Sciences Network (ESnet) is a high-performance, unclassified network built to support scientific research. Funded by the U.S. Department of Energy’s Office of Science (SC) and managed by Lawrence Berkeley National Laboratory, ESnet provides services to more than 50 DOE research sites, including the entire National Laboratory system, its supercomputing facilities, and its major scientific instruments. ESnet also connects to 140+ research and commercial networks, enabling DOE-funded scientists to productively collaborate with partners around the world.