]> China Mobile

yaokehan@chinamobile.com

ZTE Corporation

xiong.quan@zte.com.cn

hpwan Bulk data transfer is a long-lived service over the past twenty years. High Performance Wide Area Networks (HP-WANs) are the backbone of global network infrastructure, enabling the seamless transfer of vast amounts of data and supporting advanced scientific collaborations worldwide. Many of the state-of-the-art dedicated networks have been mentioned in . For non-dedicated networks like public operator's network, the case is different in terms of QoS policies, security policies, etc. This document presents use cases and requirements of HPWAN from public operator's view.

Introduction Bulk data transfer is a long-lived service over the past twenty years. Some dedicated networks have been designed to carry these kind of data transfer services, like CERN, internet2, etc. Many of the state-of-the-art dedicated networks have been mentioned in . In these dedicated networks, policies and network provisioning are all designed specifically for bulk data transfer services, which means these services have high SLA guarantee. In non-dedicated networks, for example, the operator's networks, large amount of data transfer service has grown quickly, with the increase of network bandwidth. The difference is that, in these non-dedicated networks, bulk data transfer flows need to share bandwidth with Internet traffic, which inevitably leads to resource contention and requires further planning and protocol optimization. When there are multiple data transfer requests coming to the operator's network, the scheduling of different data transfer flows will influence the completion time for each job. The scheduling primarily comes from two aspects. One is the job scheduling or orchestration, that is, at the service orchestrator, different data transmission requests need to be scheduled on bandwidth reservation and its corresponding time occupancy, according to the priorities of each job. The other is the traffic scheduling during data transmission. The second aspect needs coordination of both routing and transport technologies. The routing issues related to bulk data transfer include traffic engineering, and load balancing, etc. While transport issues cover more, including flow control, congestion control, admission control, and proxy. Due to the characteristics of a shared network like public operator's network, long-distance transmission of bulk data with packet loss rate over 0.1% and traffic contention, transport optimizations are key for bulk data transmission services. This document presents some typical bulk data transfer use cases the non-dedicated networks currently deploy, and propose some requirements for transport layer.

Definition of Terms This document uses the following terms defined in :

• High Performance Wide Area Network (HPWAN)
• Remote direct memory access (RDMA)

Use Cases

Large File Transfer Over Operator's Shared Network Astronomy and biology data have been growing fastly, with the development of advanced instruments for data collection. For example, The volume of the data generation per year of an astronomical observatory in China is around 500 terabytes(TBs), and there are around 200 observation jobs, resulting in an average transmission volume of 2.5 TB per job. The customer (astronomical observatory) currently rents leased line (10Gbps) from public operator like China Mobile. Considering there are other background traffic, the real transmission speed of an astronomical observatory job is 7Gbps, leading to a total job completion time around 47.6 minutes. Customers want to reduce cost, and they want to be charged by the time of bandwidth occupancy, rather than renting leased line every month. In the current operator's shared network, like China Mobile backbone network (CMnet), the average packet loss rate is around 0.1%, in some cases, the packet loss rate will increase to 0.5%. Customers currently use upper layer protocol like FTP or data transfer tools like RSYNC , the underlying transport protocol is TCP. TCP is sensitive to packet loss, especially in long range transmission, e.g., over 2000 kilometers(km). Under such circumstance, the transmission rate will sharply decrease to around hundreds of Mbps, e.g, 700 Mbps, leading to an increase of job completion time to 476 minutes, which can not be accepted by customers. Therefore, operators want to optimize transport behaviors to reduce the overall data transmission duration. To sum up, in the case of large file transfer over operator's shared network, the transport protocol is TCP, the typical volume of transmission job is several TBs, the completion time is required to be within several hours. The packet loss rate is around 0.1% to 0.5%. The objective is to increase the transmission rate over long range like 2000 km.

Time Constrained Traffic Across Data Centers Another case is the traffic between large data centers which are owned by public operators. Traditionally, the primary traffic across data centers is cloud file backup. Cloud file backup may have similar requirements like the case mentioned in the previous section. Currently, data centers have advanced to accommodate more artificial intelligence (AI) jobs, like AI training and inference. Cross data center traffic pattern has emerged to carry more AI job traffic, for example, cross data center training. With rapid development of large language models (LLMs), more compute instances need to be deployed in data centers, approaching to the energy and physical space limit of each single data center. Under such circumstance, operators try to train models across data centers over long distance. According to the parallel computing strategies, there are three major types, data parallel(DP), tensor parallel(TP), and pipeline parallel(PP). TP consumes more bandwidth resource, e.g., hundreds of MB for a single flow and , and has strict requirements on latency, e.g., microseconds level. therefore, TP is primarily deployed within data centers. While PP and DP consumes less bandwidth than TP, and less strict latency requirements than TP. Therefore, operators consider to deploy PP and DP traffic across data centers. For example, considering training a LLM with tens of billions of parameters like Llama3 with a thousand of Graphics Processing Units (GPUs) across two data centers, that is, deploying around 500 GPUs in each data center. The bandwidth of a single network interface card (NIC) is 200Gbps, and the total bandwidth requirements is 102.4Tbps. Under hybrid parallel computing strategies, the total volume of the cross data center traffic is the product of the number of TPs, the number of PPs, and the GPU bandwidth, which equals to 12.8Tb. These traffic is not time sensitive to microseconds level, but still needs to be transmitted as quickly as possible, thereby the overall training efficiency can be improved. In this case, the transport protocol is based on Remote Direct Memory Access (RDMA). Traditionally, in controlled environments like data centers, these traffic is implemented over RDMA over Converged Ethernet version 2 (RoCEv2). However, since these traffic will be transmitted over shared network or even Internet, the performance as well as security both require guarantee. The iWARP protocol suite works over TCP and can provide security guarantee, but its performance in throughput optimizations need further improvement.

Sharing Traffic Between Dedicated Network and Non-dedicated Network This case is more on sharing bulk data transfer service between dedicated network and non-dedicated network. Institutes and Universities are usually connected to dedicated networks like education and research networks, but data needs to be spread across public operator's networks to remote consumers. In this case, there exist some difference regarding to QoS and security policies. In dedicated networks, traffic types can be relatively simple, and the QoS mechanisms like bandwidth and priority guarantee can be flexibly adjusted. While in non-dedicated networks, the QoS mechanisms might not be the same as dedicated networks when traffic types vary a lot. Priorities of different flows may change when traffic are transmitted across a dedicated network and another non-dedicated network. Meanwhile, the security policies differ in two separate domains. The firewall or traffic shaper deployed in each domain may have some access requirements for traffic from other networks. In this case, policies related to transport layer need further coordination between two different networks, especially on QoS guarantee and security.

Summarization of the Characteristics of Use Cases This section summarizes the differences of the three aforementioned use cases from the following aspects. shows the comparison.

• What transport protocols they use?
• How large are the volume of the data that need to be transmitted?
• What are the time constraints?
• What security aspect do they care?
• What data transfer applications are currently been in use?

The basic performance indicators of the shared operator network environment is as follows.

• Packet loss rate, around 0.1%.
• Number of hops, 5 to 20.
• Transmission distance, over 1000 km.
• bandwidth, 1 to 10 Gbps at the access network, 10Gbps to 100Gbps at the core network.

Comparison among Use Cases

Requirements According to the three different use cases mentioned in the previous section, some requirements on transport layer are proposed in this section. As mentioned above, in different scenarios, there are different transport protocols carrying bulk data transfer services. For example, TCP, QUIC, RDMA, iWARP. Traffic with different transport protocols may run over the same public operator's network. Optimizing each transport protocol may incur much overhead, including congestion control algorithms design and parameter tuning, hardware adaptation, QoS policies, etc. To avoid such overhead, operators think about more flexible deployment solutions and propose some new requirements on transport protocol proxy. R1: It MUST support transport protocol proxy which adapts one transport protocol to another, for example, TCP to iWARP or QUIC to iWARP. This is to guarantee that the transport layer mechanisms like congestion control and flow control remain consistent between two proxies, when applications at endpoints run different transport protocols. R2: The proxy MUST support traffic classification, flows or sessions that need acceleration (increase priorities or guarantee bandwidth) can be selected at the proxy, while some other normal flow can be transmitted transparently. R3: The proxy SHOULD support the aggregation of some mouse flows or the split of an elephant flow into multiple flows, based on the traffic classification. When implementing iWARP in the proxy, the hardware resource needs to be considered to guarantee transmission performance. Some work may choose to implement simplified iWARP stack in the proxy, therefore, R4: If iWARP is selected as the transport protocol, SHOULD support the transform of operation types, for example, from Unreliable Delivery (UD) mode to Reliable Connection (RC) mode. Congestion control is always an important issue for data transmission over shared network where congestion might not be precisely located and predicted. Usually, HPWAN flows are large and may incur congestions into the network. There are already many congestion control algorithms that have been standardized or being standardized, like , , and . But they have poor convergence speed based on blind transmission with rate adjusting due to the unpredictable behaviour of non-dedicated network. The network should collaborate with the client to perform active congestion avoidance such as resource scheduling, rate-based acknowledgement to achieve the completion time. The proxy can get information (topology, link bandwidth, queue and buffer) from a centralized controller of the WAN, for example, a SDN controller. The proxy can serve as an agent of the SDN controller of each network domain, and can exchange information with clients. The proxy can also serve as an intermediate node for congestion information sharing with clients. R5: It MUST support signaling from client to the proxy, including the QoS guarantee request for scheduled traffic, for example, bandwidth, traffic pattern. Then the transport protocol proxy can negotiate with the SDN controller to design more flexible QoS policies and the corresponding resource planning (like bandwidth) and traffic scheduling. R6: It MUST support signaling from the proxy to the client, including the response of negotiated rate for the client to send traffic and the fast and accurate quantitative feedback when proxy performs active admission control. Some of the use cases mentioned in section three have strong security requirements, for example, biology data, financial data, and AI training data, etc. Therefore, R7: The data integrity MUST be guaranteed for these services, especially transferring data over non-dedicated network where data may be exposed to attacks. Also the security policies may differ when traffic is transmitted over dedicated network as well as non-dedicated network. Thus, R8: The security policies of dedicated network as well as non-dedicated network SHOULD be exchanged, although this might not be implemented on transport layer. A practical approach is to coordinate at the orchestrator or exchange information through border gateways.

Security Considerations TBD.

IANA Considerations TBD.

References Normative References CUBIC is a standard TCP congestion control algorithm that uses a cubic function instead of a linear congestion window increase function to improve scalability and stability over fast and long-distance networks. CUBIC has been adopted as the default TCP congestion control algorithm by the Linux, Windows, and Apple stacks. This document updates the specification of CUBIC to include algorithmic improvements based on these implementations and recent academic work. Based on the extensive deployment experience with CUBIC, this document also moves the specification to the Standards Track and obsoletes RFC 8312. This document also updates RFC 5681, to allow for CUBIC's occasionally more aggressive sending behavior. This document describes the L4S architecture, which enables Internet applications to achieve low queuing latency, low congestion loss, and scalable throughput control. L4S is based on the insight that the root cause of queuing delay is in the capacity-seeking congestion controllers of senders, not in the queue itself. With the L4S architecture, all Internet applications could (but do not have to) transition away from congestion control algorithms that cause substantial queuing delay and instead adopt a new class of congestion controls that can seek capacity with very little queuing. These are aided by a modified form of Explicit Congestion Notification (ECN) from the network. With this new architecture, applications can have both low latency and high throughput. The architecture primarily concerns incremental deployment. It defines mechanisms that allow the new class of L4S congestion controls to coexist with 'Classic' congestion controls in a shared network. The aim is for L4S latency and throughput to be usually much better (and rarely worse) while typically not impacting Classic performance. This memo is the official specification of the File Transfer Protocol (FTP) for the DARPA Internet community. The primary intent is to clarify and correct the documentation of the FTP specification, not to change the protocol. The following new optional commands are included in this edition of the specification: Change to Parent Directory (CDUP), Structure Mount (SMNT), Store Unique (STOU), Remove Directory (RMD), Make Directory (MKD), Print Directory (PWD), and System (SYST). Note that this specification is compatible with the previous edition. Informative References Lancaster University Jisc Pittsburgh Supercomputing Center ZTE Corporation High Performance Wide Area Networks (HP-WANs) represent a critical infrastructure for the modern global research and education community, facilitating collaboration across national and international boundaries. These networks, such as Janet, ESnet, GÉANT, Internet2, CANARIE, and others, are designed to support the general needs of the research and education users they serve but also the the transmission of vast amounts of data generated by scientific research, high-performance computing, distributed AI-training and large-scale simulations. This document provides an overview of the terminology and techniques used for existing HP-WANS. It also explores the technological advancements, operational tools, and future directions for HP-WANs, emphasising their role in enabling cutting-edge scientific research, big data analysis, AI training and massive industrial data analysis. Google Google Meta This document specifies the BBR congestion control algorithm. BBR ("Bottleneck Bandwidth and Round-trip propagation time") uses recent measurements of a transport connection's delivery rate, round-trip time, and packet loss rate to build an explicit model of the network path. BBR then uses this model to control both how fast it sends data and the maximum volume of data it allows in flight in the network at any time. Relative to loss-based congestion control algorithms such as Reno [RFC5681] or CUBIC [RFC9438], BBR offers substantially higher throughput for bottlenecks with shallow buffers or random losses, and substantially lower queueing delays for bottlenecks with deep buffers (avoiding "bufferbloat"). BBR can be implemented in any transport protocol that supports packet-delivery acknowledgment. Thus far, open source implementations are available for TCP [RFC9293] and QUIC [RFC9000]. This document specifies version 3 of the BBR algorithm, BBRv3. n.d.

Contributors China Mobile

yanghongwei@chinamobile.com

ZTE Corporation

huang.guangping@zte.com.cn