Briefly on The Application of Ethernet RDMA Network Card

When all kinds of data are transmitted through network protocols (such as TCP/IP) in the network medium, if the amount of information is too large and not limited, then excessive network traffic will cause the device to respond slowly, resulting in Network delay.

The lower the latency, the better, and the higher the efficiency, the better. This is not only a requirement of the data center network, but also the internal network that we usually use. The current network speeds are 10 Gigabit, 25,000 Gigabit, 4 Gigabit, or even 100 Gigabit, 200 Gigabit... The trend has come, network bandwidth does not seem to be the main bottleneck, and the server system and CPU itself gradually In order to restrict the bottleneck of network I/O and affect the overall performance of the server.

1. How to solve the problem?
The basic idea to solve the problem is: directly read and write remote memory through the application program, without CPU intervention to make multiple copies of memory, and can bypass the kernel to directly write data to the network card, achieving high throughput and ultra-low latency And the effect of low CPU overhead.
The technology to realize such a function is RDMA (Remote Direct Memory Access) technology, and the RDMA [Remote Direct Data Access] technology is applied to the network card controller.
So what is the difference between a network card with RDMA function and one without RDMA?
First of all, the transmission path process of the network card without RDMA function is: application---> system---> memory---> CPU---> memory---> hard disk----> memory---> Network card.
The network card with RDMA function, when performing data transmission, the network card bypasses the CPU to achieve memory data exchange between servers: application ---> memory ---> hard disk ----> memory ---> network card. The process is greatly simplified, and the transmission efficiency has been significantly improved. 

Working diagram of network with RDMA function and network without RDMA function [picture from network]

(Green indicates the working path of RDMA, and orange indicates the working path without RDMA function)

As a hardware-implemented network transmission technology, RDMA can greatly improve the effectiveness of network transmission and help network IO-intensive services (such as distributed storage, distributed databases, etc.) to obtain lower latency and higher throughput.
At present, the network card products that can support RDMA function in the market are not popular. For example, the Intel network card that we are familiar with, only adds a single iWARP function to X722-da2/da4.
The three domestic 10 Gigabit fiber network cards launched by Shenzhen Lianrui Electronics Co., Ltd.: Lianrui LRES1004PF-2SFP+, Lianrui LRES1005PF-4SFP+, and Lianrui LRES1009PF-SFP+ all have the RDMA function of RoCE and iWARP, and have become the first domestic batch. One of the domestic network cards with RDMA function. Next, there will be 25G fiber optic network cards that support RDMA, we will wait and see. 

2. Application of RDMA

(1) Application on server cluster
RDMA is used to connect small servers into a cluster; it can handle large databases that only high-end servers with more than a dozen processors can handle.

(2) Application in high-speed cluster and server area network
Use RDMA to obtain high-performance protocols, including SDP (Sockets Direct Protocol), SRP (SCSI RDMA Protocol) and DAFS (Direct Access File System);

(3) Application on a cluster of distributed application programs
The communication library using RDMA, including DAPL (Direct Access Provider Library), MPI (Message Passing Interface) and VIPL (Virtual Interface Provider Library), running in a distributed application cluster is one of the areas where RDMA can show its talents;

By using RDMA through DAPL or VIPL and the database software running on the cluster, you can achieve higher performance and better scalability under the same number of nodes. The cluster technology computing application using MPI achieves low latency, low overhead, and high throughput by supporting interconnected RDMA. This result also enables it to achieve huge performance improvements in other early RDMA applications;
Technologies such as iWARP/RDMA are accessed through the remote file server of DAFS and the RDMA technology through the storage device of SRP has become the basic feature technology of high-speed cluster systems and storage area networks.
There is also iSER for iSCSI extension of RDMA, which fully utilizes the functions of RDMA.
Other early applications of RDMA include remote file server access via DAFS and blade server storage access via SRP.

(4) Application on NAS and SAN
The traditional direct-attached storage DAS is a server-centric storage structure. This storage architecture has insurmountable shortcomings such as capacity limitation, limited connection distance, and difficulty in sharing and management. Can no longer meet the application needs of the Internet era,
There have been huge changes in storage technology, such as network attached storage NAS and storage area network SAN. These can not only provide rich and fast storage resources for application systems on the network, but also share storage resources and implement centralized management of them, becoming the current ideal storage management and application mode.
However, there are some difficult problems in the NAS structure, such as limited transmission capacity, limited scalability, limited data backup capacity, and unable to provide effective support for database services.
DAFS integrates the advantages of RDMA with the storage capacity of NAS. All read and write operations are directly performed through the RDMA drive, thereby reducing the system load caused by the network file protocol. The NAS storage system in the future will use DAFS technology to Improve system performance and compete vigorously with SAN storage systems in performance and price.

(5) Application in high-performance data centers
Data center, AI training, cloud storage, etc.