DetNet Yufang. Han Internet-Draft Shaofu. Peng Intended status: Standards Track ZTE Corporation Expires: 13 December 2023 Yuehong. Gao Beijing University of Posts and Telecommunications 11 June 2023 Analysis and Evaluation for TSN Queuing Mechanisms draft-hp-detnet-tsn-queuing-mechanisms-evaluation-00 Abstract TSN technology standards developed in the IEEE 802.1TSN Task Group define the time-sensitive mechanism to provide deterministic connectivity through IEEE 802 networks, i.e., guaranteed packet transport with bounded latency, low packet delay variation, and low packet loss.This document summarizes and evaluates various queuing technologies of TSN as reference information for Scaling Deterministic Networks Requirements(described in draft-ietf-detnet- scaling-requirements) and Enhancing Deterministic Forwarding. Status of This Memo This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79. Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/. Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress." This Internet-Draft will expire on 13 December 2023. Copyright Notice Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved. This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights Han, et al. Expires 13 December 2023 [Page 1] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License. Table of Contents 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 3 2. TSN queuing and shaping technologies . . . . . . . . . . . . 3 2.1. Frame Preemption . . . . . . . . . . . . . . . . . . . . 3 2.1.1. Frame Preemption Overview . . . . . . . . . . . . . . 3 2.1.2. Frame Preemption Analysis . . . . . . . . . . . . . . 4 2.2. CBS(Credit-Based Shaper) . . . . . . . . . . . . . . . . 4 2.2.1. CBS(CBS Overview) . . . . . . . . . . . . . . . . . . 4 2.2.2. CBS Analysis . . . . . . . . . . . . . . . . . . . . 5 2.3. TAS(Time-Aware Shaping) . . . . . . . . . . . . . . . . . 5 2.3.1. TAS Overview . . . . . . . . . . . . . . . . . . . . 5 2.3.2. TAS Analysis . . . . . . . . . . . . . . . . . . . . 7 2.4. CQF(Cyclic Queuing and Forwarding) . . . . . . . . . . . 7 2.4.1. CQF Overview . . . . . . . . . . . . . . . . . . . . 7 2.4.2. CQF Analysis . . . . . . . . . . . . . . . . . . . . 8 2.5. ATS(Asynchronous Traffic Shaping) . . . . . . . . . . . . 9 2.5.1. ATS Overview . . . . . . . . . . . . . . . . . . . . 9 2.5.2. ATS Analysis . . . . . . . . . . . . . . . . . . . . 10 3. Evaluation of TSN queuing mechanism with the requirements of scaling Deterministic networks . . . . . . . . . . . . . 11 3.1. Tolerate Time Asynchrony . . . . . . . . . . . . . . . . 11 3.2. Support Large Single-hop Propagation Latency . . . . . . 11 3.3. Accommodate the Higher Link Speed . . . . . . . . . . . . 12 3.4. Be Scalable to The Large Number of Flows . . . . . . . . 12 3.5. Tolerate High Utilization . . . . . . . . . . . . . . . . 12 3.6. Prevent Flow Fluctuation from Disrupting Service . . . . 13 3.7. Be Scalabcle to a Large Number of Hops with Complex Topology . . . . . . . . . . . . . . . . . . . . . . . . 13 3.8. Tolerate Failures of Links or Nodes and Topology changes . . . . . . . . . . . . . . . . . . . . . . . . . 13 3.9. Support Multi-Mechanisms in Single Domain and Multi-Domains . . . . . . . . . . . . . . . . . . . . . . 13 4. Evaluation results . . . . . . . . . . . . . . . . . . . . . 14 5. Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . 14 6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 15 7. Security Considerations . . . . . . . . . . . . . . . . . . . 15 8. Acknowledgements . . . . . . . . . . . . . . . . . . . . . . 15 9. References . . . . . . . . . . . . . . . . . . . . . . . . . 15 9.1. Normative References . . . . . . . . . . . . . . . . . . 15 9.2. Informative References . . . . . . . . . . . . . . . . . 16 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 17 Han, et al. Expires 13 December 2023 [Page 2] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 1. Introduction Time sensitive networking (TSN) makes it possible to carry data traffic of time-critical and/or mission- critical applications over a bridged Ethernet network shared by various kinds of applications with different Quality of Service(QoS) requirements, i.e., time and/or mission critical TSN traffic and non-TSN best effort traffic. TSN provides guaranteed data transport with bounded low latency, low delay variation, and extremely low data loss for time and/or mission critical traffic. By reserving resources for critical traffic, and applying various queuing and shaping techniques, TSN guarantees a worst-case end-to-end latency for critical data,and achieves zero congestion loss for critical data traffic. TSN also provides ultra- reliability for data traffic via a data packet level reliability mechanism as well as protection against bandwidth violation, malfunctioning, malicious attacks, etc. At present, TSN series standards are basically mature and provide queuing or scheduling algorithms that support different delay accuracies, such as frame preemption ([IEEE802.3br] and [IEEE802.1Qbu]), CBS ([IEEE802.1Qav]), CQF ([IEEE802.1Qch]), TAS ([IEEE802.1Qbv]), ATS ([IEEE802.1Qcr]), etc. These mechanisms provide QoS capabilities for different application scenarios, such as CBS guarantees the upper bound of latency while ensuring rate, ATS provides low latency services for emergency flows.These two can be classified as mechanisms with bounded latency.CQF can provide delay jitter independent of the number of hops, while TAS can provide extremely low jitter through precise calculations. These two can be classified as mechanisms for jitter control. The following sections will analyze these queueing technologies one by one. 2. TSN queuing and shaping technologies 2.1. Frame Preemption 2.1.1. Frame Preemption Overview Frame preemption mechanism was introduced to mitigate negative effects of the guard band reserved by the TAS. As it requires modifications of both management (IEEE 802.1) and Ethernet MAC (IEEE 802.3) functions, two working groups jointly proposed required changes to both standards. Therefore, the frame preemption is described in two different standard documents: [IEEE802.1Qbu] and [IEEE802.3br]. [IEEE802.3br], also named Interspersing Express Traffic, differentiates two types of traffic: preemptable (also called mPacket) and express. The type of a frame is identified by examining Han, et al. Expires 13 December 2023 [Page 3] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 the VLAN tag defined by IEEE 802.1Q. Frames arriving from the MAC client are serviced either by preemptable MAC (pMAC) or express MAC (eMAC). If both frames arrive at the same time, express traffic is serviced first as it has higher priority. In the case when express frame arrives while preemptable frame is already being transmitted on egress port, if certain conditions are met, it will interrupt current transmission . After express traffic has been serviced, the transmission of interrupted frame is resumed and different parts of the interrupted frame are re-assembled by a MAC Merge Sublayer (MMS) that is a part of the modified Ethernet MAC which supports frame preemption. It is important to note that frame fragmentation works on link-by- link basis, i.e., each switch forwards preemptable frame only after it is fully re-assembled. This is clearly different from end-to-end packet fragmentation that is commonly used in IP networks. This ensures compatibility with the devices that do not support frame preemption mechanism. 2.1.2. Frame Preemption Analysis As explained earlier, the main motivation behind the frame preemption mechanism is to reduce the length of guard band enforced by the TAS. Without frame preemption, reserved guard band must match the transmission time of the largest low-priority frame. In the case of 100 Mbps Ethernet, the worst-case time would be around 125us (transmission time of the largest Ethernet frame), which represents a huge bandwidth penalty.Frame preemption allows reducing the guard band down to approximately 12us which is tenfold improvement.It can also be combined with other queue technologies to minimize the interference delay from low priority packets. 2.2. CBS(Credit-Based Shaper) 2.2.1. CBS(CBS Overview) CBS proposed by [IEEE802.1Qav] divides time-sensitive services which need to be transmitted preferentially into two classes: class A and class B, and sets a certain bandwidth for them . Through priority mapping, TSN flow with different priorities enter different queues for scheduling respectively . As described in Section 8.6.8.2 of [IEEE802.1Qav], the credits of each class A or Class B increase according to the idle slope (as the guaranteed rate), and decrease according to the send slope (usually equal to idle slop minus port transmit rate), both of which are parameters of the CBS. Han, et al. Expires 13 December 2023 [Page 4] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 TSN flows are gently sent to the network by credit evaluation to deal with data burst and aggregation, CBS can limit burst traffic and prevent audio and video streams arriving at the same time from different terminals,which generates significant buffering congestion, resulting in packet loss. 2.2.2. CBS Analysis CBS set the pre-configuration of bandwidth limit for each traffic class.Typically set 75% of the maximum bandwidth for bandwidth intensive applications such as audio and video. CBS does not rely on time synchronization or frequency synchronization technology, so it can be applied in scenarios such as cross clock domains, non strict time synchronization, and asynchronous clocks. The disadvantage of CBS is that the average latency will increase , although the combination of CBS and SRP (StreamReservedProtocol Stream Reservation Protocol) can limit the latency of each bridge to less than 250us. The paper[AVB-Latency] analyzes that in small-scale networks using FE (Fast Ethernet, 100Mbps) ports, CBS can guarantees a worst-case latency of less than 2 milliseconds for Class A and less than 50 milliseconds for Class B under a maximum of 7 hops. However, other papers[ClassA-Latency-Calc] shows the conclusion is not valid,it indicates that there is still a problem of delay degradation in CBS when catching up with burst flows occur. In general, the more hops, the worse the delay degradation . In large-scale networks, the number of network hops is usually large, such as 15 or more hops, which poses great challenges for the deployment of CBS. The upper bound of latency can not meet the requirements of many services which need low latency. 2.3. TAS(Time-Aware Shaping) 2.3.1. TAS Overview In industrial IoT application scenarios, some time-sensitive streams will carry critical information. These streams require highly predictable delay and jitter in transmission. If the delay or jitter exceeds the threshold, it may cause serious consequences.At the same time,most of these streams are transmitted according to a certain time period,and streams with this characteristic are called Scheduled Traffic. For the Scheduled Traffic, CBS transmission algorithm can not meet the requirements, because in CBS algorithm, if a low priority frame is already being transmitted, then that transmission will complete Han, et al. Expires 13 December 2023 [Page 5] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 before a higher priority frame can access the transmission medium, so there could be a delay of up to a maximum-sized frame before a high priority transmission can start. If such delays occur at every hop, then the accumulated latency could be unacceptably large. To address this issue, [IEEE802.1Qbv] proposes the TAS mechanism. As Scheduled Traffic is a periodic stream, it is possible to determine the time when each packet of streams arrives at each network device after Scheduled Traffic starts to transmit. As long as sufficient bandwidth is reserved for Scheduled Traffic on these devices in advance, it can ensure that other non-scheduled traffic will not interfere with the transmission of Scheduled Traffic. TAS provides a scheduling mechanism of gate operations, which is based on high-precision clock synchronization. Each port of the TSN bridge has a gate control list for opening or closing operations, and the 8 queues at these ports need to be associated with each of the 8 Transmission Gates respectively. Each entry in the gate control list corresponds to a gate operation, and then packet are selected from the queue for transmission based on the gate control list. The gate control list contains two items: GateState and Timelnterval. GateState is used to set the state of Transmission Gate corresponding to queues , there are two states for each Transmission Gate:Open and Closed."Open" means that packets in the associated queue can be transmitted according to the corresponding transmission algorithm, while "Closed" means that packets in the associated queue are not allowed to be transmitted.TimeInterval indicates the duration of the gate state. After TimeInterval ticks have elapsed since the completion of the previous gate operation in the gate control list, control passes to the next gate operation. Since transmission operation is an "atomic operation", in order to avoid the situation that the packet in corresponding queue can not be completely transmitted before the gate is closed,TAS defines an advance check mechanism. If a packet cannot be fully transmitted within the remaining time of the corresponding gate operation state is open, this packet will not be transmitted until the next gate is opened. In order to ensure that the remaining non-scheduled traffic cannot affect the transmission of scheduled traffic, TAS uses a guard band (Guard Band) mechanism to stop the transmission of non-scheduled traffic sufficiently far in advance of the protected time slot to be certain that the last non-scheduled transmission has completed before scheduled traffic transmission starts. In the worst case, this would mean that the last non-scheduled transmission would start a maximum- sized frame transmission time before the start of the scheduled traffic "window". In effect, a guard band is created before the time Han, et al. Expires 13 December 2023 [Page 6] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 that the scheduled traffic transmission is due to start; transmission of non-scheduled traffic is not permitted between the start of the guard band and the start of the scheduled traffic window. The simplest approach is for the guard band to be as long as a maximum- sized frame transmission. 2.3.2. TAS Analysis The premise of TAS is that all terminals and network devices need to achieve nanosecond clock synchronization across the network (such as [IEEE1588],[IEEE802.1AS])to ensure that the GCL time of all outgoing ports is synchronized.Appropriate transmission "windows" can be arranged for the scheduled traffic at each egress port to achieve that the traffic can obtain extremely low transmission delay by accurate calculation. But when the network topology scale is large, that is, There may be a large number of nodes and links , it is usually difficult to achieve real-time synchronization, that limits the deployment of TAS; At the same time, large-scale networks carry a massive number of application flows, and their network topology is also more complex, which will be a great challenge for TAS that relies on precise calculations and complex configurations. On the other hand, the transmission window reserved for deterministic flows through GCL is usually exclusive. During the time period when the gate state of the queue associated with scheduled flows is open, even if the scheduled traffic does not arrive as expected, the transmission opportunities during this period will not be shared with other non-scheduled flows,Therefore, the bandwidth utilization in this scenario is insufficient. 2.4. CQF(Cyclic Queuing and Forwarding) 2.4.1. CQF Overview CQF follows the gate operations of the TAS mechanism: when the gate is open, the packets in the queue are allowed to be forwarded to the next node; when the gate is closed, incoming packets are buffered in the queue before they are allowed to transmit.CQF simplifies the design of TAS by installing fixed configurations on the GCL. Time in CQF networks is divided into cycles with equal value T,and there are two queues performing enqueue and dequeue operations in a cyclic manner under the control of RX GCL and TX GCL.When the packets enter the queue Q1 in cycle duration T(cycle c), the receiving gate of Q1 opens. Meanwhile, the output sending gate of Q2 opens, packets transmitted to the next hop;When the next cycle arrives(cycle c+1). the output sending gate of Q1 opens and sends the packets received in the previous cycle, the receiving gate of Q2 open and starts to receive new packets. This cyclic queuing and forwarding mode can Han, et al. Expires 13 December 2023 [Page 7] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 achieve transmission in a fixed duration that does not exceed 2T on per hop. CQF could provide the deterministic latency relies on two principles. First, the upstream and downstream nodes are perfectly synchronized, and the rotation of the upstream sending cycle and the downstream receiving cycle must be consistent. Second, a packet received at a cycle must be sent at the next cycle in a node. Thus, the predictable end-to-end latency only depend on the cycle size and path length,and regardless of topology.CQF is useful for applications that do not require very small latency and jitter, but which are still real-time and require bounded worst-case latency. 2.4.2. CQF Analysis CQF can provide deterministic services with a maximum jitter of no more than 2T. The key issue is how to select the size of the cycle T and calculate the start time of the flow. The length of the queue is directly related to the size of cycle. If the cycle is too small, the queue is short too. Although the single hop queuing delay for traffic transmission is very small, there is not enough space to buffer more incoming flows, which can lead to a large number of flows that can not be scheduled; If the cycle is too large, it also means that queuing delay will become too large on per hop, which will result in a large end-to-end worst-case delay. Some traffic with requirements of low latency can not be scheduled, and larger queue lengths will also require more buffer resources. Due to limited underlying hardware resources, the difficulty and cost of hardware implementation are directly proportional to the buffer size. It is necessary to carefully select the cycle size; It needs to be large enough to accommodate all deterministic traffic, and in addition, the cycle includes a time duration called dead time (DT), which is the sum of delays 1, 2, 3 and 4 defined in Figure 1 of [RFC9320]. The value of DT ensures that the last packet of a cycle in the upstream node can be fully transmitted to the buffer of the same cycle in the downstream node. In the case of LAN, DT is relatively small compared to cycle T and is considered negligible, so only two buffer queues can run well.But in some deterministic networks, a single hop over a long distance is sufficient to produce a large delay. Considering that the optical transmission speed in fiber is 200000km/s, the propagation delay of some long-distance links can be in the order of a few milliseconds, which is much larger than in LAN, and cannot be ignored. In order to cover the DT, more buffer queues need to be introduced. In complex network topologies and multi hop scenarios, it is more difficult to select the cycle time T.On the other hand,the dead time (DT) also increases, resulting in further compression of the time available for deterministic stream transmission within the cycle Han, et al. Expires 13 December 2023 [Page 8] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 time,that make it impossible to deliver high-bandwidth services with extremely low jitter. Meanwhile, like TAS, classic CQF also rely on nanosecond clock synchronization across the entire network, where all network nodes align their cycle boundaries, and they cooperation with each other. This pattern limits the application of CQF in networks where precise time synchronization cannot be deployed. In addition, a large amount of deterministic traffic demands will produce more fluctuations caused by dynamic service join and leave, which requires corresponding resource scheduling algorithms to allocate resources appropriately among multiple flows to avoid transmission conflicts. For example, in some complex aggregation situations, a large number of traffic with periodic characteristics may be gathered at a certain intermediate node. If the scheduling planning is not appropriate, it will result in traffic congestion in one queue of the intermediate aggregation node, while the other part of the queue is idle, the traffic distribution is not balanced enough, which also exacerbates the probability of traffic conflicts. Therefore, it is necessary to introduce optimized traffic planning design with path calculation and resources reservation, such as planning through a centralized controller, but it also puts higher requirements on the algorithm. 2.5. ATS(Asynchronous Traffic Shaping) 2.5.1. ATS Overview In order to solve the problem of zero congestion and packet loss in the transmission of aperiodic data, and to further optimize the bandwidth utilization of services without strict requirements for time synchronization, [IEEE802.1Qcr] defines an asynchronous traffic shaping device ATS. ATS is designed based on Urgency-Based Scheduler ([UBS]). First,it identify the packets through the stream_handle (a sub-parameter of the stream identification function in [IEEE802.1CB]) and priority (the priority field in the VLAN tag) and match it into the corresponding stream filter,which specifies the stream gate and scheduler for the packets. The specified stream gate assigns internal priority, in this way, different degrees of delay guarantee can be provided in different nodes on the transmission path, it make the allocation of latency more flexible.The packets that has been assigned an internal priority enter the specified scheduler for shaping,which uses the interleaved algorithm based on the token bucket , and then assigned a eligibility time , which is the expected transmission time of the packets.After the shaper, packets enter the corresponding shared queue according to the internal priority and wait to be sent.The transmission selection algorithm based on strict Han, et al. Expires 13 December 2023 [Page 9] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 priority transmits packets from the queues in the order from higher priority to lower priority sequentially. If the eligibility time of the first packet in the shared queue is less than the current time,then the first packet can be sent directly and executes the transmission selection algorithm from the higher priority. Otherwise, turns to the next higher priority. 2.5.2. ATS Analysis ATS adopts a principle called Rate-Controlled Service Disciplines (RCSD), which is a non work-conserving packet service discipline. It consists of two parts: the rate controller implements the rate control policy, and the scheduler implements packet scheduling based on some scheduling policy, such as static priority, first come first served, or earliest deadline.By separating the rate controller and the scheduler, RCSD effectively decouples the bandwidth of each stream from its delay bound, therefore, RCSD can support low latency and low bandwidth service. The advantage of ATS is that when packets enter the queue, packets are assigned an eligibility time, it allows urgent flows can be transmitted preferentially. ATS also has the concept of a scheduler group, where multiple ATS schedulers can belong to a single ATS scheduler group.The ATS scheduler does not rely on binding to hardware queues. From the delay analysis formula of ATS, it can be concluded that the allocation of internal priority and the assignation of scheduler directly determine the delay boundary of flows. In the stream gate component of each hop, different internal priority can be assigned to packets instead of external priority, that can more flexibly allocate the service level.Therefore, the ATS scheduler can perform flexible shaping for per flow or aggregated flows. ATS can be placed each hop of nodes,then the network will not generate large bursts or burst aggregation, and the performance will be improved. The ATS scheduler state machine operation is based on the ATS scheduler clocks,which is an implementation specific local system clock function.There is no need to require nodes in the network to achieve time synchronization. A large number of flow aggregations will occur in a complex network topology,and it is necessary to consider flow aggregation strategies at intermediate nodes in the network.At the same time, multiple hops mean that flows will pass through more network nodes , and the end- to-end delay upper bound provided by ATS is also larger. Han, et al. Expires 13 December 2023 [Page 10] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 Scaling deterministic networks require a large number of services to be carried, and the cost of interleaved regulators (IR) maintained in per hop is high. Meanwhile,it is necessary to pay attention to the problem of IR head-of-line blocking(HOL) in large-scale networks. 3. Evaluation of TSN queuing mechanism with the requirements of scaling Deterministic networks The fllowing requirements are described in [I-D.ietf-detnet-scaling-requirements]. 3.1. Tolerate Time Asynchrony - CBS: does not rely on time synchronization or frequency synchronization technology. - TAS: The premise is that all terminals and network devices need to achieve nanosecond clock synchronization across the network (such as [IEEE1588],[IEEE802.1AS])to ensure that the GCL time of all outgoing ports is synchronized. - CQF: rely on nanosecond clock synchronization across the entire network, where all network nodes align their cycle boundaries, and they cooperation with each other. - ATS: based on the ATS scheduler clocks,which is an implementation specific local system clock function. No need to require nodes in the network to achieve time synchronization. 3.2. Support Large Single-hop Propagation Latency - CBS: link delay does not affect the transmission selection of CBS - TAS: link delay impacts on the determination of TAS transmission gate window position by precise calculation, it is independent of the value of the link delay. - CQF: link delay is much smaller than cycle time and is considered negligible, so only two buffer queues can run well.If single-hop propagation delay is very large ,and can not be ignored,in order to cover the DT, more buffer queues need to be introduced. - ATS: link delay does not affect asynchronous traffic shaping on per hop. Han, et al. Expires 13 December 2023 [Page 11] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 3.3. Accommodate the Higher Link Speed - CBS: more buffer space is required - TAS: more buffer space or more precise time control is required - CQF: more buffer space is required - ATS: more buffer space is required 3.4. Be Scalable to The Large Number of Flows - CBS: stream can aggregated by priority - TAS: transmission gates are associated with each queue,TAS does not restrict the strategy of flow entering queues, for example, priority mapping is a possible approach - CQF: transmission gates of egress ports are associated with each queue,flow aggregation is allowed,but the stream filter which determine the filtering and policing actions should be placed on per node,it may need to maintain per flow or aggregated flows states. - ATS: ATS scheduler can perform flexible shaping for per flow or aggregated flows,so the stream filter which determine the filtering and policing actions should be placed on per node,it may need to maintain per flow or aggregated flows states. 3.5. Tolerate High Utilization - CBS: set the pre-configuration of bandwidth limit for each traffic class.Typically set 75% of the maximum bandwidth for bandwidth intensive applications such as audio and video.BE flows can use the unused portion of the reserved bandwidth of TSN flows. - TAS: the transmission window reserved for deterministic flows through GCL is usually exclusive. During the time period when the gate state of the queue associated with scheduled flows is open, even if the scheduled traffic does not arrive as expected, the transmission opportunities during this period will not be shared with other non-scheduled flows,Therefore, the bandwidth utilization in this scenario is insufficient. Han, et al. Expires 13 December 2023 [Page 12] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 - CQF: the Cycle includes a time duration called dead time (DT), which is the sum of output delay, link delay,frame preemption delay and processing delay.If the DT is large, the proportion of time available for deterministic stream transmission within the cycle time will be compressed, that makes it difficult to achieve high bandwidth utilization. - ATS: Can achieve high bandwidth utilization. 3.6. Prevent Flow Fluctuation from Disrupting Service - CBS: A large amount of traffic is more likely to cause catching up with burst flows, which makes the delay performance deteriorate seriously. - TAS: The configuration and management of the GCLs are more complicated and requires update GCLs frequently. - CQF: Requires corresponding resource scheduling algorithms to allocate resources appropriately among multiple flows to avoid transmission conflicts. - ATS: the cost of interleaved regulators (IR) maintained in per hop is high,the problem of IR head-of-line blocking should be considered. 3.7. Be Scalabcle to a Large Number of Hops with Complex Topology - CBS: the more hops, the worse the delay degradation. End-to-end delay will become unacceptable. - TAS: will be a great challenge for TAS that relies on precise calculations and complex configurations. - CQF: it is more difficult to select the cycle time T,need Making trade-offs between end-to-end delay and cycle time. - ATS: need to consider flow aggregation strategies at intermediate nodes,end-to-end delay upper bound provided by ATS is also larger. 3.8. Tolerate Failures of Links or Nodes and Topology changes Not related to queuing mechanisms directly. 3.9. Support Multi-Mechanisms in Single Domain and Multi-Domains Not related to a single queuing mechanism directly. Han, et al. Expires 13 December 2023 [Page 13] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 4. Evaluation results According to the evaluation in section 3, the evaluation results of queuing mechanisms proposed in TSN are shown in the table below: ====================================================================== | | evaluation results of TSN | | requiremens of scaling | queuing mechanisms | | Deterministic Networks +-------------------------------+ | | CBS | TAS | CQF | ATS | ====================================================================== | Tolerate Time Asynchrony | Yes | No | No | Yes | +------------------------------------+-------+-------+-------+-------+ | Support Large Single-hop | Yes | Yes | No | Yes | | Propagation Latency | | | | | +------------------------------------+-------+-------+-------+-------+ | Accommodate the Higher Link Speed |Partial|Partial|Partial|Partial| +--------------------------------------------+-------+-------+-------+ | Be Scalable to The Large | Yes | Yes |Partial|Partial| | Number of Flows | | | | | +------------------------------------+-------+-------+-------+-------+ | Tolerate High Utilization | Yes |Partial|Partial| Yes | +------------------------------------+-------+-------+-------+-------+ | Prevent Flow Fluctuation from |Partial|Partial|Partial|Partial| | Disrupting Service | | | | | +------------------------------------+-------+-------+-------+-------+ | Be Scalabcle to a Large Number of | No |Partial|Partial|Partial| | Hops with Complex Topology | | | | | +------------------------------------+-------+-------+-------+-------+ | Tolerate Failures of Links or Nodes| Not directly related to | | and Topology changes | queuing mechanisms | +------------------------------------+-------------------------------+ | Support Multi-Mechanisms in | Not directly related to a | | Single Domain and Multi-Domains | single queuing mechanism | +------------------------------------+-------------------------------+ Figure 1: Evaluation Results of TSN Queuing Mechanisms 5. Conclusion Various applications in deterministic networks have different requirements for deterministic service indicator,and different queuing mechanisms can provide different levels of delay, jitter, and other guarantees. There may also be situations where network devices provide multiple queuing mechanisms simultaneously. For example, network aggregation devices can use the mechanisms specified in Han, et al. Expires 13 December 2023 [Page 14] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 [IEEE802.1Qbv] and [IEEE802.1Qcr] to forward traffic to different paths with different SLA at the same time. By providing multiple queuing mechanisms to meet diversified deterministic service requirements, this demand is particularly prominent in large-scale networks compared to small-scale environments. This document uses the requirements of scaling deterministic networks to evaluate several existing queue mechanisms in TSN, analyze their characteristics, and provide a basis for selecting suitable queue mechanisms for services with different deterministic requirements. At the same time, the challenges faced by their deployment in scaling networks were also analyzed, and brings some thoughts to the design of several new queuing mechanisms proposed for enhanced deterministic forwarding. 6. IANA Considerations This document has no IANA actions 7. Security Considerations TBD. 8. Acknowledgements TBD. 9. References 9.1. Normative References [IEEE1588] "IEEE Standard for a Precision Clock Synchronization Protocol for Networked Measurement and Control Systems", 2008, . [IEEE802.1AS] "IEEE Standard for Local and Metropolitan Area Networks-- Timing and Synchronization for Time-Sensitive Applications", 2020, . [IEEE802.1Qav] "IEEE Standard for Local and metropolitan area networks -- Virtual Bridged Local Area Networks - Amendment 12: Forwarding and Queuing Enhancements for Time-Sensitive Streams", 2010, . Han, et al. Expires 13 December 2023 [Page 15] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 [IEEE802.1Qbu] "IEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks -- Amendment 26:Frame Preemption", 2016, . [IEEE802.1Qbv] "IEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks - Amendment 25:Enhancements for Scheduled Traffic", 2016, . [IEEE802.1Qch] "IEEE Standard for Local and metropolitan area networks -- Bridges and Bridged Networks - Amendment 29: Cyclic Queuing and Forwarding", 2017, . [IEEE802.1Qcr] "IEEE Standard for Local and Metropolitan Area Networks-- Bridges and Bridged Networks Amendment 34:Asynchronous Traffic Shaping", 2020, . [IEEE802.3br] "IEEE Standard for Ethernet-Amendment 5:Specification and Management Parameters for Interspersing Express Traffic.", 2016, . [RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, . 9.2. Informative References [AVB-Latency] "AVB Latency Math", 2010, . [ClassA-Latency-Calc] "Class A Bridge Latency Calculations", 2010, . Han, et al. Expires 13 December 2023 [Page 16] Internet-Draft Analysis and Evaluation for TSN Queuing June 2023 [I-D.ietf-detnet-scaling-requirements] Liu, P., Li, Y., Eckert, T. T., Xiong, Q., Ryoo, J., zhushiyin, and X. Geng, "Requirements for Scaling Deterministic Networks", Work in Progress, Internet-Draft, draft-ietf-detnet-scaling-requirements-02, 24 May 2023, . [IEEE802.1CB] "IEEE Standard for Local and metropolitan area networks-- Frame Replication and Elimination for Reliability", 2017, . [RFC9320] Finn, N., Le Boudec, J.-Y., Mohammadpour, E., Zhang, J., and B. Varga, "Deterministic Networking (DetNet) Bounded Latency", RFC 9320, DOI 10.17487/RFC9320, November 2022, . [UBS] "Urgency-Based Scheduler for Time-Sensitive Switched Ethernet Networks", 2016, . Authors' Addresses Yufang Han ZTE Corporation China Email: han.yufang1@zte.com.cn Shaofu Peng ZTE Corporation China Email: peng.shaofu@zte.com.cn Yuehong Gao Beijing University of Posts and Telecommunications China Email: yhgao@bupt.edu.cn Han, et al. Expires 13 December 2023 [Page 17]