Enabling Passive Measurement of Zoom Performance in Production Networks

Summary

This talk presents a novel approach to passively measuring Zoom performance in large-scale production networks without end-host access. The researcher reverse-engineers Zoom’s previously undocumented packet structure, enabling extraction of detailed metrics such as latency, jitter, frame rate, and packet loss directly from network traffic. By developing techniques to detect Zoom flows, decode headers, and group streams into meetings, the study reveals insights into real-world usage and performance across a university network. Findings show that simple metrics like bitrate are insufficient, emphasizing the need for deeper analysis to distinguish network issues from application behavior and improve video conferencing quality monitoring.

Takeaways

• 😀 Video conferencing, especially Zoom, has become an essential tool across industries such as healthcare, education, and enterprise. However, ensuring the best performance and quality of service remains challenging for network operators.
• 😀 Measuring video conferencing performance, particularly for latency-sensitive applications like Zoom, requires more than just monitoring data rates. It requires metrics like latency, jitter, packet loss, retransmissions, and frame rates to understand the network's impact.
• 😀 Previous research on network performance during video calls often relied on basic metrics, such as flow data rate, which is insufficient for understanding the quality of the video conferencing experience.
• 😀 Zoom uses UDP for media transport, and it often multiplexes multiple media streams (video, audio, screen sharing) over a single flow. This makes interpreting data rate drops more complicated as they may not always be caused by network issues.
• 😀 To effectively analyze Zoom traffic, we need to look deeper into packet-level data, considering various metrics such as frame rate, packet loss, and jitter in relation to the meeting's composition (number of participants, media shared).
• 😀 Measuring performance passively in large-scale networks is complex due to Zoom's proprietary packet header format and the encryption of much of its traffic. Researchers developed methods to detect Zoom traffic and extract relevant performance metrics from it.
• 😀 The detection of Zoom traffic, especially for peer-to-peer connections, relies on monitoring session traversal utility for NAT (STUN) packets and ephemeral port numbers, which can be used to identify direct connections between participants.
• 😀 By performing entropy analysis on Zoom packets, the research team was able to reverse-engineer Zoom’s packet format, revealing headers for server encapsulation, media encapsulation, RTP headers, and encrypted media.
• 😀 A two-step heuristic approach was used to group individual media streams into a single Zoom meeting, allowing for accurate latency measurements and performance analysis of the entire meeting across all participants.
• 😀 The study revealed key insights into Zoom's network behavior, including detailed latency measurements, frame rate distributions, and media type breakdowns. The data showed that video traffic is most affected by changes in resolution (e.g., switching to thumbnail mode).
• 😀 The research team built a custom Wireshark plugin and a program to capture and analyze Zoom traffic in real time. They also conducted a large-scale measurement study, providing detailed insights into the campus-wide Zoom traffic, including packet rates, latency, and frame rate data.
• 😀 Despite challenges with the accuracy of Zoom's jitter measurements, the study showed that analyzing the network and media streams individually can pinpoint problems like network congestion, faulty access points, or client-specific issues during meetings.

Q & A

Why is video conferencing performance measurement important for network operators?

Video conferencing applications like Zoom are latency-sensitive and bandwidth-hungry, making it crucial for network operators to configure their networks to ensure optimal performance. Measuring video conferencing performance helps identify network issues that can impact the quality of service for users.

What challenge do network operators face when measuring video conferencing performance?

The primary challenge is that traditional network traffic metrics (like overall data rate) are insufficient to analyze video conferencing performance, as they don't provide meaningful insights into issues like latency, jitter, packet loss, and the actual media being transmitted.

What did the study reveal about video conferencing traffic analysis using Zoom's data?

The study showed that Zoom uses a proprietary and previously undocumented packet header format, which made it difficult for traditional traffic analysis tools to interpret Zoom's network traffic. The research involved reverse-engineering this packet format to extract performance metrics from the traffic.

How does Zoom's use of UDP and packet multiplexing complicate network performance analysis?

Zoom uses UDP for media transmission, and multiple media streams (video, audio, screen sharing) are multiplexed over a single UDP flow. This complicates the analysis because a drop in data rate can occur due to a participant leaving the meeting or a change in meeting mode, which isn't necessarily a network issue.

How did the researchers detect direct peer-to-peer traffic between Zoom participants?

The researchers identified a specific pattern of session traversal utility for NAT (STUN) packets exchanged before establishing a direct peer-to-peer connection. By monitoring these packets, they were able to detect when peer-to-peer connections were being set up between Zoom clients.

What was the approach used to reverse-engineer Zoom’s packet format?

The researchers used graphical entropy analysis to inspect the packet payload after the UDP header. By plotting byte values over time, they identified distinct patterns corresponding to encryption, stream identifiers, and sequence numbers, which helped them map out the packet format Zoom uses.

What kind of data was extracted from Zoom's traffic, and how was it useful?

The researchers extracted various packet fields, such as sequence numbers, timestamps, media types, and stream identifiers. This allowed them to measure key performance metrics like media bit rate, frame rate, latency, jitter, packet loss, retransmissions, and out-of-order deliveries.

What did the study find about Zoom’s frame rate and how it relates to video quality?

The study found that frame rates in Zoom can drop significantly, with 75% of samples showing a frame rate below 20 frames per second. This was often due to Zoom adjusting video resolution, not necessarily network issues. Frame rate drops to zero were also observed during screen sharing when no new frames were being generated.

How did the researchers validate their network performance measurements?

The researchers validated their measurements by comparing them against Zoom's own provided metrics, using controlled experiments where they injected cross-traffic and compared the results of their network measurements with Zoom's data. This helped ensure their methods were accurate.

What are the next steps for the research team regarding video conferencing performance analysis?

The team is working on improving quality assurance metrics, such as estimating the likelihood of video stalls or freezes. They're also exploring how to integrate these measurement techniques into programmable switches for real-time monitoring and offloading selective forwarding unit (SFU) functionality to reduce latency and infrastructure costs.