Introduction
Measuring performance – and reporting publicly the results – is always a complex and sensitive issue. It sometimes turns into a fool’s game where making the most spectacular marketing announcement takes precedence over the quality and relevance of the measurement. For instance, the realism of streaming workload patterns is key to a meaningful performance benchmark.
The concept of performance has also evolved in the past years. While the throughput-cost ratio is still the most important criterion, security (encryption/authentication) has become a critical requirement and energy efficiency a major attribute of streaming performance.
Finally, performance metrics matter. HTTP/S-, TCP- and Ethernet-level egress throughputs are all valid measurement units, but they need to be more explicit in the reporting results.
At Broadpeak we wish to have a transparent approach when it comes to measuring and reporting performance.
And this does not prevent us from keeping breaking records thanks to a fine integration of our streaming software with quality hardware available on the market.
Broadpeak’s guiding principles on testing conditions
As max performance tests are usually performed in controlled lab environments, we are using at Broadpeak two means in our methodology to simulate real-world network conditions.
First, we generate realistic traffic patterns. This is achieved thanks to a full-featured benchmarking tool that we have developed (resulting from a collaboration with Intel). The tool simulates highly realistic workloads of video players in use by consumers and supports the latest protocol evolutions. Besides, throughput and power-efficiency are always measured under high concurrency, up to one million simultaneous connections. If you want to be a reference in streaming performance, you first need to be a reference in measuring performance.
Second, we use relatively modest, cost-effective hardware configurations. It does not make sense to benchmark performance on costly, top-of-the-range hardware that is unlikely to be massively deployed in the field. Our latest public performance records have been achieved using servers equipped with a range of processors that we have deployed in thousands.
About performance metrics
There are various possible metrics to report performance. The streaming industry has implicitly agreed to use the Ethernet throughput (server’s egress transmission bit rate at Ethernet level) vs. the so-called “goodput”, that reports the useful HTTP/S payload throughput. The drawback with the goodput metric is that results vary a lot on workloads and traffic patterns, which does not facilitate the comparison between the different solutions on the market.
While the Ethernet throughput does not exactly report the useful streaming bit rate, this metric has the advantage of being compared to the server’s “line rate”, which is the maximum theoretical server’s egress throughput. Thus, the Ethernet throughput gives a good indication of how well the streaming software is using the capabilities of the underlying hardware.
Broadpeak’s performance journey
Since 2021, we have been paving the way towards a more efficient streaming.
– May 2021: Release of our benchmarking tool used to reproduce operators’ production environments. The tool has been enhanced since then with new observability features, higher versatility and performance.
– February 2022: Tests on Broadpeak’s new BkS450 streaming software before its launch: 302 Gbps in HTTPS from a single-socket server equipped with an AMD EPYC Zen 2 processor (7502P). 0.57 Gbps/Watt power-efficiency.
– December 2022: First tests on Broadpeak’s new BkS450 streaming software after its launch: 627 Gbps1) in HTTPS from a dual-socket server equipped with two 3rd generation Intel® Xeon® Gold processors (5318Y). 1.09 Gbps/Watt power-efficiency.
Ref.: Broadpeak Sets New Performance Standard for Streaming with BkS450 (intel.com)
– September 2023: 1.09 Tbps2) in HTTPS from a dual-socket server equipped with two 4th generation Intel® Xeon® Gold processors (6454S). 1.21 Gbps/Watt power-efficiency.
Ref.: Broadpeak BkS450 Streaming Software Passes 1Tbps
– 2024: Our plan is to focus on smaller configurations (far edge streaming servers) and security enhancements.
Why push the performance envelope even further?
1. First, to make our software design more scalable, without flaws that would be a showstopper for supporting future hardware evolution (e.g., more processor cores, or smaller cores such as Intel’s “Sierra Forest” and AMD’s “Siena” processors).
2. Second, to offer a greater safety margin without compromising the performance-cost ratio. This allows to accommodate exceptional workloads getting well beyond system specifications, and unexpected external events like a series of hardware failure of neighboring servers. Performance has always been sensitive to workloads. For instance, even slight variations such as lower hit ratio than expected, more spread popularity (with more requests going to SSDs compared to RAM) negatively impact the performance.
3. Third, to improve energy efficiency. While the streaming software can’t do much about the baseline power consumption of the server, it can do a lot to stretch the throughput performance with a relatively small increase of the power consumption.
4. Last but not least, to increase sustainability, thanks to a longer lifetime of the installed hardware. Because Broadpeak BkS450 software does not rely on specific hardware accelerators (e.g., TLS offload) but only on software optimizations, Service Providers benefit from Broadpeak BkS450’s successive performance enhancements without changing their already-deployed hardware platforms.
Conclusion
The crazy race on streaming performance records will carry on, and fortunately performance benchmarks now includes energy efficiency as a key criterium.
We often forget that, regardless of its marketing impact, outstanding throughput performance guarantees a safety margin and allows CDNs to sustain high throughputs even in presence of unexpected circumstances.
Mimicking real-life production environments in evaluations, an essential point of Broadpeak’s benchmarking methodology, is key if we want performance reports to be meaningful to the industry.
We will keep pushing the power efficiency envelope towards further heights, working with stimulating industry leaders like Intel and AMD, leveraging for instance their very promising next-gen processors like Intel’s Xeon E-core based “Sierra Forest” and AMD’s EPYC™ 8004 Processors “Siena” recently launched.
1) The Intel whitepaper in reference used TCP egress both for raw throughput and power efficiency; we switch in this blogpost, and in upcoming publications, to Ethernet throughput which is more commonly displayed – Ethernet gives approximately a 5% difference over TCP payload throughput & corresponding power efficiency.
2) Ethernet throughput
Author
Nivedita Nouvel is Broadpeak’s VP Marketing. She is in charge of communication and product strategy and positioning. Before joining Broadpeak, she worked for 3 years as a Product Manager for Envivio, specialist of H.264 encoding and for Thomson, where she was in charge of the IPTV and Mobile TV Service Platform. She graduated from IMT Atlantique (formerly Télécom Bretagne) engineering school and holds a Master of Science in Satellite Communications from UCL, London.
