This blog post is a reflection about the quality of experience for streaming services: how the industry defines it, values it, measures it and improves it. It is based on questions formulated for the 2020 DTVE survey, with a focus on the experience for streaming services, as well as where service providers and content providers deliver adaptive bit-rate content to consumers.
The survey starts with this simple question to start the conversation:
How important is video quality of experience to the attractiveness of a service?
We asked it to confirm an underlying assumption in the industry: consumers today are not going to put up with poor streaming experiences.
And indeed, the respondents to the survey confirmed that the quality of experience is a mandatory feature to attract consumers: While content offering is the number one reason why someone subscribes to video services, the quality of the experience when consuming this content is the second most important aspect.
The situation is clear: Content providers and service providers have to aim for the best possible experience to maximize the value of their content offering.
—
We often hear that user expectations for video streaming have evolved, including various comparisons between streaming and broadcast TV services. In fact, with immediate channel change, no rebuffering, no latency, and a service always available, broadcast TV seems like a good benchmark to challenge. So, we asked the question:
What quality of experience do viewers expect from a streaming service?
The answers confirmed that consumers do expect streaming services to be comparable with broadcast: 43% of the respondents told us that. But what is really interesting here is that the majority of the respondents (51%) now expect the quality of streaming services to be higher than broadcast.
There are two angles to this:
1. As streaming services use adaptive bit-rate (ABR) technology to deliver video, it is by nature uncoupled from the network — as in over-the-top (OTT): the video is delivered in chunks and without guarantees from the network. It is, therefore, more likely to encounter delivery problems.
So while the streaming industry has to meet high expectations, the limitations of the underlying technology must not get in the way: Problems such as poor start time, stream failure, high latency during live sports content, video freezing or rebuffering must all be eliminated to deliver the quality consumers want.
2. Beyond delivering streams free of network problems, consumers have gotten used to immersive experiences: The streaming giants were the first to deliver 4K services on a large scale, coupled with improved video technology like HDR and Dolby Vision, as well as enhanced audio technology with Dolby Atmos. These are now requirements for new streaming platforms.
As Netflix has undeniably raised the bar, delivering a high-definition stream free of network interruptions is no longer enough: Service providers and content providers have to also offer immersive experiences.
—
From conversations with our customers, we established that the quality of the video experience can have a direct impact on revenues. With the following question we wanted to see if the industry also consciously links QoE and revenues:
What impact does video quality experience have on service providers’ revenues?
The result is clear: nearly 80% of the respondents confirmed that for them QoE has at least a significant impact on the revenues.
In a market with an increasing number of video services and fierce competition, providers know that poor quality of experience risks an increase in audience churn.
To substantiate this point further, we can recommend a report published a year ago by Akamai: Understanding the Value of Consistency in OTT Video Delivery. Through a series of interviews and a case study, the report tries to estimate the cost of poor experience. It establishes a direct link between rebuffering and abandonment, with each instance of rebuffering resulting in a 1% abandonment rate.
—
The next question was asked openly to see how the QoE is being measured today:
How do service providers measure video quality of experience?
We expected to see a mix of preventive and corrective actions, but the respondents indicate that the industry relies primarily on corrective actions: investigating customer complaints/problems and customers surveys. While these tools offer very valuable information after an event, they will only paint a partial picture, as some consumers won’t report issues or give feedback, and others may simply tune out.
At Broadpeak, we believe the quality of the experience needs to be measured proactively across the system: This is why we rely on an approach that combines real-time data from the CDN and from the client.
- The data coming from the CDN includes all the network-related information: Our customers can paint a very accurate view of what is happening from the network based on parameters such as status of the content (hit/miss), CDN cache, content size, and transfer speed.
- The data from the client side is collected by the SmartLib, an add-on to the video players that captures all the experience-related metrics: failed sessions, startup time, rebuffering events, etc. This allows our customers to get a real-time view the quality of the delivery.
Talking to our customers, there is an increasing demand for proactive monitoring. They want to be able to, for example, spot unusual trends in the traffic graphs and generate automated alerts.
To do this, we have introduced the BkA200, a monitoring solution capable of collecting and displaying all real-time metrics across the delivery solution.
—
For the next question we asked respondents about the perspective of consumers:
What is important for video quality of experience in video streaming?
The industry says that the No. 1 metric is rebuffering, and the respondents confirmed it: rebuffering is the huge irritation with ABR technology. As mentioned above, rebuffering undermines the consumer interest and leads to abandonment.
Respondents then named “fast startup, how quickly the stream starts” as the second most important metric, followed by “avoidance of quality up/down shifts,” “delivery of resolution as high as possible,” and “minimization of latency for live.”
It is worth noting that these four metrics are considered by 86% of the respondents as at least “moderately important,” meaning that once the rebuffering issues are out of the way, consumers pay close attention to these criteria to evaluate their experience.
The next step will be to find ways to improve these metrics to boost the perceived quality. Doing so can be difficult since isolated testing (in a lab for instance) cannot characterize how the system will behave as a whole. In fact, real conditions are required to assess the impact of the QoE optimizations: many factors come into play like network conditions (network underload on Friday night for example), in-home connectivity (such as varying WiFi coverage), broad range of devices and operating systems used to playback video.
This is where A/B testing tools come in.
(If you are not familiar with A/B testing, the idea is simply to identify what is the most efficient configuration by experimenting variants on a group of users — for example working out which layout of a website leads to higher sales).
Applied to the world of video delivery, A/B testing allows experimentations with the production systems. With A/B testing providers give one version of the configuration to one group and the other version to another group. The performance of each variant is analyzed in real time and allows low-risk testing and rapid iterations.
Similar to a competition, it becomes possible to pit two versions of the video delivery configuration against one another to see which comes out on top.
This powerful tool allows providers to test and validate which changes made to the production platform are the most beneficial. It is particularly useful when implementing new features like low-latency live delivery, new delivery protocols like HTTP/2 or HTTP/3, or new caching algorithms in the CDN.
It is worth noting that experimenting in real conditions is a robust way to see data guide the optimization process: By making decisions based on data from experiments, content and service providers will converge toward the most effective configuration of their video systems.
—
Now that we have the tools to monitor the experience, and a methodology to improve it, we asked the following question to identify where problems were likely to be coming from:
Where are problems relating to video QoE most likely to come from?
Respondents have identified the combination of CDN/delivery and network capacity as the most likely sources of problems related to quality. We think this is a fair assumption and, at Broadpeak, we believe the CDN has a huge influence when it comes to delivering the best and most adapted experience, which consumers expect.
At the end of the day, our customers deploy the CDN first to reduce their costs of delivery, putting CDN capacity near the consumers. This reduces their transport costs as they don’t carry the video traffic through the rest of the network. By doing so, they are also reducing latency, increasing the bandwidth available, and therefore improving the video experience for consumers.
—
Finally, we wanted to assess what tools the industry would consider for improving the experience, so we asked:
What tools are useful to delivering high quality of experience?
Respondents highly ranked multicast ABR technology, which is indeed a great tool for managed networks. Since ABR is now replacing many legacy TS-based video distribution systems, coupling ABR with multicast in solutions like Broadpeak nanoCDN™ is a powerful combination to deliver live experiences with high-bitrates, low latency and fast start time.
Edge caching technologies also ranked high on the list. This approach involves deploying CDN caches deep inside the networks to reduce even further network distance and provide superior experiences. This is something that we begin to see happening in mobile networks and Broadpeak has been working on various proof of concepts in the last 12 months.
Other video optimizations like more efficient compression techniques, converged packaging, or new delivery protocols are also worth mentioning.
In the end, Broadpeak believes that technology vendors need to take a solution-centric approach. For example, low latency can only be delivered with optimizations along the delivery chain (i.e., encoders, origin servers, CDN, clients).
The reality is that the consumer experience can only be guaranteed if the various components play well together. For example, at Broadpeak we have implemented unified packaging for DASH and HLS using CBCS so that the same chunks of video can be referenced via a DASH or an HLS manifest file. This improves the cache efficiency, as only one version of the content is stored in cache and served to both the HLS and DASH clients. This also results in more content being stored near the consumers, leading to improved performance in terms of bandwidth and start time.
We think this collaborative approach between the various elements of the chain produces the best results. Interactions in the video delivery path are varied and they each provide opportunities to improve the experience. The diagram below shows some of the integration points.
As the ecosystem of ABR video remains fragmented with different vendors providing encoding, packaging, CDN and players, the content and service providers are right to be looking for technology vendors that work openly and that have a strong network of partners, as they will bring innovation to their video offerings.
In conclusion
We have established that the quality of experience is a key criterion to attract and retain consumers to video offerings. Content providers and service providers who aim to deliver the best possible experience will maximize the value of their content offering.
We now also know that while the streaming industry has high expectations to meet, the limitations of the underlying technology must not get in the way. Problems such as poor start time, stream failure, high latency on live content, and rebuffering, must all be eliminated to deliver the quality consumers demand.
When it comes to expectations, delivering a high-definition stream free of interruptions is no longer enough. Service providers and content providers have to also offer immersive experiences, superior to broadcast TV to attract consumers.
We have also verified that rebuffering really hurts! Studies have shown that rebuffering and abandonment are directly linked. Every content or service provider that cares about their viewers needs to work hard to eliminate rebuffering while also considering other metrics like start time, up/down shifts, high bitrates, and latency to get a complete picture.
The best way to do that is with data. Providers should take proactive measurements both on the client and on the network sides. These two sources allow troubleshooting and optimization to take place on the networking and content delivery sides, resulting in a huge impact on the experience.
In this area of CDN and networking, we have recently launched a new technology called S4Streaming.
By enabling dynamic bandwidth management, congestion avoidance, real live low latency, S4Streaming is a solution that guarantees the best possible ABR experience for consumers.
As the environment becomes more complex with more moving parts, Broadpeak believes that it is with innovation that the consumer experience will continue to be improved. New approaches like A/B testing are modern solutions to modern challenges, and this is how our industry continues to move forward.
You can download the DTVE Industry Survey 2020 here.
