Broadpeak’s lessons learned and best practices after deploying and operating its content delivery and video streaming solutions on the public cloud for the benefit of more than 60 customers.
Introduction
Much has already been said and written about the pros and cons of using public cloud services. And this debate is not over because customer needs as well as the cloud market itself are constantly evolving. In any case one thing is certain: the use of public cloud is growing and its utilization has become instrumental to the success of many businesses.
The lack of expertise and experience in the cloud has led many enterprises to make sometimes inappropriate and hasty decisions, fostered by the attractive game of committed spend. Some Businesses have realized that the significant cost increase of their operations after migrating all or part of their assets to the cloud was not worth the benefits of speed, agility and scalability. Not to mention that their committed spending was locking them in for some time with their chosen cloud service provider.
While some organizations have decided to migrate some workloads back to on-prem for these cost and lock-in reasons, cloud platforms and their usage are globally better known nowadays. Pay-as-you-grow (PAYG) model is not the only rule anymore and new entrants on the cloud market integrate egress, a major part of cloud bills, in their offers. Finally, global awareness of all possible cost optimizations when using or migrating to public cloud, have greatly increased.
The use of public cloud in the streaming industry
These positive trends also apply to the streaming and content distribution industry.
Public cloud providers bring value to this market thanks to the scale and elasticity of their platforms, the relatively easy setup of a service on their user interfaces, and the short time to market (TTM) they allow.
Among the main reasons why video service providers turn to the public cloud, we can mention:
- the average increase in video quality and resolution requires greater amount of compute and storage resources,
- the large audiences and huge – sometimes unpredictable – spikes of requests to video systems need important processing and network resources; cloud resources are particularly useful to handle overload traffic,
- the need for video systems to be resilient require important disaster recovery infrastructures that should be mobilized at any time,
- the “pay-as-you-grow” model allows video service providers to start small and scale up seamlessly as their subscriber base expands.
Our experience and commitment to the public cloud
We have fully integrated the use and benefits of the public cloud in our content delivery and video streaming solutions.
We deploy and operate our software on AWS, Google Cloud and OVHcloud platforms for more than 60 customers worldwide.
Telus in Canada, CETIN in Europe, and StarHub in Asia are some examples of Tier 1 customers for whom we have deployed our video solutions on the public cloud.
Making an effective use of the public cloud
It was obvious for us since the advent of public cloud platforms that some video functions would hardly be cost-effective when only deployed on these infrastructures. Video CDN is one of them, due to the prohibitive costs generated by important volumes of egress traffic and the direct competition with cloud providers’ own CDN services (AWS CloudFront or Google Cloud’ Media CDN for instance).
However, designing hybrid architectures that combine both cloud-based CDN instances (for overload, short-term needs, global coverage) with service providers’ own on-prem CDN infrastructures, proved to be a successful way forward.
We also learned from our extensive experience with cloud projects and our collaboration with public cloud service providers that many cost optimizations were possible and desirable for video applications deployed and operated in the cloud.
VOD and Cloud DVR systems are the first to be concerned due to their important needs in storage (volume and duration). Some content personalization functions like targeted ad insertion also require special attention when deployed on public cloud, due to the potentially high number of requests and executions of manifest manipulation.
broadpeak.io: a turning point in Broadpeak’s public cloud expertise
Our practice and know-how of the public cloud took another step forward in 2021-2022 with the creation and launch of our cloud-based SW-as-a-Service (SaaS) platform, broadpeak.io, offered in full-serve and self-serve models.
We did not only devise and deploy our platform on the public cloud (AWS): we operate it, we manage on it all the commercial SaaS video services that we offer to our customers.
We made a lot of findings through the different stages of that journey, from the design until the launch and over the course of the operational phase. We faced obstacles, we made mistakes, we failed, we bear unanticipated costs, we consulted, we changed strategies and architectures, we optimized, and we succeeded.
Today, the broadpeak.io platform is thriving and serving a growing number of content and service providers.
Broadpeak also partially relies on the public cloud for its other major platform, peakVU.TV, an IPTV managed service that enables U.S. broadband, cable, telco, utilities, and wireless operators to launch video streaming services with all the features subscribers demand.
All the expertise we gained with broadpeak.io and peakVU.TV, we brought it and keep bringing it to those of our customers who have decided to deploy our solutions on the public cloud.
The best practices we inherited from our broadpeak.io cloud platform experience
Let’s specifically deep dive into the lessons learned and the expertise we have gained from the design, launch and operation of our broadpeak.io platform.
We leveraged AWS Migration Acceleration Program (MAP)
Public cloud providers propose special programs to help service providers in their greenfield deployment or migration to the cloud (AWS Migration Acceleration Program, Google Cloud Rapid Migration and Modernization Program, Microsoft Azure Migrate and Modernize).
Since we were deploying on AWS, we used AWS MAP and it was very useful:
- The program provided us precious consulting services on our architecture.
- AWS MAP also included significant cloud credits (25% credit on our future spending) that helped us financially get our cloud project started.
- Finally, AWS MAP program also connected us to useful partners like DoIT, a cloud broker specialized in FinOps (read also below).
We greatly benefited from our collaboration with cloud broker DoIT
Our collaboration with DoIT turned out to be very profitable.
- DoIT helped us identify compute instances not covered by our existing AWS spending commitment and apply 1-year discount rates on them.
- We extended and optimized our use of AWS Spot Instances and reduced our compute expenses by up to 90% (since many of our broadpeak.io services are fault tolerant and can withstand possible instance preemptions).
- We obtained much better CloudFront pricing without commitment, allowing us to divide our volume- and HTTP request-based cost by a factor of 15 (NB. DoIT like other cloud brokers give access to negotiated wholesale rates, with much reduced committed spend, that one would never get other than through an enormous consumption of cloud services).
We advantageously started using AWS Saving Plans when our architecture got more mature
In addition to our FinOps work with DoIT, we obviously used the cost saving plans offered by AWS (NB. also available at Google Cloud, through their Compute Engine Committed use discounts – CUDs).
While these plans are also meant to push for more usage at cloud providers’ advantage, it allowed us to optimize our costs further using different types of commitment, from direct to more flexible ones. We used these plans on our broadpeak.io platform when our architecture became mature enough and our cloud costs more predictable.
We started closely monitoring our actual cloud costs… although sometimes too late!
We didn’t realize at the launch of our broadpeak.io platform how important the rigorous and detailed monitoring of our cloud consumption (usage and costs) would be. At the infancy stage of broadpeak.io, we made bad choices in our architecture, mistakes in our use of cloud options and tiers, and we unfortunately had to bear some unanticipated extra costs.
Later on, the thorough observation of our AWS cloud consumption triggered a wealth of optimizations on the platform, in various domains, as described below.
Storage
- We changed database (infrastructure). We realized that using a fully managed database like Amazon Aurora was not needed and that the performance of Amazon Relational Database Service (RDS) would be more than enough. At the end of a smooth migration that was spread over 6-month, we divided by a factor 3 our database costs for our content personalization services (Fig.1).
Compute
- We increased efficiency of our software (application) – In some of the video services our broadpeak.io platform is offering, computing can represent close to 80% of the cloud cost because of the massive manifest manipulations that are required; therefore, we decided to accelerate the improvement of our manifest manipulation engine’s software efficiency; as a result, we maximized the hardware utilization of AWS EC2 instances and at the end of the day, paid for less machines to handle our load.
- We carefully planned compute resources (infrastructure) – It was all about preventing over-dimensioning and paying only for the compute resources we needed. We also leveraged the scalability of the cloud through Amazon’s managed Kubernetes proposition (Elastic Kubernetes Service – EKS), to ensure our compute consumption stick as much as we could to the manifest manipulation operations we had to execute on the platform.
- We use up-to-date platforms (infrastructure) – We follow the latest developments of hardware technologies available on AWS and we deploy as much as we can on the most recent platforms with the highest performance; going forward, we will consider deploying some of our instances on new ARM-based servers to save about 30% to 40% compute cost compared to x86-based hardware, without compromising performance.
Networking
- We changed our cloud ingress architecture (infrastructure) – While we knew that AWS (like the other cloud providers) didn’t charge ingress traffic to the cloud, we didn’t realize at the beginning of broadpeak.io operation that the way ingress traffic was distributed within the cloud could generate extra costs; for instance, when load balancing is required to spread that traffic to multiple cloud instances and/or multiple regions, costs are incurred for these load balancing services; therefore we optimized our ingress distribution architecture accordingly.
- We compressed manifest files (application) – To optimize our egress traffic bill with AWS, we compressed our HLS and DASH manifest files for our ad/content insertion and replacement services, using gzip and Brotli algorithms (more information on our blog: Compressing broadpeak.io manifests).
Other cloud optimizations we achieved on broadpeak.io
- We realized it was most cost-effective to use AWS managed Kubernetes proposition EKS instead of deploying our own orchestration layer.
- We disabled after some time the upload of our log data into AWS monitoring application Cloud Watch, a paid service, because we were using our own observability tools.
We took advantage of the public cloud to create our multitenant video system
Building a multi-tenant infrastructure on-prem for our SaaS video proposition broadpeak.io would have been a challenge for us in terms of up-front costs, scalability (add new tenants in a reasonable timeframe), and global coverage (NB. same challenges would have been encountered to create a platform for an Enterprise or wholesale proposition).
Not mentioning that we needed to keep the data, compute resources and business logics of our tenants totally separate and secure, which would have not allowed to benefit much from mutualization.
AWS public cloud platform was particularly well suited to operate our multi-tenant system, thanks to its scale, elasticity and global presence. Adding new tenants to the platform has been easy and has not required much engineering anticipation nor financial up-front investment.
We have fully benefited from AWS’s agile, scalable and cost-effective platform for our multi-tenant broadpeak.io SaaS proposition.
Other lessons we learned from our various cloud customer projects
While broadpeak.io was a key milestone for us to learn how to best use the public cloud and then be able to share that know-how with our customers, we also learned and applied some other best practices thanks to our experience of cloud deployments with our customers. Here below are some examples.
Storage
- We leveraged cold storage (infrastructure). In a recent Cloud DVR customer use case prepared with Google Cloud, thanks to our comprehensive experience of the life cycle of a record, we scheduled the move of assets to Google Cloud Coldline storage after 15 days (with instant retrieval to keep the experience). We saved from 45% to 65% of the storage cost.
- We did the same on a similar case with AWS after realizing that the savings based on the Usage Model of the operator’s subscribers and on the retention period could be huge. On our case, we had to store 10 TBytes of media with 1 million requests to the storage; using AWS’s Intelligent-Tiering Storage Class, we saved 40% when transitioning to Infrequent Access and 80% with a final transition towards Archive Access Instant Tier.
- We used fragment containerization (application) – On another Cloud DVR case using AWS, we aggregated all TV channels streams (videos, audio, subtitle tracks) into one single mp4 fragment to reduce the amount of PUT and GET onto the storage; indeed, AWS like the other public cloud services charge storage costs based on the number of requests made to the object storage; we cut storage cost by 90% on this case thanks to this technique.
- We used manifest aggregation (application) – On that same customer case, in order to avoid sending a PUT every time the manifests was updated, i.e. every fragment duration, we stored every hour an intermediate manifest (so of one hour long) in a shared RAM storage (from Redis); we reduced the number of PUT requests and saved 10% on the storage costs.
- We optimized topology & rules (architecture) – As we were planning the deployment on Google Cloud platform our VOD Origin-Packaging solution for a customer, we studied a new architecture option: we reconfigured the caching tiers and the affinity rules between just-in-time-packagers (in agreement with our customer); we evaluated that storage costs would be reduced by up to 25%.
Networking
- We optimized cloud egress (architecture) – Because a guaranteed bandwidth was not absolutely required, we had the opportunity on another recent VOD/Live Origin & Cloud DVR case to choose to connect the Origin-Packaging applications deployed in the cloud to the CDN on-prem infrastructure through CloudFront – used as a shield cache – instead of using AWS Direct Connect; we saved 20% on the cost (less egress, and no need for on-prem shield cache).
- In that project, we also suggested our customer to bring its own IP addresses after we noticed that AWS was charging the rental of IP addresses.
Conclusion
Public cloud platforms and services can be a real benefit for the video streaming and content delivery industry if one knows how to use them appropriately. We have gained a solid expertise in that field both in terms of making a smart use of public cloud services, options and tiers, but also in terms of optimizing our applications and architectures to make them consume less cloud resources.
This has been made possible thanks to the creation of our own flagship SaaS proposition broadpeak.io on the public cloud (AWS), the deployment of our solutions on public cloud platforms for our customers and our close collaboration with cloud service providers.
Today, the public cloud is fully integrated in our proposition.
Deploying video streaming solutions on any public cloud environment – vs. on premises – remains in any case a challenging decision for service providers. Therefore, we keep advising our customers on the best cloud options to choose as well as all the possible optimizations to make, according to their video use cases and needs.
Author
Nivedita Nouvel, Vice President of Marketing at Broadpeak.
Nivedita is in charge of communication and product strategy and positioning.
Before joining Broadpeak, she worked for 3 years as a Product Manager for Envivio, specialist of H.264 encoding and for Thomson, where she was in charge of the IPTV and Mobile TV Service Platform.
She graduated from IMT Atlantique (formerly Télécom Bretagne) engineering school and holds a Master of Science in Satellite Communications from UCL, London.
