Introduction & Setup
The most watched annual sporting event in the world deserved our biggest Snapshot set-up to date. The idea was to provide a worldwide view on the Quality of Experience of the Super Bowl, across many different devices, channels and distribution networks.
The environment for this Snapshot consisted of several Witbe “Video & Media” Robots hosted in 6 Witbe Offices across the world: Denver, London, Montreal, New York, Paris, and San Jose. 14 different devices, 8 apps (Watch ESPN, DirecTV, Molotov, Playstation Vue, MyTF1, Hulu, BBC iPlayer, YoutubeTV), and 7 TV channels (CBS, ESPN, CTV, RDS, TSN1, TF1, BBC) were tested through various delivery technologies (Cable, DVB-T, Fiber, IPTV, OTT, Satellite).
This represented the following 27 unique configurations:
Witbe London office:
- BBC played on BBC iPlayer app on iPhone 8 (OTT)
- BBC played on BBC iPlayer app on Galaxy S8 (OTT)
- BBC One played on BT STB (DVB-T)
Witbe New York office:
- CBS played on CBS Sports app on Apple TV (OTT)
- CBS played on FuboTV app on Roku (OTT)
- CBS played on Verizon Fios STB (Fiber)
- CBS played on Optimum STB (Cable)
Witbe Denver office:
- CBS played on Comcast Xfinity STB (Cable)
- CBS played on DirecTV app on Apple TV (OTT)
- CBS played on Hulu in Google Chrome (OTT)
- ESPN Deportes played on Watch ESPN app on Roku (OTT)
Witbe San Jose office:
- CBS played on ATT Uverse STB (IPTV)
- CBS played on Hulu in Google Chrome (OTT)
- CBS played on PlayStation Vue app on Roku (OTT)
- CBS played on YouTubeTV app on Apple TV (OTT)
Witbe Montreal office:
- CTV played on Vidéotron STB (Cable)
- RDS played on Vidéotron STB (Cable)
- TSN1 played on Vidéotron STB (Cable)
- CTV played in Google Chrome (OTT)
- RDS played in Google Chrome (OTT)
- TSN1 played in Google Chrome (OTT)
Witbe Paris office:
- TF1 played on MyTF1 in Google Chrome (OTT)
- TF1 played on Freebox Revolution STB (Fiber)
- TF1 played on Molotov app on Apple TV (OTT)
- TF1 played on Molotov app on AndroidTV Nvidia Shield (OTT)
- TF1 played on Brandt TNT STB (DVB-T)
- TF1 played on CanalCube STB (Satellite)
The images above display these configurations in Witbe Remote Eye Controller, showing several streams on a single screen.
We measured and verified the following KPIs:
- Global Video Availability: percentage of viewing sessions for which the video both successfully started to play & successfully played for two minutes
- Video Initial Buffering Time: time between the Robot clicking on the play button and the video starting to play
- Witbe VQ-MOS Score: a score representing the video quality of the first 2 minutes of video, measured with our famous Video Quality Mean Opinion Score algorithm
The KPIs were measured every 5 to 10 minutes (depending on the test platform) on every 27 configurations, over the duration of the event (Sunday February 3rd 6:30pm to 10:10pm EST), for a total of more than 800 viewing sessions around the world. Every viewing sessions included a 2-minute Witbe VQ-MOS and VQ-ID analyses, representing more than 50 000 measured KPIs.
All graphs and other visual representations of KPIs in this QoE Snapshot are generated with Witbe Datalab, our new restitution interface for fault finding, root cause analysis and advanced analytics.
Global Overview
The Super Bowl is traditionally watched on a TV screen. When looking at the “main screen devices” KPIs – especially the Witbe VQ-MOS score – we were surprised that OTT devices like Apple TV and Roku performed better on average than regular North American STBs. Based on our measurements, the best overall Quality of Experience was reached when watching the game on CBS on the DirecTV NOW app for Apple TV with the best Witbe VQ-MOS score (4.4/5), while also boasting the second-best Video Initial Buffering Time (2.1s). It is also interesting to note that the same channel, when played on the Verizon Fios STB, obtained the second-lowest Witbe VQ-MOS score (3.5/5).
We also noted that OTT devices did not suffer from any significant delays in Video Initial Buffering Time during the game, even at the beginning or during the half-time show. This can be explained by 2 factors: the OTT services were seemingly adequately sized to handle the load of the event, and most viewers were probably watching the game on traditional cable TV. However, when analyzing Video Initial Buffering Time in more details, we can see that the YoutubeTV app on Apple TV, as well as every tested Roku App (PlayStation Vue, Watch ESPN, and FuboTV), took noticeably longer to load the stream – the longest being FuboTV on Roku with a 17.3s loading time (more than 4x longer than the global average).
When looking at the “secondary screen devices” KPIs, we can see that the Quality of Experience for viewing sessions on web and mobile was satisfying, even if it was on average lower than the “main screen devices”.
To conclude this overview, by looking at the Global Video Availability KPI, we were glad to notice that only 3 viewing sessions – out of 800 – did not successfully stream the Super Bowl, with 1 login error on TSN and 2 playback issues on the Molotov app for Apple TV. You can see the screenshots of these errors below:
The half-time show customary technical difficulties
Managing the streaming of a live event of this scale is a real technical prowess, it requires hundreds of people and a lot of effort and dedication. However, the Super Bowl is regularly prone to technical mishaps.
This year was no different and the half-time show was impacted by three separate one-second audio drops. Witbe Robots detected this problem on the following configurations:
- CBS played on the CBS Sports app on Apple TV in New York
- CBS played on the Comcast Xfinity STB in Denver
- CBS played on the AT&T Uverse STB in San Jose
- CBS played on Hulu in Google Chrome in San Jose
- RDS played in Google Chrome in Montreal
In contrast, this problem was confirmed to be absent on MyTF1 in Paris and on ESPN Deportes in Denver. We can therefore infer that the root cause of this audio glitch was linked to the CBS live feed.
The video above shows the audio drops the Robots detected (Witbe Robots record video traces for every test, in order to replay the scenario and understand what the Robot measured — this video was re-encoded and compressed).
An expensive dynamic ad insertion challenge
To measure major quality degradations, every viewing session included a 2-minute Witbe VQ-ID analysis, in addition to Witbe VQ-MOS.
Witbe VQ-ID focuses on detecting severe quality deterioration events, such as macro blocking, video upscaling, frozen images or uniform screens, error or no broadcast frames, and even saturated audio or absence of sound. The alarms raised by Witbe VQ-ID are displayed in real-time in Witbe Remote Eye Controller, and integrated to Witbe Datalab to understand where major incidents happened.
Witbe VQ-ID raised six 2-second and one 12-second “no sound” alarms. When investigating the cause of these alarms, we realized they were all associated to a black screen and that they all occurred during ads breaks. We deduced they happened because of a dynamic ad insertion issue, where the inserted ad was shorter than the allowed time frame. Considering the 5M$ price tag of a 30-second ad during the Super Bowl, dynamically inserting ads is something worth optimizing.
This issue is similar to last year’s 30-second black screen. During Super Bowl LII, the black screen was also due to a problem with local ad insertion, as we mentioned on our blog at the time.
The video above shows the 12-second black screen the Robot detected (Witbe Robots record video traces for every test, in order to replay the scenario and understand what the Robot measured — this video was re-encoded and compressed).
Conclusion
From a technical perspective, the Super Bowl LIII went as good as one could have expected. No major downtime was detected by the Witbe Robots and the overall Quality of Experience was quite satisfying. Considering the scale of the event and the number of viewers, the technical teams in charge of streaming the event successfully handled the load, especially the teams working on OTT services.
Even with all the machine learning algorithms our Witbe Robots come loaded with, the enjoyment of the game goes beyond the KPIs we measured. We are confident that the Patriots’ fans had a better time than the Rams’. Congratulations to the winning team!
For more details about the results of this QoE Snapshot, don't hesitate to contact us!
See you soon for the next Witbe QoE Snapshot!
About Witbe QoE Snapshots
In the same way that a consumer report tests a product and publishes an analysis of its overall quality, the Witbe QoE Snapshots test digital services to make available to the market information on the true Quality of Experience delivered internationally. These QoE Snapshots should not serve as benchmarks nor as rankings of operators by service or by device. Rather, the goal of these QoE Snapshots is to provide a global overview of digital services with multiple configurations, and in various environments. The public will thus be able to better understand the technological complexity inherent to today’s services, like the distribution of video content. It is quite a technical feat – considering the efforts and means implemented – to broadcast videos on different devices and networks, with a quality that is acceptable by consumers with high expectations.
Since its origin, Witbe relies on a non-intrusive technology, based on Robots measuring the quality truly delivered. The Witbe Robots are placed at the edge of distribution, and connected to test devices, the same ones as those used by real users. The Robots measure the Quality of Experience actually delivered to the end-users by providing KPIs on the availability, performance and integrity of the service.
Each snapshot is composed of several analytical frames, highlighting interesting findings about the KPIs that were measured. In our last QoE Snapshot, we looked at who was the best mobile operator for video apps in Canada. This time, we are publishing an analysis on the Quality of Experience of the most watched annual sporting event in the world: the Super Bowl!
