Postmortem Index

Explore incident reports from various companies

Google Meet Livestream degraded quality

Google · Google Meet Livestream

2021-10-25 – 2021-10-26 config-change time

On October 25, 2021, starting at 11:00 UTC, the Google Meet Livestream feature experienced disruptions. These issues led to intermittent degraded quality of experience for a subset of viewers, including higher rebuffer rates and latency in video playback. The problem was resolved by October 26, 2021, at 17:00 UTC, with quality degraded for a total of 4 hours during this period.

Up to 15% of livestream viewers were affected, experiencing higher rebuffer rates and latency. Specifically, viewers with ongoing livestream events encountered rebuffering issues, with affected livestreams experiencing greater than 5% rebuffering of total viewing time.

The incident was traced to an unexpected surge in workload on the backend message delivery mechanism for Google Meet livestreams. This surge caused the system to hit a configuration threshold, leading to the degraded performance.

Google engineers mitigated the impact by reconfiguring the resources required for processing data and system messages, aiming for higher quality livestream playbacks.

Identified improvements include increasing resource allocation for the backend message delivery system in the short term and implementing automatic detection of message delivery overload in the long term. Enhancements to monitoring systems to capture real-time data on livestream quality and updates to alert logic were also planned to proactively address spikes in rebuffering rates.

Keywords

google meetlivestreambufferinglatencyvideobackendworkloadconfiguration