Mass participant disconnections while in Zoom meetings

[CG5978]

I am curious to know if any other Zoom customer out there is seeing what I am seeing in the enterprise account I manage...
Zoom account size" 45K+
We started "monitoring" all meetings that have more than 300 participant connections (not unique user connections, instead the participant count Zoom reports in the API QoS/Call Detail records). On some of those 300+ meetings, we have massive spikes in participant reconnections due to network disconnection errors. To quantify "massive" - sometimes we see 80-100 disconnections in a matter of 1-5 seconds. Disconnects happen for users both working from home and on a corporate managed network... both wired and Wi-Fi.

We have a Zoom support case open, and Zoom has investigated their MMRs - stating there is no issue on the Zoom side.

Are there other large enterprise customers out there who 1) monitor their meetings for network errors, and 2) see massive spikes at times in disconnections due to network errors?

I know we can't control the network end-to-end anymore with the move to cloud based SaaS solutions like Zoom, but this kind of behavior is having a negative impact on the perception of the service for our corporate users.

[Rupert]

Hi @CG5978 

I have some ideas and questions, if you have the time to brainstorm it.

Is there any possible correlation to some other criteria...

Like are all the disconnects in the same geographic region? On the same ISP?

Are the disconnects ever at a notable time, or after a notable meeting duration? 

Are the disconnects reliably from Meetings with Breakout Rooms, or other features like Live Transcription?

Are you running Group HD? 

Are all users on enterprise provisioned hardware, with other common applications? (centrally managed firewall and antivirus for example).

Is this only Meeting, not Webinar?

Certain versions of the Zoom client? Windows or Mac?

Are you using VPNs?

Do you have any on-prem Zoom infrastructure?

Do all your users have Enterprise seats (Meetings up to 500 Participants)?

In a 300+ Meeting, what would be the largest concentration of users in one location (regardless of them having a problem or not).

I have access to some other expertise if we really need to dig deep. But at an Enterprise level, Zoom should be able to help here also.

[CG5978]

Thanks for the reply and nice list of qualifying questions... let me reply back to each:

<<Like are all the disconnects in the same geographic region? On the same ISP?
>> So far, we have these Zoom Datacenter and Geographic locations represented: SG, IN, TY, US, JP - therefore it does not appear to be geo related
<<Are the disconnects ever at a notable time, or after a notable meeting duration?
>> No apparent pattern here... meetings from 5 hours down to 1 hour impacted
<<Are the disconnects reliably from Meetings with Breakout Rooms, or other features like Live Transcription?
>> Also not finding any correlation to the features used in the meetings
<<Are you running Group HD?
>> Yes - at the 720p setting
<<Are all users on enterprise provisioned hardware, with other common applications? (centrally managed firewall and antivirus for example).
>> Yes - most information workers in the company would be using their enterprise provisioned and controlled laptop, which has a typical enterprise suite of cyber security related utilities. e.g. virus scanning, Digital Gaurdian, etc.
<<Is this only Meeting, not Webinar?
>> I am only monitoring (via API pull of QoS/Call Detail data) the 300+ meetings at this time. That said, webinar licenses are tightly controlled, and very large webinars are IT Managed, so if there were mass disconnects being seeing in webinars, I could expect to hear about it.
<<Certain versions of the Zoom client? Windows or Mac?
>> I've pulled data from back in June (version at that time was 5.6.6) where we see the issue occur - through now (version 5.7.6). We don't have a large enough Mac user base to be statistically significant.
<< Are you using VPNs?
>> Yes - however all Zoom traffic (signaling and media) is set to bypass both Proxy and VPN - going direct out the ISP tunnels when at home, and direct out the corporate firewalls when on premises.
<<Do you have any on-prem Zoom infrastructure?
>> No
<<Do all your users have Enterprise seats (Meetings up to 500 Participants)?
>> Yes - Large Meeting 1000
<<In a 300+ Meeting, what would be the largest concentration of users in one location (regardless of them having a problem or not).
>> Largest office worker geo locations would have 3K-9K

[CG5978]

My intent of this posting is to see if other large customers of Zoom see the same thing. 

[CG5978 · Accepted solution]

Update on this posting...
In summary, what we were seeing was being caused by a networking issue in one of the Zoom datacenters over in Singapore. After many months of working the case with Zoom, Zoom found and fixed the issue in late Oct, 2021. I've attached an image/graph (2022-01-27_11-15-47.jpg) of the spike in packet loss we measured, and that was contributing to disconnections in Zoom meetings.