Audio lags when switching video sources

I use Zoom to broadcast our church service and use a usb line out from our mixer as the audio and two HDMI inputs as the video. When I switch from one video source to the other the audio volume drops by about 50% for about 30-40 seconds and then returns to normal volume. This only happens through Zoom, the actual audio in the church is unaffected.

There is nothing obvious in the settings but I understand that this is outside of what Zoom was really intended for but has anyone had a similar experience or have any thoughts on how to fix this?