When joining zoom calls my volume spikes back to full.

As an audio engineer I can tell you that normally, for most users, setting the speaker level on Zoom to max does not present a problem. So there is something happening in your case that you have not shared. At this point I have no info about your computer, its sound card, or the device you are using to listen to Zoom's speaker output. Are voices too loud? Are music tracks over Zoom too loud? Are shared video sound tracks too loud? Without being able to see your system and situation it is difficult to offer a solution. External audio interface have different behaviors and often better control. Also, in addition screenshots of the settings > audio window that shows the mic and speaker settings would be helpful.