I've done this in the past, and never had any problems. I have the latest version of Zoom (it asked to upgrade the experience and I said yes).
Under invitations, I click edit beside 'invite panellists' and get the option to add people. I do this, click send, but nothing happens and they are not saved as a participant, and don't receive an email.
Am I missing something?