Feature request: Limit who can answer poll questions

My organisation hosts meetings for member organisations and need to limit who can answer polls to those entitled to vote. As per now we have to move all meeting participants who are not entitled to vote to a breakout room before launching a poll. I suggest a feature be added that allows host and co-hosts to assign voting rights to participants, both on-the-fly and in meeting invitations. Such a feature would eliminate the need for breakout rooms and make meetings go smoother.