Mismatch between results of virtual agent in test view and when deployed via campaign

[mseigal]

Hello,

My company has signed up to Zoom Contact and Zoom Phone Enterprise. I have a good working knowledge of AI voice agents having worked on POC agents in Ragflow, N8N and Elevenlabs. It's early days so I am happy to park this to one side as we are working with a professional partner.

What I am struggling with is the change in behaviour between querying a knowledge base, querying a virtual agent with a minimal prompt and querying a  deployed virtual agent in full screen url campaign deployment. Given I have spent much of the last year, preparing, cleaning and building RAG agent  workflows for text and speech, I am trying to figure out why the same knowledgebase would retrieve different matched content from the same knowledge base using a vanilla prompt.

Are there video tutorials and documentation that go through end to end example projects to give blueprints and examples for start to finish workflows, as is often the case with N8N youtubers?

[VanessaVaZoomie]

@mseigal -- Hello! Thank you for the thoughtful and detailed message and using ZVA! 

There's a few follow-up questions, but it sounds like if your implementation does not meet the criteria below, then you need to submit a support ticket. 

1. Are you observing that a chat agent is providing different responses in preview vs. live chat campaigns? 
2. Are you observing a voice agent is providing different responses in preview vs. live phone calls? 


From there, here are some causes of why the two would be different: 

1. The RAG generation takes in context the history of the chat, so if you had more/less/different history between your preview and live results then that would cause different errors. 

2. Is the difference in answers minimal or noticeably incorrect? If the answer is minimal, and it's pulling the correct information but saying it slightly different, that is because the RAG model is trained to slightly change responses to sound "less robotic." If you don't want it to say a specific word or alter it's vocabulary as much, you can give it more detailed "tone" guidance. This is especially true if your guidance prompts are minimal and not following the template structure. When you have less description in your prompts for how the agent should represent your company, then it sees that as more creative liberty.

2. If you have filler words on and "temperature" above 0, then that is another indicator for the responses to change from engagement to engagement. 

3. Is there two articles with the same-ish title but different information? We see this a lot when you use a website as a knowledge source. It will crawl and ingest all content on the website, so if you have an old blog article that has a different process/price/details and a newer web page maybe announcing an upgrade/change, it doesn't weight them differently. So, it can pull one answer in preview and then will pull the other article in a different preview session or production because it has more than one valid answer to pull from. This is when you should use "exclude" and ensure those older articles/pages are not included. 

But, in the end, if any of these above are not true, then take a Zoom Clip showing the differences, along with engagement ID and file a support ticket under the product = Zoom Virtual Agent. We will need to know the name of the knowledge base that you are using too. 

Thank you!