How to improve Generative Responses
Update GenResponses to answer customer questions
There are two ways to update GenResponses to answer customer queries:
- Add human-readable content that contains relevant keywords (recommended)
- Modify prompts (longer prompts typically slow results down)
Response faithfulness
Generative responses are based on existing documentation and the interpretation of the completions engine (LLM). Instead of thinking of a response as being right or wrong, it is helpful to think about how faithful the response is to the existing documentation.
Most "incorrect" responses are faithful to the existing documentation, but may be perceived as incorrect by an SME because of the additional context and insight the SME has. The completion engine can only base its responses on the available context and insight provided within the documentation, and therefore its response can only be “faithful” to the documentation, not to the SME's tacit knowledge.
Example:
You have content for a retirement community, and there are rules associated with the level of service customers paid for. Level 1 costs $10,000 US, and Level 2 costs $100,000 US per month. A user asks your GenSearch, "Can Level 2 members drive a golf cart while intoxicated?" A human will know driving while intoxicated is never allowed. GenSearch will not know this unless it is specifically called out in the content. If specifics about intoxication are not called out in the content, the model may not answer this question, or may answer incorrectly.
To rectify this, we recommend triaging your content faithfulness.
Content faithfulness triage
There are 3 main categories that unfaithful question / answer pairs fall into as part of a triaging resolution approach:
- Content does not exist.
- Content does not have enough context to make it the most relevant article for a generative response.
- Context is inaccurate or missing.
Identify and fill content gaps
Filling content gaps helps ensure customers can find answers to all of their questions.
Steps:
- Collect information about search queries and responses.
- Fill the content gaps.
New content published in Expert will automatically be indexed and will appear in search results shortly after. - Adjust the GenResponse persona.
After you complete these steps, retest your search queries to check that the content gaps you identified have been filled and customers can access the information they are seeking.
Collect information about search queries and responses
There are three ways to collect information about how your customers use GenResponses:
- Use GenResponses the way your site visitors do. This gives you firsthand experience and insight into the experiences your customers are having.
- Run a report of customer queries. Customers may be using different terminology than you would to search, and being able to provide answers using their own words is helpful in getting them connected to the information they are looking for.
- Examine your search queries to get an idea of how users phrase their questions.
- Identify content gaps using the Kernel report.
- Download a report of queries and responses using the API endpoint
@api/deki/llm/completion/report?month=YYYY-MM.
Fill content gaps by adding relevant content
Now that you have gathered the information about how people use GenResponses, you can identify content gaps, and then remedy them with relevant content
- Create a new user to test GenResponses. This allows you to later reverse any actions that you have made under one easily identifiable account.
- Generate a list of test questions and responses, and add these to a spreadsheet. This sheet is where you will identify content gaps.
The test questions can be from customer queries, internal queries, queries you or your team think customers will ask, or a combination. The responses can be an "expected response" or an "actual response" depending on whether the question is asked to create a GenResponse or not. - Note information about the responses. The goal is to verify if content is available or not, so that you know if the content you need is missing, outdated, or difficult to find.
- Create content for questions that have no answers. This can be as simple as a few items in a bulleted list, or a brief intro paragraph to generate a response.
Adjust the persona and threshold settings
Once content exists to fill the gaps, adjust the persona and threshold settings to preserve your brand's voice and tone. We recommend creating a persona document to outline aspects of GenResponses such as: tone / voice, personality, and how your GenSearch should interact with your site visitors.