How to evaluate GenResponses

The Completions Report API endpoint gives you data about the queries and responses site visitors are generating using Expert LLM tools on your site. This information shows:

• How users ask questions
• The responses generative AI returned for those questions
• The pages the responses were pulled from to answer the query.

Each row in the report represents a Completions event.  

API endpoint:  GET {site-url}/@api/deki/llm/completion/report?month=YYYY-MM

Admins can call the report from the browser by modifying their URL as follows: {site-url}/@api/deki/llm/completion/report?month=YYYY-MM.

• Expected Answer: Provide the correct or desired response that should have been generated. 
• Answer Source: Indicate the source of the expected answer (e.g., specific documentation or knowledge base article). 
• Quality Score: SMEs assign a quality score from 1 to 5, where: 
  • 5 Excellent response. 
  • 4 Good response with minor improvements needed. 
  • 3 Acceptable response needing enhancements. 
  • 2 Poor response requiring significant improvements. 
  • 1 Unsatisfactory response. 

Example of response evaluation:

A user asks GenSearch, "What are Touchpoints?"

• Excellent answer (Rating = 5): "A Touchpoint is an easy-to-use embeddable tool that allows the extension of Expert content into third-party locations to efficiently achieve desired customer success outcomes."
  • This response scores highly because it is a context-specific overview of what Expert Touchpoints are and what they can do.
• Unsatisfactory answer (Rating = 2): "A touchpoint is a point of contact or interaction between a business or brand and a customer."
  • This response scores poorly but above a 1 because it is not incorrect, but the response is based more off an LLM's foundational training than specific information found in the Expert KB, so it likely will not meet audience intent.