Support model to manage & triage ExpertAI responses
Implementing a structured support model and effectively using the issue tracking spreadsheet ensures that generative responses remain accurate, relevant, and continuously improved. This enhances the overall user experience and trust in the system.
Short-term support model (Pilot phase)
Pilot users
During the pilot phase, users will continue to interact with GenSearch. These users, selected for their diverse skill sets and credentials, play a crucial role in identifying potential improvements in generative responses.
Issue Reporting
When an issue is identified, it should be reported to a Subject Matter Expert (SME) for validation. If the SME agrees with the concern, the issue is logged in the issue tracking process, with all relevant information filled out and validated.
Use the issue tracking spreadsheet
The provided Excel sheet "GenSearch Issue Tracker" is a comprehensive tool for tracking and resolving issues with generative responses.
Issue reporting
- Raised By: Enter the name of the person or team who identified the issue.
- Question: Document the specific question that led to the undesirable response.
- Answer: Record the AI-generated response that was found to be unsatisfactory.
Evaluation by SMEs
- Expected Answer: Provide the correct or desired response that should have been generated.
- Answer Source: Indicate the source of the expected answer (e.g., specific documentation or knowledge base article).
- Quality Score: SMEs assign a quality score from 1 to 5, where:
- 5 Excellent response.
- 4 Good response with minor improvements needed.
- 3 Acceptable response needing enhancements.
- 2 Poor response requiring significant improvements.
- 1 Unsatisfactory response.
Business impact and comments
- Business Impact: Assess and document the impact of the issue on business operations.
- Comments/Issues: Provide additional comments or specific issues noted during the evaluation.
Detailed analysis
- Answer in a Table?: Indicate if the answer was expected to be in a tabular form.
- Answer in an Attachment?: Indicate if the answer was expected as an attachment.
- Root Cause: Identify the root cause of the issue, such as content gaps or lack of context.
Proposed solution and status
- Proposed Solution: Suggest a solution to resolve the issue, which could involve revising the content or providing additional context.
- Status: Update the current status of the issue (e.g., Open, In Progress, Resolved).
Root cause analysis
The SME or project team should conduct root cause analysis and resolve issues before escalating them further.
Typically, issues can be mitigated by:
- Providing more explicit context within the content
- Addressing content gaps
- Revising the content to improve its semantic clarity
Evaluation criteria
It is crucial to step back and ask whether the content makes sense in isolation. Consider if a person with no prior knowledge or experience could understand and answer the question based on the provided content.
Long-term support model
Transition to business knowledge teams
As the system matures, the support model transitions to involve broader business knowledge teams. This ensures sustained support and continuous improvement over time.
Ownership and championing
- Over time, the pilot group and SMEs become experienced in resolving content gaps and issues. This leads to a steady decline in the number of issues that need to be escalated.
- These experienced users and SMEs become internal champions of issue resolution. They take ownership of the issue resolution process and help embed best practices within the organization.
Train the trainer approach
- The internal champions can implement a "train the trainer" approach, where they train other team members in knowledge management and resolution practices.
- This approach ensures that knowledge and skills are disseminated throughout the organization, making the overall team more self-sufficient and capable of managing and resolving issues independently.
Formalized issue resolution plan
- Evaluating / Tracking: Continuously evaluate and track issues to ensure they are addressed promptly.
- Authoring / Resolving: Work on authoring solutions and resolving issues as they arise.
Roles and responsibilities
- Pilot Authors / SMEs: Handle immediate concerns and provide quick resolutions in the short term.
- Business knowledge teams: Take over in the long term, ensuring ongoing support and improvement. They continue to monitor, evaluate, and update the content based on user feedback and evolving needs.