IT Ops Genie

ABOUT THE PROJECT

Ops Genie is an AI-powered assistant designed to help IT-Ops staff efficiently retrieve accurate answers from a curated library of over 700 internal documents, including policies, procedures, and FAQs. Built using retrieval-augmented generation (RAG) techniques, the tool integrates Amazon Kendra and Bedrock to deliver reliable, source-cited responses tailored to user queries.

Impact: Ops Genie supports 200–300 IT staff responsible for delivering critical services to over 500,000 Massachusetts constituents. By streamlining access to institutional knowledge, the tool reduces time spent searching for information, enhances response consistency, and accelerates onboarding for new staff. Leadership gains visibility through an integrated evaluation dashboard, which offers real-time insights into tool usage, performance, and training needs.

Partner: Massachusetts Executive Office of Health and Human Services (EOHHS)

Project Status: The project is in its final development phase, with pilot testing currently underway. FedRAMP compliance work is actively in progress, and the team is preparing for a broader organizational rollout.

PARTNERS

EXECUTIVE SUMMARY

– Challenge: Massachusetts’ Health and Human Services (HHS) IT-Ops team struggled to retrieve critical information needed to support field workers. Staff relied on inefficient methods like scrolling through chat logs or emailing colleagues, despite a large repository of over 700 support documents.

– Solution: We built Ops Genie, a secure, AI-powered assistant that leverages Retrieval-Augmented Generation (RAG) to answer IT-Ops staff questions using internal documentation from SharePoint and other systems.

– Impact: The tool supports 200–300 IT personnel who indirectly serve over 500,000 Massachusetts constituents.. It improves information access, reduces decision delays, and supports onboarding for new hires.

– Partners: Executive Office of Health and Human Services (EOHHS), Burnes Center for Social Change, EOTSS (AI COE), AWS (cloud infra/tools)

PROBLEM CONTEXT

– Background: IT-Ops staff are the backbone of digital support for HHS programs. With inconsistent access to documentation and informal knowledge-sharing practices, critical services risk delays and miscommunication.

– Urgency/Need: With growing demands and staff turnover, frontline digital workers needed better tools to quickly access up-to-date policies, application processes, and technical steps.

– Target Audience: IT support agents, new hires, training teams, and operational leadership across HHS programs.

INNOVATION PROCESS

– Approach: The solution was developed using agile sprints, iterative testing, and co-design sessions with IT-Ops users. We used a working-backwards approach from user pain points.

– Co-Creation: We conducted interviews and feedback sessions with support agents, gathered insights from training leads, and involved EOTSS for security and compliance alignment.

– Data Sources: Internal SharePoint document libraries, FAQs, policy documents, process guides, and training manuals (over 700 total).

AI SOLUTION OVERVIEW

– What was built:
Ops Genie is a RAG-based chatbot built on AWS infrastructure. It uses Amazon Kendra and Bedrock (Claude 3.5) to retrieve and summarize answers from internal documentation, with built-in source citations and feedback collection.

– Key Features:

• Conversational Interface: Secure web app with SSO login where users ask natural-language questions and receive contextual answers.
• Real-time Citations and Feedback Loop: Responses include source links; admins can review user feedback and adjust documents.
• Evaluation Dashboard: Admins can view usage stats, model accuracy (via RAGAS metrics), and system performance over time.

OUTCOMES & IMPACT

Quantitative Results:

• Time to decision reduced by ~65% during pilot testing
• Answer accuracy improved by 40% after prompt tuning and document refinement

Qualitative Feedback:

“I no longer have to dig through five folders or ping three people just to find one answer.” — IT Support Staff

LESSONS LEARNED

What Worked:

• Close collaboration with end users during design and testing
• Cited answers and daily syncs increased trust and reliability

What Didn’t:

• Inconsistent document quality reduced initial performance
• Early hallucinations required prompt refinement and feedback loop

Adaptability:

• Easily replicable across departments with large document bases

FUTURE ROADMAP

• Migrate to Bedrock Knowledge Bases for simpler architecture and cost savings
•  Integrate topic modeling to identify most asked queries