MassHealth Helper

ABOUT THE PROJECT

Generative AI tool for MassHealth Enrollment Centers (MEC) workers to efficiently reference the large volume of documents about eligibility requirements and application procedures when assisting Commonwealth residents enrolling in MassHealth. MassHealth has programs reaching over 25% of the entire state, and helps vulnerable populations at an even higher rate, serving more than 40% of children in the state and over 60% of residents living in nursing facilities. Everyday thousands of residents call in to these centers with hopes of getting support in the application process to the various healthcare programs that are provided by MassHealth. However, when they do call they constantly face challenges getting the help they need from the 377 current MEC workers.

Anticipated Impact: Faster call resolution and shorter waits. Massachusetts hired one of the co-ops and the other went to work for Apple.

Partner: Massachusetts Executive Office of Health and Human Services

Status: The tool is fully functional and the agency is in the process of deploying to its six call centers.

PARTNERS

Introduction to the MECA

The Difficulty of Enrolling in MassHealth Programs

Reddit users have expressed frustration with the MassHealth enrollment process because of seemingly complex application procedures, long wait times, and difficulties in obtaining prompt assistance in enrolling. Despite these challenges, many acknowledge that MassHealth offers valuable programs that provide essential healthcare coverage to residents. But still the hurdles that residents encounter when enrolling in these programs can be discouraging.

A lot of these responses are due to a lack of a central resource management system. Navigating the extensive array of resources required to assist residents applying for affordable healthcare programs creates a significant challenge for MassHealth Enrollment Center (MEC) workers. With over 250 documents to reference and the added weekly updated policies, workers often spend valuable time manually searching for the right information. This decentralized and fragmented process not only increases the strain on staff—particularly newer employees—but also compromises the consistency of assistance provided to residents.

This inefficiency impacts healthcare accessibility for Massachusetts residents, as workers are less equipped to address questions accurately and promptly. Without a centralized system for resource management, policy updates risk being overlooked, further hindering the ability of staff to deliver high-quality support. By addressing these challenges, we can empower MEC workers with tools that streamline information retrieval, reduce workload, and improve the overall accessibility of healthcare services statewide.

Overview of the MECA

The MassHealth Enrollment Center Assistant (MECA) leverages a conversational AI model to consolidate over 250 documents into a single, accessible chatbot interface. Unlike generic AI tools, MECA is built specifically to address the unique needs of MEC workers, ensuring quick, accurate, and transparent assistance for healthcare-related inquiries. By integrating seamlessly with existing workflows and offering Single Sign-On (SSO) integration, MECA delivers a secure, efficient user experience that enhances the organization’s performance.

Designed for MEC call center workers, MECA enables users to:

• Prompt the chatbot with questions and receive answers within seconds.
• Open all the source documents the chatbot used to generate its response for full transparency.
• Provide feedback on chatbot responses, allowing administrators to quickly address and resolve any issues identified.

For administrators, MECA offers advanced capabilities that ensure robust performance and oversight:

• An LLM evaluator for uploading test cases, assessing chatbot performance, and tracking historical evaluations.
• Simplified file uploads to seamlessly expand the chatbot’s knowledge base.
• Comprehensive analytics, including user feedback, usage rates, response times, and daily user counts.

By integrating these features, MECA not only alleviates the burden of navigating decentralized resources but also fosters transparency, accountability, and ongoing improvement in the support provided to the MEC.

Current Work: MECA’s Features

The MECA has two main interfaces: one for workers in the call center and one for administration.

Chatbot Interface

The chatbot interface is designed to provide MEC call center workers with an intuitive and efficient platform for accessing critical information to answer residents’ questions. Its primary goal is to streamline workflows and reduce response times by offering a centralized hub for navigating MassHealth resources.

From the interface, users can:

• Prompt the chatbot with specific questions and receive accurate answers in seconds.
• Open all source documents the chatbot used to generate its response, ensuring transparency and credibility.
• Provide feedback on chatbot responses, which administrators can review to address issues or improve processes.

Admin Dashboards

Administrators can easily upload documents to the chatbot’s knowledge base through a simple interface. This streamlined process allows them to quickly add new resources, ensuring that the chatbot remains up-to-date with the latest policies and information. The tool supports efficient document management, enabling administrators to maintain an organized and comprehensive knowledge repository for MEC workers.

Administrators can view user feedback in a detailed table that includes each report’s problem, topic, submission date, and the user prompt. The interface allows admins to delete specific data points, refresh the feedback for the latest updates, and download the data as a CSV file for further analysis. Additionally, they can filter results by time frame and topic.

The LLM Evaluator is a powerful feature of the admin dashboard designed to assess the chatbot’s performance. It includes four distinct tabs to ensure comprehensive evaluation capabilities:

1. Current Evaluation: This tab displays the results of the latest evaluation, including detailed scores on accuracy, similarity, and relevance.

2. Past Evaluations: This tab presents a table of all previous evaluations, showing key details such as the date, the test case file used, the evaluation name, and scores for accuracy, similarity, and relevance. This allows for easy tracking of performance trends and improvements over time.

3. Add Test Cases: This tab is where admins can upload new test case files and provides clear guidelines on how to format them for successful evaluations.

4. New Evaluation: This tab enables administrators to select a test file from their uploads and begin a new evaluation.

Together, these features empower administrators to monitor the chatbot’s performance with ease

Lastly, the KPI dashboard provides administrators with a comprehensive table detailing each instance of chatbot usage. This includes key metrics such as the user, prompt, chatbot response, response time, and date. Administrators can download the data for further analysis, refresh the table to access the latest activity, delete specific data points, and filter results by a selected timeframe.

Additionally, the dashboard features a dedicated tab displaying a bar chart of daily user activity, showing the number of unique users per day. This visual representation helps administrators track engagement trends and measure the tool’s adoption over time, complementing the detailed data in the table.

Future Work

Going forward, AI for Impact would like to continue work with EHS to test the current version of MECA to ensure transparency for users and ease of retrieving information. User testing sessions would help refine MECA and assess its impact as we move into the next phase of development.

To address these questions and expand MECA’s capabilities, we would recommend the following improvements:

• Topic Modeling Integration: Introduce advanced topic modeling to categorize and organize usage data, giving administration a clearer picture of the soft spots in employee knowledge.
• Switch to AWS Bedrock Knowledge Bases: Transition the chatbot’s backend to AWS Bedrock to enhance scalability, reliability, and integration with other AWS services, providing a more robust foundation for MECA’s operations.
• Integration into My Workspace: Seamlessly incorporate MECA into the My Workspace platform to unify tools and streamline workflows for MEC workers, ensuring effortless access to the chatbot alongside other critical resources.

These planned developments along with others you may have in mind, will ensure that MECA continues to evolve, empowering MEC workers to provide accurate and timely support to residents across Massachusetts.