Exchange Complaints Review Categorization ML/AI

Section 1: Use Case Identifiers

Use Case ID: HHS-CMS-00006
Agency: HHS
Op Div/Staff Div: CMS
Use Case Topic Area: Law & Justice
Is the AI use case found in the below list of general commercial AI products and services?
None of the above.
What is the intended purpose and expected benefits of the AI?
The Exchange Complaints Review Contractor (ECRC) receives hundreds of consumer complaints from the Federally-facilitated Exchange (FFE) call center daily, who then must manually read each complaint and categorize each complaint into one or more categories according to the Complaint Review Standard Operating Procedure (SOP). The categorization assists CPI in determining the nature of the complaint, the action necessary, and the component within CMS who the complaint should be routed to. In general, each complaint can take up to 10 minutes to review and categorize, including a quality review component.

The output (list of all complaints categorized) allows CMS to expediate routing the complaint to the appropriate CMS caseworker or Issuer to take the appropriate action necessary to make the consumer whole and ensure they can receive the necessary medical treatment. The output when done by AI is done in a matter of minutes, versus weeks when completed by human reviewers.
Describe the AI system's outputs.
CPI utilizes machine learning/artificial intelligence (AI) and continuously feeds the complaints review model thousands of complaints in order to teach the model how to accurately categorize the complaints, based on keywords. This is done in a matter of minutes instead of weeks by a human complaint reviewer.
Stage of Development: Operation and Maintenance
Is the AI use case rights-impacting, safety-impacting, both, or neither?
Neither

Section 2: Use Case Summary

Date Initiated: 01/2022
Date when Acquisition and/or Development began: 01/2022
Date Implemented: 10/2024
Date Retired: N/A
Was the AI system involved in this use case developed (or is it to be developed) under contract(s) or in-house?
Developed with both contracting and in-house resources.
Provide the Procurement Instrument Identifier(s) (PIID) of the contract(s) used.
N/A
Is this AI use case supporting a High-Impact Service Provider (HISP) public-facing service?
No
Does this AI use case disseminate information to the public?
No
How is the agency ensuring compliance with Information Quality Act guidelines, if applicable?
N/A
Does this AI use case involve personally identifiable information (PII) that is maintained by the agency?
Yes
Has the Senior Agency Official for Privacy (SAOP) assessed the privacy risks associated with this AI use case?
ongoing

Section 3: Data and Code

Do you have access to an enterprise data catalog or agency-wide data repository that enables you to identify whether or not the necessary datasets exist and are ready to develop your use case?
Yes
Describe any agency-owned data used to train, fine-tune, and/or evaluate performance of the model(s) used in this use case.
HICS, Marketplace
Is there available documentation for the model training and evaluation data that demonstrates the degree to which it is appropriate to be used in analysis or for making predictions?
Documentation has been partially completed
Which, if any, demographic variables does the AI use case explicitly use as model features?
N/A
Does this project include custom-developed code?
Yes
If the code is open-source, provide the link for the publicly available source code.
N/A

Section 4: AI Enablement and Infrastructure

Does this AI use case have an associated Authority to Operate (ATO) for an AI system?
Yes
System Name: MIDAS
How long have you waited for the necessary developer tools to implement the AI use case?
6-12 months
For this AI use case, is the required IT infrastructure provisioned via a centralized intake form or process inside the agency?
Yes
Do you have a process in place to request access to computing resources for model training and development of the AI involved in this use case?
Yes
Has communication regarding the provisioning of your requested resources been timely?
Yes
How are existing data science tools, libraries, data products, and internally-developed AI infrastructure being re-used for the current AI use case?
None
Has information regarding the AI use case, including performance metrics and intended use of the model, been made available for review and feedback within the agency?
No documentation is available