Automated Analysis of Injury Control Research Center (ICRC) Annual Progress Reports (APRs) using Large Language Models

Section 1: Use Case Identifiers

Use Case ID: HHS-CDC-00003
Agency: HHS
Op Div/Staff Div: CDC
Use Case Topic Area: Mission-Enabling (internal agency support)
Is the AI use case found in the below list of general commercial AI products and services?
None of the above.
What is the intended purpose and expected benefits of the AI?
This AI affects the process of reviewing and analyzing the Annual Progress Reports (APRs) submitted by Injury Control Research Centers (ICRCs). Specifically, the AI will focus on identifying and analyzing text sections of the APRs that detail reported challenges, as well as other text-heavy sections in future stages of the project. The AI is designed to streamline the review process, improve efficiency, and support the evaluation of the performance and progress of ICRC-funded activities.

The output of the AI will help to quickly and efficiently identify key challenges and insights from ICRC APRs, enabling more effective decision-making in the review process. By automating the extraction and analysis of critical information, the AI allows the ICRC team to focus on higher-level evaluation and strategic planning. The AI will help reduce the time and resources needed for manual review, improve the consistency and accuracy of assessments, and facilitate faster responses to ICRC needs. Ultimately, this will support ICRCs in overcoming challenges and achieving their research and injury control goals, benefiting the public health system as a whole.
Describe the AI system's outputs.
The AI analyzes the textual content of APRs. We have started our analysis by focusing on the sections detailing the challenges faced by ICRCs during the 2019 funding cycle. The AI identifies key themes, trends, and critical information that may require further attention. The AI methodology extracts insights and patterns from the data, which can then be compared with manual qualitative analysis outcomes. In subsequent stages, the AI will be expanded to analyze other sections of the APRs, such as progress toward goals and program impact.
Stage of Development: Acquisition and/or Development
Is the AI use case rights-impacting, safety-impacting, both, or neither?
Neither

Section 2: Use Case Summary

Date Initiated: 08/2023
Date when Acquisition and/or Development began: 12/2023
Date Implemented: N/A
Date Retired: N/A
Was the AI system involved in this use case developed (or is it to be developed) under contract(s) or in-house?
Developed in-house.
Provide the Procurement Instrument Identifier(s) (PIID) of the contract(s) used.
N/A
Is this AI use case supporting a High-Impact Service Provider (HISP) public-facing service?
N/A
Does this AI use case disseminate information to the public?
N/A
How is the agency ensuring compliance with Information Quality Act guidelines, if applicable?
N/A
Does this AI use case involve personally identifiable information (PII) that is maintained by the agency?
N/A
Has the Senior Agency Official for Privacy (SAOP) assessed the privacy risks associated with this AI use case?
ongoing

Section 3: Data and Code

Do you have access to an enterprise data catalog or agency-wide data repository that enables you to identify whether or not the necessary datasets exist and are ready to develop your use case?
No
Describe any agency-owned data used to train, fine-tune, and/or evaluate performance of the model(s) used in this use case.
Injury Control Research Center Annual Progress Reports
Is there available documentation for the model training and evaluation data that demonstrates the degree to which it is appropriate to be used in analysis or for making predictions?
Documentation is widely available
Which, if any, demographic variables does the AI use case explicitly use as model features?
N/A
Does this project include custom-developed code?
N/A
Does the agency have access to the code associated with the AI use case?
N/A
If the code is open-source, provide the link for the publicly available source code.
N/A

Section 4: AI Enablement and Infrastructure

Does this AI use case have an associated Authority to Operate (ATO) for an AI system?
Yes
System Name: Enterprise Data Analytics and Visualization (EDAV) Platform
How long have you waited for the necessary developer tools to implement the AI use case?
Less than 6 months
For this AI use case, is the required IT infrastructure provisioned via a centralized intake form or process inside the agency?
Yes
Do you have a process in place to request access to computing resources for model training and development of the AI involved in this use case?
Yes
Has communication regarding the provisioning of your requested resources been timely?
Yes
How are existing data science tools, libraries, data products, and internally-developed AI infrastructure being re-used for the current AI use case?
Use of existing data platforms
Has information regarding the AI use case, including performance metrics and intended use of the model, been made available for review and feedback within the agency?
Limited documentation for review