2022
Towards Generalizable Methods for Automating Risk Score Calculation
Jennifer J Liang, Eric Lehman, Ananya Iyengar, Diwakar Mahajan, Preethi Raghavan, Cindy Y. Chang, Peter Szolovits
Proceedings of the 21st Workshop on Biomedical Language Processing, pp. 426--431, Association for Computational Linguistics, 2022
Abstract
Clinical risk scores enable clinicians to tabulate a set of patient data into simple scores to stratify patients into risk categories. Although risk scores are widely used to inform decision-making at the point-of-care, collecting the information necessary to calculate such scores requires considerable time and effort. Previous studies have focused on specific risk scores and involved manual curation of relevant terms or codes and heuristics for each data element of a risk score. To support more generalizable methods for risk score calculation, we annotate 100 patients in MIMIC-III with elements of CHA2DS2-VASc and PERC scores, and explore using question answering (QA) and off-the-shelf tools. We show that QA models can achieve comparable or better performance for certain risk score elements as compared to heuristic-based methods, and demonstrate the potential for more scalable risk score automation without the need for expert-curated heuristics. Our annotated dataset will be released to the community to encourage efforts in generalizable methods for automating risk scores.
Hierarchy-aware Adverse Reaction Embeddings for Signal Detection
Venkata Naga Sreeram Joopudi, Bharath Dandala, Ching-Huei Tsou, Jennifer J. Liang
AMIA Annu Symp Proc, 2022
Abstract
Post-market drug surveillance monitors new and evolving treatments for their effectiveness and safety in real-world conditions. A large amount of drug safety surveillance data is captured by spontaneous reporting systems such as the FAERS. Developing automated methods to identify actionable safety signals from these databases is an active area of research. In this paper, we propose two novel network representation learning methods (HARE and T-HARE) for signal detection that jointly utilize association information between drugs and medical outcomes from the FAERS and ancestral information in medical ontologies. We evaluate these methods using two publicly available reference datasets, EU-ADR and OMOP corpus. Experimental results showed that the proposed methods significantly outperformed standard methodologies based on disproportionality metrics and the existing state-of-the-art aer2vec method with statistically significant improvements on both EU-ADR and OMOP datasets. Through quantitative and qualitative analysis, we demonstrate the potential of the proposed methods for effective signal detection.
2021
Toward Understanding Clinical Context of Medication Change Events in Clinical Narratives
Diwakar Mahajan, Jennifer J Liang, Ching-Huei Tsou
AMIA Annu Symp Proc, 2021
Abstract
Understanding medication events in clinical narratives is essential to achieving a complete picture of a patient's medication history. While prior research has explored identification of medication changes in clinical notes, due to the longitudinal and narrative nature of clinical documentation, extraction of medication change alone without the necessary clinical context is insufficient for use in real-world applications, such as medication timeline generation and medication reconciliation. Here, we present a framework to capture multi-dimensional context of medication changes documented in clinical notes. We define specific contextual aspects pertinent to medication change events (i.e. Action, Negation, Temporality, Certainty, and Actor), describe the annotation process and challenges encountered while creating the dataset, and explore models based on state-of-the-art transformers to automate the task. The resulting dataset, Contextualized Medication Event Dataset (CMED), consisting of 9,013 medications annotated over 500 clinical notes, will be released to the community as a shared task in 2021-2022.
Reducing Physicians' Cognitive Load During Chart Review: A Problem-Oriented Summary of the Patient Electronic Record
Jennifer J. Liang, Ching-Huei Tsou, Bharath Dandala, Ananya Poddar, Venkata Joopudi, Diwakar Mahajan, John Prager, Preethi Raghavan, Michele Payne
AMIA Annu Symp Proc, 2021
Abstract
Overabundance of information within electronic health records (EHRs) has resulted in a need for automated systems to mitigate the cognitive burden on physicians utilizing today? s EHR systems. We present ProSPER, a Problem-oriented Summary of the Patient Electronic Record that displays a patient summary centered around an auto-generated problem list and disease-specific views for chronic conditions. ProSPER was developed using 1,500 longitudinal patient records from two large multi-specialty medical groups in the United States, and leverages multiple natural language processing (NLP) components targeting various fundamental (e.g. syntactic analysis), clinical (e.g. adverse drug event extraction) and summarizing (e.g. problem list generation) tasks. We report evaluation results for each component and discuss how specific components address existing physician challenges in reviewing EHR data. This work demonstrates the need to leverage holistic information in EHRs to build a comprehensive summarization application, and the potential for NLP-based applications to support physicians and improve clinical care.
IBMResearch at MEDIQA 2021: Toward Improving Factual Correctness of Radiology Report Abstractive Summarization
Diwakar Mahajan, Ching-Huei Tsou, Jennifer J Liang
Proceedings of the 20th Workshop on Biomedical Language Processing, pp. 302-310, Association for Computational Linguistics, 2021
Abstract automatic summarization, natural language, correctness, transformer, natural language processing, consistency, task, process, computer science, impression, artificial intelligence
Although recent advances in abstractive summarization systems have achieved high scores on standard natural language metrics like ROUGE, their lack of factual consistency remains an open challenge for their use in sensitive real-world settings such as clinical practice. In this work, we propose a novel approach to improve factual correctness of a summarization system by re-ranking the candidate summaries based on a factual vector of the summary. We applied this process during our participation in MEDIQA 2021 Task 3: Radiology Report Summarization, where the task is to generate an impression summary of a radiology report, given findings and background as inputs. In our system, we first used a transformer-based encoder-decoder model to generate top N candidate impression summaries for a report, then trained another transformer-based model to predict a 14-observations-vector of the impression based on the findings and background of the report, and finally, utilized this vector to re-rank the candidate summaries. We also employed a source-specific ensembling technique to accommodate for distinct writing styles from different radiology report sources. Our approach yielded 2nd place in the challenge.
doi
automatic summarization, natural language, correctness, transformer, natural language processing, consistency, task, process, computer science, impression, artificial intelligence
emrKBQA: A Clinical Knowledge-Base Question Answering Dataset
Preethi Raghavan, Jennifer J Liang, Diwakar Mahajan, Rachita Chandra, Peter Szolovits
Proceedings of the 20th Workshop on Biomedical Language Processing, pp. 64--73, Association for Computational Linguistics, 2021
Abstract
We present emrKBQA, a dataset for answering physician questions from a structured patient record. It consists of questions, logical forms and answers. The questions and logical forms are generated based on real-world physician questions and are slot-filled and answered from patients in the MIMIC-III KB through a semi-automated process. This community-shared release consists of over 940000 question, logical form and answer triplets with 389 types of questions and \textasciitilde7.5 paraphrases per question type. We perform experiments to validate the quality of the dataset and set benchmarks for question to logical form learning that helps answer questions on this dataset.
KAAPA: Knowledge Aware Answers from PDF Analysis
Nicolas Fauceglia, Mustafa Canim, Alfio Gliozzo, Jennifer J Liang, Nancy Xin Ru Wang, Douglas Burdick, Nandana Mihindukulasooriya, Vittorio Castelli, Guy Feigenblat, David Konopnicki, Yannis Katsis, Radu Florian, Yunyao Li, Salim Roukos, Avirup Sil
Proceedings of the AAAI Conference on Artificial Intelligence 35(18), 16029-16031, 2021
2020
Timely and Efficient AI Insights on EHR: System Design
Parthasarathy Suryanarayanan, Edward A. Epstein, Abhishek Malvankar, Burn L. Lewis, Lou Degenaro, Jennifer J. Liang, Ching-Huei Tsou, Divya R. Pathak
AMIA ... Annual Symposium proceedings. AMIA Symposium, pp. 1180-1189, 2020
Abstract analytics, documentation, systems architecture, medical record, data science, health care, medical history, systems design, computer science, scheduling
A patients electronic health record (EHR) contains extensive documentation of the patients medical history but is difficult for clinicians to review and find what they are looking for under the time constraints of the clinical setting. Although recent advances in artificial intelligence (AI) in healthcare have shown promise in enhancing clinical diagnosis and decision-making in clinicians day-to-day tasks, the problem of how to implement and scale such computationally expensive analytics remains an open issue. In this work, we present a system architecture that generates AI-based insights from analysis of the entire patient medical record for a multispecialty outpatient facility of over 700,000 patients. Our resulting system is able to generate insights efficiently while handling complexities of scheduling to deliver the results in a timely manner, and handle more than 30,000 updates per day while achieving desirable operating cost-performance goals.
analytics, documentation, systems architecture, medical record, data science, health care, medical history, systems design, computer science, scheduling
Identification of Semantically Similar Sentences in Clinical Notes: Iterative Intermediate Training Using Multi-Task Learning
Diwakar Mahajan, Ananya Poddar, Jennifer J Liang, Yen-Ting Lin, John M Prager, Parthasarathy Suryanarayanan, Preethi Raghavan, Ching-Huei Tsou
JMIR Med Inform 8(11), e22508, 2020
Abstract
Background: Although electronic health records (EHRs) have been widely adopted in health care, effective use of EHR data is often limited because of redundant information in clinical notes introduced by the use of templates and copy-paste during note generation. Thus, it is imperative to develop solutions that can condense information while retaining its value. A step in this direction is measuring the semantic similarity between clinical text snippets. To address this problem, we participated in the 2019 National NLP Clinical Challenges (n2c2)/Open Health Natural Language Processing Consortium (OHNLP) clinical semantic textual similarity (ClinicalSTS) shared task. Objective: This study aims to improve the performance and robustness of semantic textual similarity in the clinical domain by leveraging manually labeled data from related tasks and contextualized embeddings from pretrained transformer-based language models. Methods: The ClinicalSTS data set consists of 1642 pairs of deidentified clinical text snippets annotated in a continuous scale of 0-5, indicating degrees of semantic similarity. We developed an iterative intermediate training approach using multi-task learning (IIT-MTL), a multi-task training approach that employs iterative data set selection. We applied this process to bidirectional encoder representations from transformers on clinical text mining (ClinicalBERT), a pretrained domain-specific transformer-based language model, and fine-tuned the resulting model on the target ClinicalSTS task. We incrementally ensembled the output from applying IIT-MTL on ClinicalBERT with the output of other language models (bidirectional encoder representations from transformers for biomedical text mining [BioBERT], multi-task deep neural networks [MT-DNN], and robustly optimized BERT approach [RoBERTa]) and handcrafted features using regression-based learning algorithms. On the basis of these experiments, we adopted the top-performing configurations as our official submissions. Results: Our system ranked first out of 87 submitted systems in the 2019 n2c2/OHNLP ClinicalSTS challenge, achieving state-of-the-art results with a Pearson correlation coefficient of 0.9010. This winning system was an ensembled model leveraging the output of IIT-MTL on ClinicalBERT with BioBERT, MT-DNN, and handcrafted medication features. Conclusions: This study demonstrates that IIT-MTL is an effective way to leverage annotated data from related tasks to improve performance on a target task with a limited data set. This contribution opens new avenues of exploration for optimized data set selection to generate more robust and universal contextual representations of text in the clinical domain.
Extraction of Information Related to Drug Safety Surveillance From Electronic Health Record Notes: Joint Modeling of Entities and Relations Using Knowledge-Aware Neural Attentive Models
Bharath Dandala, Venkata Joopudi, Ching-Huei Tsou, Jennifer J Liang, Parthasarathy Suryanarayanan
JMIR Med Inform 8(7), e18417, 2020
Abstract
Background: An adverse drug event (ADE) is commonly defined as ``an injury resulting from medical intervention related to a drug.'' Providing information related to ADEs and alerting caregivers at the point of care can reduce the risk of prescription and diagnostic errors and improve health outcomes. ADEs captured in structured data in electronic health records (EHRs) as either coded problems or allergies are often incomplete, leading to underreporting. Therefore, it is important to develop capabilities to process unstructured EHR data in the form of clinical notes, which contain a richer documentation of a patient's ADE. Several natural language processing (NLP) systems have been proposed to automatically extract information related to ADEs. However, the results from these systems showed that significant improvement is still required for the automatic extraction of ADEs from clinical notes. Objective: This study aims to improve the automatic extraction of ADEs and related information such as drugs, their attributes, and reason for administration from the clinical notes of patients. Methods: This research was conducted using discharge summaries from the Medical Information Mart for Intensive Care III (MIMIC-III) database obtained through the 2018 National NLP Clinical Challenges (n2c2) annotated with drugs, drug attributes (ie, strength, form, frequency, route, dosage, duration), ADEs, reasons, and relations between drugs and other entities. We developed a deep learning--based system for extracting these drug-centric concepts and relations simultaneously using a joint method enhanced with contextualized embeddings, a position-attention mechanism, and knowledge representations. The joint method generated different sentence representations for each drug, which were then used to extract related concepts and relations simultaneously. Contextualized representations trained on the MIMIC-III database were used to capture context-sensitive meanings of words. The position-attention mechanism amplified the benefits of the joint method by generating sentence representations that capture long-distance relations. Knowledge representations were obtained from graph embeddings created using the US Food and Drug Administration Adverse Event Reporting System database to improve relation extraction, especially when contextual clues were insufficient. Results: Our system achieved new state-of-the-art results on the n2c2 data set, with significant improvements in recognizing crucial drug−reason (F1=0.650 versus F1=0.579) and drug−ADE (F1=0.490 versus F1=0.476) relations. Conclusions: This study presents a system for extracting drug-centric concepts and relations that outperformed current state-of-the-art results and shows that contextualized embeddings, position-attention mechanisms, and knowledge graph embeddings effectively improve deep learning--based concepts and relation extraction. This study demonstrates the potential for deep learning--based methods to help extract real-world evidence from unstructured patient data for drug safety surveillance.
2019
A Novel System for Extractive Clinical Note Summarization using EHR Data
Jennifer Liang, Ching-Huei Tsou, Ananya Poddar
Proceedings of the 2nd Clinical Natural Language Processing Workshop, pp. 46--54, Association for Computational Linguistics, 2019
Abstract
While much data within a patient's electronic health record (EHR) is coded, crucial information concerning the patient's care and management remain buried in unstructured clinical notes, making it difficult and time-consuming for physicians to review during their usual clinical workflow. In this paper, we present our clinical note processing pipeline, which extends beyond basic medical natural language processing (NLP) with concept recognition and relation detection to also include components specific to EHR data, such as structured data associated with the encounter, sentence-level clinical aspects, and structures of the clinical notes. We report on the use of this pipeline in a disease-specific extractive text summarization task on clinical notes, focusing primarily on progress notes by physicians and nurse practitioners. We show how the addition of EHR-specific components to the pipeline resulted in an improvement in our overall system performance and discuss the potential impact of EHR-specific components on other higher-level clinical NLP tasks.
2018
2017
Ground Truth Creation for Complex Clinical NLP Tasks - an Iterative Vetting Approach and Lessons Learned
Jennifer J. Liang, Ching-Huei Tsou, Murthy V. Devarakonda
AMIA Joint Summits on Translational Science proceedings. AMIA Joint Summits on Translational Science2017, 203-212
Abstract problem list, ground truth, vetting, cognitive load, task, natural language processing, computer science, quality, cognition, artificial intelligence, in patient
Natural language processing (NLP) holds the promise of effectively analyzing patient record data to reduce cognitive load on physicians and clinicians in patient care, clinical research, and hospital operations management. A critical need in developing such methods is the "ground truth" dataset needed for training and testing the algorithms. Beyond localizable, relatively simple tasks, ground truth creation is a significant challenge because medical experts, just as physicians in patient care, have to assimilate vast amounts of data in EHR systems. To mitigate potential inaccuracies of the cognitive challenges, we present an iterative vetting approach for creating the ground truth for complex NLP tasks. In this paper, we present the methodology, and report on its use for an automated problem list generation task, its effect on the ground truth quality and system accuracy, and lessons learned from the effort.
problem list, ground truth, vetting, cognitive load, task, natural language processing, computer science, quality, cognition, artificial intelligence, in patient
Automated Problem List Generation and Physicians Perspective from a Pilot Study
Devarakonda, Murthy V and Mehta, Neil and Tsou, Ching-Huei and Liang, Jennifer J and Nowacki, Amy S and Jelovsek, John Eric
International Journal of Medical Informatics, Elsevier, 2017
Abstract
Objective An accurate, comprehensive and up-to-date problem list can help clinicians provide patient-centered care. Unfortunately, problem-lists created and maintained in electronic health records by providers tend to be inaccurate, duplicative and out of date. With advances in machine learning and natural language processing, it is possible to automatically generate a problem list from the data in the EHR and keep it current. In this ÿ&
SemanticFind: Locating What You Want in a Patient Record, Not Just What You Ask For
J.M.Prager, J.J.Liang, M.V.Devarakonda
2017 AMIA Joint Summits on Translational Science
Abstract
We present a new model of patient record search, called SemanticFind, which goes beyond traditional textual and medical synonym matches in locating patient data that a clinician would want to see rather than just what they ask for. The new model is implemented by making extensive use of the UMLS semantic network, distributional semantics, and NLP, to match query terms along several dimensions in a patient record with the returned matches organized accordingly. The new approach finds all clinically related concepts without the user having to ask for them. An evaluation of the accuracy of SemanticFind shows that it found twice as many relevant matches compared to those found by literal (traditional) search alone, along with very high precision and recall. These results suggest potential uses for SemanticFind in clinical practice, retrospective chart reviews, and in automated extraction of quality metrics.
2016
A Watson generated problem list: Restoring order to the EMR entropy?
Neil Mehta, John Eric Jelovsec, Julie Tibo, Murthy V Devarakonda, Ching-Huei Tsou, Jennifer L. Liang, Amy S. Nowacki
(Poster) Society of General Internal Medicine (SGIM) 2016 Annual Meeting
2015
Toward Generating Domain-specific / Personalized Problem Lists from Electronic Medical Records
Ching-Huei Tsou, Murthy V. Devarakonda, Jennifer Liang
AAAI Fall Symposia, 2015
2014
914