Introduction
Adverse drug reactions (ADRs) are an important source of medication-related harm, yet spontaneous reporting remains affected by under-reporting, delayed recognition, and incomplete clinical context.1., 2. Electronic health record (EHR)-based active surveillance can complement spontaneous reports by integrating medication orders, clinical notes, laboratory results, and treatment changes, but its practical value depends on identifying a small, reviewable subset from a much larger exposed population.3., 4., 5.
This challenge is particularly relevant to medical-institution preparations in China. In this manuscript, “hospital-prepared formulation” refers to a preparation produced by a licensed medical institution under China's medical-institution preparation framework to meet local clinical needs when an appropriate marketed product is unavailable; it is not intended to describe routine bedside compounding of commercially available products.6., 7. Chinese Hospital Pharmacovigilance System (CHPS)-based active monitoring has also been applied in Chinese hospital pharmacovigilance studies.8., 9.
Qingyanhou Mixture is one such institution-prepared, multi-herbal oral formulation containing Gangmei (Ilex asprella), wild chrysanthemum flower (Chrysanthemum indicum), and Taraxaci Herba and is used for acute and chronic pharyngitis, tonsillitis, and related throat symptoms. Because it contains multiple botanical constituents, the present study evaluates suspected ADR causality at the formulation-exposure level and does not attempt to assign an event to an individual constituent or active ingredient. The formulation provides a demanding surveillance case because several indication-related symptoms, including cough, fever, and throat discomfort, overlap with potential adverse-event phenotypes.
Electronic trigger tools offer a transparent first-stage strategy. Automated trigger studies have shown that EHR-derived rules can reduce the number of records requiring manual review, although reported sensitivity and positive predictive value (PPV) vary substantially according to trigger definitions, clinical setting, and reference standard.4., 5. A trigger is therefore most defensible as a candidate-retrieval mechanism rather than an autonomous causality assessment.
In the present setting, baseline symptoms, infection, perioperative status, and concomitant medications create substantial competing explanations that are difficult to resolve using simple drug–symptom co-occurrence alone.
Artificial intelligence (AI) has been applied across pharmaceutical research, including formulation and drug development.10 Large language models (LLMs) provide a complementary capability because they can organize temporally dispersed evidence into a structured case summary. Recent pharmacovigilance and clinical natural language processing (NLP) studies have evaluated language models for medication–adverse event extraction, treatment-emergent event identification, and relation extraction.11., 12., 13., 14., 15., 16., 17. However, general-purpose LLMs may over-attribute non-specific symptoms to a target drug when baseline status and competing etiologies are insufficiently weighted.
Fine-tuning with institution-specific examples may reduce this problem by exposing the model to local note structure, documentation style, temporal presentation, and conservative ADR review conventions.18., 19., 20.
We therefore evaluated a staged, pharmacist-supervised workflow in which an evidence-derived electronic trigger first reduces the search space, after which a locally fine-tuned LLM using low-rank adaptation (LoRA) performs structured case-level ADR assessment and classifies trigger-positive candidates. The workflow differs from a conventional trigger-only study and from direct LLM screening of all exposed episodes. The full exposure cohort was manually adjudicated, allowing the upstream trigger to be evaluated against a cohort-level reference standard, while the LLM was evaluated specifically within the enriched candidate set.
The primary objectives were to quantify candidate reduction and trigger performance, compare the base and LoRA-fine-tuned models on sensitivity, specificity, PPV, and false-positive control among candidates, and define the safety limitations of the staged workflow.
Materials and methods
Study design, data source, and ethics
We conducted a single-center retrospective cohort evaluation using routinely collected inpatient data from 1 January to 31 December 2022. Data were obtained through CHPS, which served as the integration and rule-execution layer linking the hospital information system, EHR, laboratory information system, inpatient medication orders, and progress notes. Hospitalization exposure episodes involving Qingyanhou Mixture were the unit of analysis.
The 7950 eligible exposure episodes corresponded to 7950 unique patients, so no patient contributed more than one study episode. We included all eligible episodes within the calendar-year window specified by the approved institutional project and permitted data scope; we did not perform an a priori sample-size calculation. Reporting was informed by the Standards for Reporting of Diagnostic Accuracy Studies–Artificial Intelligence (STARD-AI) and the Developmental and Exploratory Clinical Investigations of Decision Support Systems Driven by Artificial Intelligence (DECIDE-AI) guidelines.21., 22.
The operational workflow and the independent full-cohort reference-standard pathway are summarized in Fig. 1.Fig. 1Operational workflow for ADR surveillance.
Trigger-flagged cases undergo independent assessment by the base and LoRA-tuned LLMs. Pharmacist adjudication of the full exposed cohort establishes the reference standard. Orange dashed lines indicate comparisons with this standard.
ADR, adverse drug reaction; EHR, electronic health record; LLM, large language model; LoRA, low-rank adaptation.Fig. 1
Inpatient medication orders containing “Qingyanhou Mixture” were used to identify target exposure. Unique patient identifiers, hospitalization numbers, admission times, and index administration times were used for deduplication and record linkage. Direct identifiers were removed before analysis, and model training and inference were performed on a local graphics processing unit (GPU) server; original EHR data were not uploaded to an external cloud service.
The study was approved by the Medical Ethics Committee of Sun Yat-sen Memorial Hospital, Sun Yat-sen University (Approval No. SYSKY-2024-009-01). The requirement for informed consent was waived for this retrospective analysis of de-identified routine clinical data.
Full-cohort pharmacist reference standard
We established the reference standard by manually reviewing all 7950 exposure episodes. We used a structured, qualitative five-domain framework consistent with World Health Organization–Uppsala Monitoring Centre (WHO-UMC) causality principles, rather than a formal Naranjo probability score.23., 24. We considered: (1) a new or clinically worsened manifestation after exposure; (2) a plausible temporal relationship; (3) supportive dechallenge, interruption, symptomatic treatment, or rescue-medication evidence when present; (4) whether the same manifestation was documented before exposure; and (5) whether primary disease, infection, perioperative status, concomitant medication, or another competing explanation was more plausible.
These domains were not conditions that all had to be met. A new or clinically worsened post-exposure manifestation with plausible timing could be classified as Possible despite missing dechallenge information or unresolved competing causes. Pharmacists assessed timing on clinical grounds rather than applying the electronic trigger's > 0 and < 7-day retrieval window as a causality criterion.
For the binary reference standard, we classified Certain, Probable/Likely, and Possible episodes as ADR-positive, using a prespecified threshold of at least Possible. Unlikely, Conditional/Unclassified, and Unassessable/Unclassifiable episodes were non-positive and coded as ADR-negative for analysis. We chose this threshold to evaluate case detection and the active surveillance workflow.
WHO-UMC notes that Possible and Probable are the most frequent categories in routine case reports.24 A previous electronic-trigger study also counted Certain, Probable, and Possible cases as true-positive adverse drug events.5 ADR-positive therefore denotes an operational classification in our reference standard; it does not establish a definite causal relationship.
Two pharmacists independently reviewed all 7950 episodes while blinded to electronic-trigger status and model outputs. They used the same adjudication framework and depth of EHR review for trigger-negative and trigger-positive episodes. We calculated agreement from their initial classifications before consensus, reporting both the full cohort and the trigger-positive subset because κ is affected by outcome prevalence and marginal distributions.
Disagreements were first discussed by the two reviewing pharmacists. Three of the five disagreements were resolved by consensus, while the remaining two were referred to a third senior pharmacist for adjudication.
Evidence-derived electronic trigger
Because the Qingyanhou Mixture package insert does not specify a well-characterized ADR profile, the trigger vocabulary was assembled from multiple evidence sources rather than from the product label alone. Sources included the product information, safety information for three reference preparations selected primarily for overlap in botanical constituents and/or closely related clinical use, 1450 national ADR feedback records retrieved through CHPS, and 29 publications identified from China National Knowledge Infrastructure (CNKI), Wanfang, and the VIP Chinese literature database. Clinical pharmacists and physicians discussed and revised the initial rules with attention to local documentation and rule executability.
These sources were used only to define candidate manifestations for retrieval and were not treated as evidence that the same manifestations were established ADRs of Qingyanhou Mixture.
Candidate manifestations were grouped into six domains: gastrointestinal reactions; skin or hypersensitivity-like reactions; non-specific neurological symptoms; systemic or cardiovascular manifestations; respiratory or throat-related manifestations; and laboratory-abnormality signals. Examples included diarrhea, nausea, vomiting, abdominal pain, rash, pruritus, urticaria, flushing, dizziness, headache, vertigo, fatigue, fever, chest pain, palpitation, cough, dyspnea, throat discomfort, liver-function abnormalities, thrombocytopenia, leukopenia, renal-function abnormalities, and electrolyte disturbances.
The production CHPS trigger was implemented as a composite AND/OR/NOT rule. It required an inpatient exposure episode involving Qingyanhou Mixture and a qualifying event occurring >0 and < 7 days after the index administration. A case met the trigger if either the prespecified laboratory branch or the clinical-note branch was satisfied.
The clinical-note branch required ward-round or progress-note documentation containing at least one prespecified clinical manifestation together with at least one supporting textual cue: the combined terms “patient reports” and “patient developed,” an explicit “adverse reaction” term, or a term indicating drug interruption or discontinuation. Admission or baseline documentation containing a candidate manifestation was excluded through the predefined NOT branch. The complete human-readable trigger is provided in Supplementary Fig.
S1, with its corresponding Extensible Markup Language (XML) representation in Supplementary Data S1. Triggering created a candidate set for further review and did not itself assign drug causality. Multiple trigger hits within the same hospitalization episode were collapsed to one candidate episode.
We finalized and locked the trigger, including its >0 and < 7-day time window, before full-cohort reference-standard review and model evaluation. We did not use the 7950 evaluation episodes for patient-level pilot testing or iterative tuning, and the rule was unchanged after the 28 ADR-positive reference labels became available. We chose this window to capture early inpatient events after administration.
Related Chinese medicine safety studies provide context for this choice: Xuebijing surveillance covered treatment and seven days after withdrawal, while all 54 ADRs in the Shu-Xue-Ning study occurred within seven days of dosing.25., 26. These studies involved different products and observation definitions, so they do not validate the exact cutoff for Qingyanhou Mixture. The window may miss delayed reactions, and alternative windows need further evaluation.
Case-level review dataset
For trigger-positive episodes, a case-level dataset was generated to standardize the information available to both LLMs. Fields included target-drug exposure, index administration time, admission and discharge diagnoses, progress notes within the observation window, concomitant medication orders, relevant laboratory abnormalities, drug interruption or discontinuation, symptomatic treatment or rescue medication, and baseline manifestations. Manual labels and model outputs were stored separately and were not included in model inputs.
When multiple records existed within a field, entries were retained in chronological order to preserve temporal context.
LLM assessment of trigger-positive cases and institution-specific LoRA fine-tuning
Qwen3-4B-Instruct served as the base model under a prompt-only configuration. Each de-identified trigger-positive case was converted into a structured prompt requiring the model to identify the suspected event, pre-existing symptoms, post-exposure evidence, interruption or discontinuation of the target drug, symptomatic treatment, relevant laboratory abnormalities, alternative explanations, and an overall suspicion level. Model outputs followed a predefined JavaScript Object Notation (JSON) schema.
We defined “highly suspected” as a clear temporal relationship with strong support from dechallenge or treatment response and no more plausible competing explanation; “suspected” as a new or clinically worsened post-exposure manifestation with plausible timing but incomplete dechallenge evidence or remaining competing factors; “negative” as a pre-existing manifestation without clinically meaningful post-exposure worsening, or an event with a clearly more plausible alternative cause; and “insufficient evidence” as inadequate temporal or clinical information for meaningful causality assessment. For binary evaluation, “highly suspected” and “suspected” were model-positive, while “negative” and “insufficient evidence” were model-non-positive.
The structured output also included a risk score ranging from 0 to 100, reflecting the strength and completeness of the available evidence supporting drug-relatedness. Higher scores indicated stronger evidentiary support for an ADR-related interpretation. The score can support review prioritization but should not be interpreted as a calibrated probability of ADR occurrence.
Supplementary Data S2 and S3 provide the generic prompt template and predefined output schema.
LoRA fine-tuning used 410 de-identified medication-related EHR cases from the same hospital and the same calendar period as the exposure cohort (1 January to 31 December 2022), but involving medicines other than Qingyanhou Mixture. The fine-tuning set included 110 pharmacist-labelled ADR-positive cases and 300 ADR-negative cases: 185 conventional-medicine-only cases (45.1%; 52 positive, 133 negative), 82 herbal/traditional Chinese medicine-only cases (20.0%; 8 positive, 74 negative), 102 mixed cases (24.9%; 43 positive, 59 negative), and 41 other or unclassified cases (10.0%; 7 positive, 34 negative). None involved Qingyanhou Mixture (0/410).
Clinical pharmacists prepared and checked the structured target outputs using a unified ADR review template. We used these cases to adapt the model to the hospital's EHR structure, documentation style, presentation of temporal evidence, and ADR review conventions. No patient in the 410-case fine-tuning set overlapped with any of the 167 trigger-positive evaluation cases, and none of the 167 evaluation cases was used for supervised fine-tuning.
The experiment used 4-bit model loading with LoRA supervised instruction fine-tuning for six epochs, a learning rate of 2 × 10−4, batch size 1, gradient accumulation of 8, maximum input length of 2048 tokens, LoRA rank 8, alpha 16, and dropout 0.05. LoRA adapters targeted the attention projection modules (q_proj, k_proj, v_proj, and o_proj) and feed-forward projection modules (up_proj, down_proj, and gate_proj). Inference used a maximum generation length of 256 tokens with do_sample = False.18., 19., 20.
Because the archived configuration documents 4-bit loading but not the quantization backend, compute dtype, or double-quantization setting, we describe the procedure conservatively as 4-bit quantized LoRA fine-tuning rather than claiming a specific quantized low-rank adaptation (QLoRA) implementation. Model training and inference were performed locally; routine review inference was feasible on a single NVIDIA GeForce RTX 4080 GPU, without sending EHR data to an external cloud service.
Outcomes and statistical analysis
The full-cohort reference standard was used to evaluate the electronic trigger. Trigger sensitivity, specificity, PPV, negative predictive value (NPV), and candidate fraction were calculated from the 2 × 2 table. Candidate enrichment was defined as the prevalence of pharmacist-adjudicated ADR-positive episodes among trigger-positive candidates divided by the prevalence in the full exposed cohort.
The reduction in LLM input volume was calculated as 1 − (trigger-positive episodes / all exposed episodes). Because the 7950 study episodes came from 7950 unique patients, no adjustment for within-patient clustering was required.
The two LLMs were evaluated only among the 167 trigger-positive candidate episodes. We calculated sensitivity, specificity, PPV, NPV, F1 score, and the proportion of candidates classified as positive. Exact Clopper–Pearson 95% confidence intervals (CIs) are reported for binomial proportions.
Cohen's κ was calculated from the pharmacists' pre-consensus binary classifications, with 95% CIs based on the asymptotic standard error and normal approximation. Because the data remain imbalanced, no single aggregate metric was treated as sufficient evidence of model performance.
Because both models were applied to the same 167 candidates, paired model comparisons used two-sided exact McNemar tests. Differences in sensitivity were tested within the 28 pharmacist-adjudicated ADR-positive cases, and differences in specificity were tested within the 139 ADR-negative cases. A two-sided P < 0.05 was used as a conventional reference threshold; exact P values were interpreted together with the small number of ADR-positive cases and the paired reclassification pattern.
We estimated review time retrospectively from routine work records and our assessment of the workflow. These estimates were descriptive; no prospective time-and-motion study was conducted.
Results
Exposure cohort and pharmacist-adjudicated ADR prevalence
CHPS identified 7950 inpatient exposure episodes involving Qingyanhou Mixture from 7950 unique patients. Full-cohort pharmacist adjudication classified 28 episodes as ADR-positive and 7922 as ADR-negative, corresponding to an observed ADR prevalence of 0.35% (28/7950). Baseline characteristics of the exposure cohort are summarized in Table 1.Table 1Descriptive characteristics of the full exposure cohort.CharacteristicAll exposed episodes (n = 7950)Age, years, median (interquartile range [IQR])46.7 (33.4–58.5)Age < 18 years, n (%)894 (11.2)Female sex, n (%)4934 (62.1)Length of stay, days, median (IQR)4.9 (3.0–8.9)Target-drug order count, median (IQR)1.0 (1.0–2.0)Oncology/chemotherapy diagnosis category, n (%)2751 (34.6)Ear, nose, and throat (ENT) diseases, n (%)777 (9.8)Digestive/hepatobiliary diseases, n (%)365 (4.6)Perioperative/surgical conditions, n (%)318 (4.0)Respiratory/infectious diseases, n (%)165 (2.1)Other/unclear, n (%)3574 (45.0)Note: Diagnosis categories are mutually exclusive.
Continuous variables are reported as median (IQR), and categorical variables as number (percentage).
Before consensus, both pharmacists classified 26 episodes as ADR-positive and 7919 as ADR-negative; two were positive only for pharmacist A and three only for pharmacist B. Full-cohort agreement was 99.94% (7945/7950), with Cohen's κ = 0.912 (95% CI, 0.835–0.989). All 7783 trigger-negative episodes were initially negative for both pharmacists.
Among the 167 trigger-positive candidates, agreement was 97.0% (162/167), with κ = 0.894 (95% CI, 0.803–0.985). Three of the five disagreements were resolved by discussion and two by a third senior pharmacist. Of these five episodes, two were ultimately ADR-positive and three ADR-negative, yielding 28 positive and 139 negative episodes in the candidate set.
Electronic trigger performance and candidate enrichment
The electronic trigger selected 167 of the 7950 exposure episodes (2.10%), leaving 7783 trigger-negative episodes. All 28 pharmacist-adjudicated ADR-positive episodes were present in the trigger-positive set. The resulting 2 × 2 table was true positives (TP) = 28, false positives (FP) = 139, true negatives (TN) = 7783, and false negatives (FN) = 0.
Observed trigger sensitivity was 100.0% (95% CI, 87.66–100.0%), specificity 98.25% (97.93–98.52%), PPV 16.77% (11.44–23.31%), and NPV 100.0% (99.95–100.0%).
The trigger therefore reduced the number of episodes requiring LLM processing from 7950 to 167, a 97.90% reduction or a 47.6-fold smaller candidate pool. Because the same 28 ADR-positive episodes were concentrated into 167 candidates, the observed ADR prevalence increased from 0.35% in the full cohort to 16.77% in the trigger-positive set, corresponding to 47.6-fold candidate enrichment.
Base model and LoRA-fine-tuned model performance among trigger-positive candidates
Among the 167 trigger-positive candidates, the base model classified 39 episodes as positive. Of these, 23 were true positives and 16 were false positives; 5 ADR-positive episodes were classified as non-positive, and 123 ADR-negative episodes were correctly classified as non-positive. The resulting sensitivity was 82.14% (95% CI, 63.11–93.94%), specificity 88.49% (81.98–93.28%), PPV 58.97% (42.10–74.43%), NPV 96.09% (91.12–98.72%), and F1 score 68.66%.
These performance measures are summarized in Table 2 and Fig. 2A.Table 2Performance of the electronic trigger and LLM review stages.Stage / modelTPFPTNFNSensitivity, % (95% CI)Specificity, % (95% CI)PPV, % (95% CI)NPV, % (95% CI)F1, %Electronic trigger(full cohort)2813977830100.00 (87.66–100.00)98.25 (97.93–98.52)16.77 (11.44–23.31)100.00 (99.95–100.00)28.72Base model(n = 167)2316123582.14 (63.11–93.94)88.49 (81.98–93.28)58.97 (42.10–74.43)96.09 (91.12–98.72)68.66LoRA-fine-tuned model(n = 167)267132292.86 (76.50–99.12)94.96 (89.90–97.95)78.79 (61.09–91.02)98.51 (94.71–99.82)85.25Abbreviations: LLM, large language model; LoRA, low-rank adaptation; CI, confidence interval; FN, false negative; FP, false positive; NPV, negative predictive value; PPV, positive predictive value; TN, true negative; TP, true positive. Metrics are stage-specific: the electronic trigger was evaluated in the full cohort, whereas both LLMs were evaluated only among the 167 trigger-positive candidates; values should therefore not be compared directly across stages.
The trigger F1 score is shown for completeness but is not the principal screening metric because the trigger was designed to prioritize sensitivity and candidate reduction.Fig. 2Model performance among 167 trigger-flagged cases. (A) Points show estimates and bars indicate exact 95% confidence intervals; F1 scores are point estimates only.
(B) Confusion matrices: rows indicate pharmacist adjudication and columns model assessments. +/− denote ADR-positive/negative; darker shading indicates larger case counts. Two-sided exact McNemar P values were 0.250 and 0.0039 for reference-positive and reference-negative cases, respectively. PPV, positive predictive value; TP, true positive; FN, false negative; FP, false positive; TN, true negative; LLM, large language model; LoRA, low-rank adaptation.Fig.
2
The LoRA-fine-tuned model classified 33 candidates as positive, including 26 true positives and 7 false positives; 2 ADR-positive episodes were classified as non-positive, and 132 ADR-negative episodes were correctly classified as non-positive. Sensitivity was 92.86% (95% CI, 76.50–99.12%), specificity 94.96% (89.90–97.95%), PPV 78.79% (61.09–91.02%), NPV 98.51% (94.71–99.82%), and F1 score 85.25%.
Compared with the base model, the LoRA-fine-tuned model produced fewer total positive classifications (33 vs 39), more true-positive classifications (26 vs 23), and fewer false-positive classifications (7 vs 16). The false-positive count decreased by 56.25%, candidate-level sensitivity increased by 10.71 percentage points, and PPV increased by 19.81 percentage points. The proportion of trigger-positive cases classified as model-positive decreased from 23.35% to 19.76%.
Paired case-level analysis clarified where the performance difference occurred. Among the 28 ADR-positive candidates, the LoRA-fine-tuned model recovered 3 cases that the base model classified as non-positive, while no true positive from the base model became a LoRA false negative; the sensitivity difference was not statistically significant (exact McNemar P = 0.250). Among the 139 ADR-negative candidates, LoRA corrected 9 false positives from the base model and introduced no new false positive (exact McNemar P = 0.0039).
Thus, the strongest inferential evidence concerned false-positive control, whereas the observed gain in sensitivity remained numerically favorable but statistically uncertain. The confusion matrices are shown in Fig. 2B.
Safety interpretation of model-non-positive candidates
Although the LoRA-fine-tuned model achieved candidate-level sensitivity of 92.86%, two of 28 pharmacist-adjudicated ADR-positive episodes would have been missed if model-non-positive candidates were automatically excluded. We therefore propose 100% pharmacist review of cases classified as highly suspected or suspected, as well as model-non-positive cases with a high risk score. The threshold defining a high risk score was not established in the present retrospective study and would need to be prespecified for implementation.
We propose random pharmacist review of at least 20% of the remaining model-non-positive cases, with the review proportion progressively increased if missed ADRs are identified, as detailed in Section 4.4.
Discussion
Principal findings
In this cohort, all 7950 exposed episodes were manually adjudicated, yielding 28 ADR-positive episodes independently of model outputs. The electronic trigger narrowed the LLM review pool to 167 episodes and retained all 28 positives. Within this enriched set, LoRA fine-tuning mainly improved false-positive control while preserving most positive cases.
False-positive classifications among ADR-negative candidates decreased significantly (P = 0.0039). Sensitivity increased from 82.14% to 92.86%, but the paired difference was not statistically significant (P = 0.250), which should be interpreted in light of the small number of ADR-positive cases.
Full-cohort adjudication allowed us to assess whether the trigger retrieved ADR-positive episodes from all exposed patients. We then assessed model classification among the retrieved candidates. These analyses address different stages of the workflow and use different denominators.
Why a trigger-first and LLM-second architecture is appropriate
Electronic triggers are transparent, inexpensive to run, and easy to audit, but they are coarse by design. A high-recall trigger can tolerate false positives when the downstream review can resolve baseline symptoms and competing explanations. In our cohort, the trigger increased the prevalence of pharmacist-adjudicated ADR-positive episodes from 0.35% to 16.77% and reduced the records requiring LLM inference by 97.90%.
This large reduction in review volume is the main practical advantage of the first stage.
The LLM serves a different function. ADR review depends on longitudinal context rather than isolated symptom recognition. A new rash after drug exposure, for example, carries different evidentiary weight from a rash documented before exposure; fever during active infection differs from unexplained fever followed by target-drug interruption.
Prior NLP and LLM pharmacovigilance studies have shown that adverse events and medication relationships can be extracted from text,11., 12., 13., 14., 15., 16., 17. although many studies focus on individual notes, reports, or benchmark extraction tasks. Here, the case-level input combined medication timing, baseline manifestations, post-exposure notes, treatment changes, laboratory data, and alternative explanations before the model assigned a suspicion category.
Parameter-efficient fine-tuning has improved structured clinical language-model tasks in prior studies.17., 18., 19. In our data, the clearest gain after LoRA fine-tuning was false-positive control: 9 false positives from the base model were corrected without introducing a new false positive among 139 ADR-negative candidates. Sensitivity also increased, with 3 additional ADR-positive cases recovered and no positive case lost relative to the base model, but this difference was not statistically significant.
The 410 fine-tuning cases involved other medicines, included both ADR-positive and ADR-negative examples, and were patient-disjoint from the evaluation set, making direct memorization of Qingyanhou Mixture-specific evaluation labels unlikely. The study design, however, cannot identify the mechanism responsible for the performance difference.
Review of cases discordant with the base model pointed to three recurring sources of false-positive attribution: lexical co-occurrence of the target drug and a symptom without enough causal context; conflation of pre-existing or retrospectively documented symptoms with post-exposure events; and underweighting of competing explanations such as infection, perioperative status, primary disease, or concomitant medication. The LoRA-fine-tuned model corrected 9 such false positives and introduced no new false positives among the 139 ADR-negative candidates. This finding suggests more conservative use of baseline, temporal, and competing-cause information, although the study was not designed to establish why fine-tuning produced that change.
Comparison with electronic trigger and pharmacovigilance literature
Automated trigger studies in hospital EHRs have reported widely varying sensitivity and PPV because event prevalence, trigger definitions, and reference standards differ across settings.4., 5. An institution-specific e-trigger tool developed in a Swiss university hospital, for example, achieved moderate sensitivity and highlighted the trade-off between detection performance and review time.5 Our trigger addressed a different, low-prevalence setting, so direct numerical comparison would be misleading. In this cohort, its main value was the combination of substantial candidate reduction and observed capture of all 28 pharmacist-adjudicated positive episodes.
The sensitivity CI remained wide, and external validation is still needed.
The LLM component also differs from traditional machine-learning ADR classifiers. Earlier systems for detecting medications, adverse events, or their relations in clinical text, as well as rapidly trained classification models, focused primarily on extraction or classification tasks,12,13 whereas more recent LLM studies have explored broader treatment-emergent event extraction and pharmacovigilance report processing.14., 15., 16., 17. Recent studies have also evaluated pretrained longitudinal EHR models for adverse-event prediction and examined safeguards for LLM processing of pharmacovigilance reports.27., 28.
These tasks differ from case-level ADR causality assessment. Our workflow retains an interpretable rule-based retrieval stage and an independent pharmacist reference standard. The LLM is then applied only to trigger-positive candidates, where it integrates case-level clinical context and assigns a structured suspicion category, rather than screening all exposed episodes directly.
This modular design also allows the trigger logic, model prompt, fine-tuning data, model classification, and pharmacist adjudication to be examined separately. Table 3 compares the main surveillance approaches by function, use of clinical context, and human review; differences in populations and reference standards preclude direct ranking of their accuracy.Table 3Comparison of approaches to adverse drug reaction surveillance.ApproachFunction and clinical contextHuman review and auditabilityMain strength and limitationSpontaneous reporting1., 2.Reports clinically suspected events; context depends on reporter detail.Healthcare professionals recognize and report cases; documentation varies.Can identify unexpected reactions; under-reporting and missing exposure denominators limit interpretation.Full-cohort EHR reviewReviewers integrate longitudinal records for every exposed patient.Pharmacists directly assess causality and document evidence.Broad case assessment; substantial workload and reviewer subjectivity.Electronic triggers3., 4., 5.Rules retrieve candidates using coded data, text, and time conditions.Explicit rules are auditable; positive hits require clinical review.Efficient, transparent retrieval; limited handling of competing causes.NLP or machine learning12., 13.Extracts events or predicts labels from specified text or features.Human labels and clinical verification are needed; auditability varies by method.Scalable extraction; performance depends on annotation, features, and setting.Direct LLM extraction or classification14., 15., 16., 28.Processes notes or reports; longitudinal coverage depends on the input.Source evidence and generated classifications require verification.Flexible evidence extraction; unsupported attribution and inconsistent output remain concerns.Our staged workflowA drug-specific trigger retrieves candidates; a locally adapted LLM integrates longitudinal evidence and assigns a structured ADR suspicion category and risk score.Pharmacists retain final causality assessment; full review of high-priority cases and random review of the remainder are proposed.Separates retrieval from contextual review; requires local validation and safeguards for residual misses.Note: This is a qualitative comparison of surveillance functions, not a comparative performance study. ADR, adverse drug reaction; EHR, electronic health record; LLM, large language model; NLP, natural language processing.
Practical implications, workload, and safety considerations
The trigger reduced the number of records requiring LLM processing from 7950 to 167 (97.90%). From routine work records and our experience of the workflow, we estimated that full-cohort manual review would take about 20 working days, trigger-assisted review about 4 working days, and trigger-first review with LLM assistance about 1 working day. These were exploratory retrospective estimates.
We did not prospectively time individual reviewers, so the estimates cannot establish productivity or cost-effectiveness. A prospective workflow study should record individual pharmacists' review times, throughput, disagreement resolution, and safety outcomes.
Under the review approach described in Section 3.4, a pharmacist-confirmed false negative in the random sample would prompt an immediate increase in the review proportion among the remaining model-non-positive cases during the current review cycle. If additional review continued to identify missed ADRs, we would progressively expand review according to their frequency, recurrence, and clinical severity. Frequent, repeated or clustered misses, or clinically important ADRs, could warrant review of all remaining model-non-positive cases in that cycle, together with case-level error analysis and revalidation of the trigger and LLM.
Random review would remain at or above 20% even when no further misses were found. We propose 20% as an initial quality-control minimum, not an optimal threshold established by this study.
Generalizability and limitations
Several limitations should be considered. The number of ADR-positive episodes was modest (n = 28), so the observed trigger sensitivity of 100% should be read together with its 95% CI of 87.66–100%, rather than as evidence that future ADRs would never be missed. The study was also retrospective and single-center, and both the trigger vocabulary and the LoRA-fine-tuned model may reflect local documentation, prescribing patterns, and pharmacist conventions.
Qingyanhou Mixture is a Chinese hospital-prepared herbal formulation and is not directly comparable with commercially marketed drugs or with hospital compounding practices in many other countries. The transferable component is the development and review process: an evidence-derived, auditable electronic trigger; local LLM adaptation for structured case-level ADR classification, longitudinal evidence integration, and risk scoring; and final pharmacist causality assessment. The trigger vocabulary, laboratory thresholds and time window, EHR structure, 410-case training set and resulting weights, and risk-score strata and review rules are specific to the local setting.
Changes in ADR prevalence and case mix may alter PPV, NPV, alert burden, and review workload. Use with another medicine, hospital, or EHR system would require local adaptation and independent validation.
Reference-standard adjudication remains susceptible to reviewer subjectivity despite high pre-consensus agreement in both the full cohort and the trigger-positive subset. The 410-case LoRA fine-tuning set came from the same institution and calendar period as the evaluation data, although it involved other medicines and was patient-disjoint from the 167 evaluation cases. This reduces the risk of target-drug-specific label leakage but does not remove the possibility that the observed benefit reflects local documentation structure or review conventions.
The review-time estimates were retrospective rather than prospectively timed. Finally, although the trigger was locked before the 28 positive reference labels became available and was not tuned against those labels, its 100% observed sensitivity remains an internal retrospective estimate that requires temporal or external validation.
Future perspectives
Temporal and multicenter studies should validate the workflow with other medicines and EHR systems and assess longer windows for delayed ADRs. Prospective workflow studies should measure pharmacist review time, case throughput, disagreement resolution, missed ADRs, and clinical outcomes, while evaluating the graduated review strategy described above. For clinical integration and explainability, model outputs should be linked to the underlying source records so that pharmacists can trace and verify the evidence supporting each classification and assess competing explanations.
Deployment will also require appropriate institutional and regulatory review, protection of patient data, version control, and revalidation when the model or clinical data change. Pharmacists and other responsible healthcare professionals should retain final authority over causality assessment and clinical action.
Conclusions
In this retrospective cohort of 7950 inpatient exposure episodes, a CHPS-based electronic trigger reduced the candidate pool to 167 episodes and retrieved all 28 pharmacist-adjudicated ADR-positive episodes observed in the cohort. Among trigger-positive candidates, the LoRA-fine-tuned model showed better false-positive control than the base model (exact McNemar P = 0.0039), while the increase in sensitivity from 82.14% to 92.86% was not statistically significant (P = 0.250). These findings support further evaluation of a staged architecture in which transparent candidate retrieval is followed by LLM-based structured ADR assessment, including evidence integration, categorical classification, and risk scoring.
Because two ADR-positive candidates remained model-non-positive, model outputs should not be used as a hard exclusion rule. Temporal or external validation and prospective human–AI workflow evaluation are required before routine deployment.
CRediT authorship contribution statement
Canhui Liao: Writing – original draft, Visualization, Software, Formal analysis, Data curation. Xinjun Lai: Writing – original draft, Visualization, Software, Formal analysis, Data curation. Wenge Chen: Writing – review & editing, Methodology, Conceptualization.
Jianxiong He: Writing – review & editing, Validation, Methodology, Investigation. Xiaoxia Yu: Writing – review & editing, Visualization, Supervision, Investigation, Conceptualization.
Consent for publication
Not applicable. The manuscript does not contain identifiable individual patient data, images, or case details requiring consent for publication.
Ethical approval
This study was approved by the Medical Ethics Committee of Sun Yat-sen Memorial Hospital, Sun Yat-sen University (Approval No. SYSKY-2024-009-01). The requirement for informed consent was waived because the study used retrospective, de-identified routine clinical data and involved no direct patient contact or intervention.
Funding
This work was supported by the Guangzhou Science and Technology Program (Key Research and Development Program; grant No. 2024B03J1293), the General Program of Qilu Pharmaceutical Joint Fund 2026 (No. 2025A1515220174), and the Guangzhou Science and Technology Plan Project (No.
2024A03J1199).
Declaration of competing interest
The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper