We used the blood transfusion (n=21,443 admissions) and comparison (n=25,468 admissions) groups described in prior work [57] (Figure 1C).

Our goal for steps 3 and 4 was to filter document vectors to only include terms that were significant to the transfused group and then model the topics within those terms in the transfused group to identify experiences emblematic of transfusion. We formalized the process of extracting these terms by looking at term coefficients associated with a classifier that learns to differentiate the two groups. We underwent an iterative process of trying multiple hyperparameters and classification models to identify these terms. We observed that an ensemble of two classification methods (naïve Bayes [NB] and logistic regression [LR]) and filtering [58-62] was useful for capturing common, infrequent, and rare terms that were significant for T. This term selection resulted in 41,664 terms (Figure 2). We reduced the T document vectors to include only the 41,664 terms (see Figure 1E for a truncated example).

Topic modeling is an unsupervised method commonly used in NLP to extract the most relevant terms for each topic (cluster) of similar documents [63,64]. We chose latent Dirichlet allocation (LDA) [65] to accomplish topic modeling of the T documents. LDA is a generative probabilistic model that results in interpretable dimensionality reduction, which means that we reduced 41,664 terms to 45 topics for our data. A topic is a multimodal distribution of terms over an entire vocabulary (in our case, all the filtered terms). A topic consists of co-occurring terms in this corpus of T documents. Each document can have a mixture of these topics. Each topic contribution in a document is a probability (we refer to this as a document topic score); thus, the scores of all topics for a document sum to 1 (Figure 3D).

We performed topic modeling (Figure 1F,G) by applying the LDA model to the filtered document word vectors (Figure 1E) to find co-occurring terms and group them into topics.

Topic modeling resulted in a matrix of scores for each term by each topic, which we refer to as term scores (Figure 1F). An additional matrix shows the probability of fit for each topic (Figure 1G).

Figure 1G shows the topic document scores, and the maximum topic for each document is circled. This maximum topic is the topic that is the strongest for a document. When the maximum topic score is low, we can infer that the document fits many topics, which in critical care could mean that the patient has many clinical issues, some of which might be PTAEs and should be reviewed.

The maximum document topic scores distribution was plotted in the maximum topic histogram shown in Figure 3A. There were few documents in this corpus with a high maximum topic probability score (Figure 3B, right tail). Most of the documents were comprised of two or more topics (6.1 was the mean number of topics with a minimum score of ≥0.03).

A small number of documents in the left tail of Figure 3C had a low (<20%) maximum topic probability score, meaning that these documents were comprised of many topics. This was further illustrated in the inset (Figure 3D) displaying the topic distribution of a single document from this left tail, which had multiple topics. These extreme documents in the right and left tails were selected for manual review.

An important consideration for LDA is that the number of topics must be selected a priori. The results of topic modeling change depend on the number of topics assigned to a corpus—this is an iterative (hyperparameter tuning) process that requires human judgment to interpret the topics (based on the top terms in each topic) and determine which number of topics best fits the corpus. With too few topics assigned, topics are not cohesive and do not add any clarity or information to an analysis. With too many topics assigned, “incoherent” topics that do not capture terms common to the member documents proliferate; additionally, useful topics are likely split among smaller, more specific topics, although that does not limit the ability to analyze true clusters in the corpus.

To tune the hyperparameters of the LDA model, we calculated models with the following numbers of topics: 25, 35, 45, 55, 65, 75, and 85. We observed (data not shown):

To evaluate whether topics described PTAEs, we selected the following records for manual document review: the three top-scoring documents for each of the 45 topics (Figures 1H and 3A,B), the 7 documents with the most topics with significant scores (≥0.03) (such as in Figure 3C), and 24 randomly selected documents from the T group. We abstracted events, observations, clinicians’ attributions of causality, and clinicians’ diagnoses, as well as their dates (where offered). We used further abstractions and tabulations to protect patients’ confidentiality.

We tested comparisons with the Fisher exact test [69].

A histogram of maximum topic scores (Figure 3A) showed the distribution of each document’s maximum (strongest) topic. There were few documents in this corpus with a high maximum topic probability score (Figure 3B, right tail). The left tail of Figure 3C shows a small number of documents with a maximum topic probability score that is low, or less than 20%, suggesting these documents comprised many topics. Figure 3D illustrates this with the topic distribution of a single document from this left tail. The lowest maximum topic document score was 0.022. Two documents had topic document scores of 0.022 for every topic (Figure 3E). They each had only one short record: a brief electrocardiogram report.

There was no strict relationship between top word score and the frequency distribution of document topic scores (Figure 4). Table 1 shows the categories of maximum document topic scores per number of topics. It shows that if there is one topic, the score is over 0.50. As the number of topics increases, the maximum topic score declines. The average number of topics with a topic document score >0.03 was 6.1. The maximum topic document score was 0.994.

Table S1 (Multimedia Appendix 1) shows, for each topic, the score for the top term, the top 20 terms, the top document score, and the distribution of documents by document score range. The rows are sorted by top document score. The maximum word score ranged from 26 to 91,911. The terms with the top 20 scores included plain English words, clinical words, acronyms, shortened words, and misspellings. The maximum document score for a topic went as high as 0.994. The document scores were widely distributed.

Table S2 (Multimedia Appendix 1) presents the summaries of 135 documents. As is expected when hyperparameters of the model are optimal, most topics (n=35) were “coherent,” meaning the top documents had clear common themes within topics consistent with the lists of the top 20 terms in the topic. The coherent topics had higher top document scores and tended to be the maximum-scoring topics. Among the least coherent topics, the tendency for documents was to have some other topic as the maximum-scoring topic. This is expected with LDA, as the words that do not fit into a coherent topic will be allocated to separate “junk” topics.

The tabulation of the presence or absence of the notes expected to have the most clinical information showed that 122 had a discharge summary, 66 had a nursing note, and 21 had a physician progress note. None of the documents attributed an AE to transfusion in the billing codes.

New or worsening PTAEs occurring within 1 to 2 days in the T group were:

Many documents (n=40) could not be evaluated for TAEs because either the transfusion dates were missing or there was no identified treatment when transfusion could be presumed. For others, there was a clear alternate reason for heart or lung problems: advanced cancer (n=7), thrombotic thrombocytopenic purpura present at admission (n=1), liver failure (n=1), and lung infection (n=1).

Out of the remaining 85 documents with transfusion data, 52 had evidence of PTAEs; the most common were heart PTAEs (n=35) and lung PTAEs (n=33), while non–infection-related fever or chills (n=12) and fluid overload (n=12) were less common. A few documents explicitly considered transfusion as the cause of AEs: in topic 30 (blood disease), one attributed disseminated intravascular coagulation to transfusion and another listed but discarded the possibility of TRALI or TACO, a document in topic 3 (bone trauma from motor vehicle accident) proposed PTAEs, and a document in topic 40 attributed a drop in platelets to transfusion. In 2 documents, the PTAEs were attributed to contrast (topic 37, kidney failure), a brand name for metronidazole (topic 38, colon problem), and surgery (3 cases of bone trauma from a motor vehicle accident).

Documents with transfusion timing but no apparent TAE were in the following topics: 10 (one of the mechanical ventilation topics), 2 (esophageal varices banding), 7 (spine surgery), 18 (gastrointestinal bleeding), 31, and 8. For 10 documents, separate transfusion and PTAE codes were present but were not conceptually linked.

We read 24 randomly selected documents to obtain 20 that did not have advanced cancer, cirrhosis, or severe lung trauma. They are summarized at the bottom of Table S2 (Multimedia Appendix 1).

The documents in the cardiovascular topic group were more likely than the random group to have any of the heart PTAEs (proportion difference=0.47; P=.02). The analogous analysis for 14 documents in the lung failure topic group showed a higher rate of any lung PTAEs (proportion difference=0.37; P=.049).

Table S3 (Multimedia Appendix 1) depicts the characteristics of the 8 documents that had 13 or 14 topics. Their document topic scores were distributed across many topics, and the notes described a large number of medical challenges to the patients. All of these documents had both discharge summaries and nurse progress notes. One physician wrote that the patient developed alloantibodies and had a delayed transfusion reaction. None of the billing codes linked transfusion to an AE, and in 2 records, the codes included an outcome code. All 8 documents provided dates of transfusion, including 3 for which cancer was the more likely cause of the AE. Of the remaining 5 documents, 3 had pulmonary PTAEs:

Incoherent topics had few or no documents with high topic document scores; most documents scored at or close to zero (see example in Figure 5A). A coherent topic follows a similar distribution, but the range is much greater, as seen in the x-axis of Figure 5B when compared to Figure 5A. The coherent topics received higher scores in many documents.

Many topics were conditions that can be reasons for transfusion: anemia [68]; heart attack [71]; blood disease (including blood cancers, chemotherapy, bone marrow transplant, neutropenia, thrombocytopenia, and pancytopenia) [68,72]; major surgery, vascular occlusion or repair, and gastrointestinal problems or bleeding [73]; and tPA to lyse thrombus, because antithrombotic treatment can cause bleeding [74].

Some topics could be consequences of the reasons for transfusion. Tapped pericardial effusion is a candidate because pericardial effusions can result from cancers, heart disease, aortic dissection, and other conditions [75] that prompt transfusion [76]. Past sternotomy, a consequence of heart surgery [77], is often a reason for transfusion [78]. Pneumomediastinum could be caused by surgery, or tearing of the esophagus or trachea [79], which in turn could be a reason to transfuse [73]. Skin breakdown can be a consequence of long-term bed rest [76,80], which is generally associated with critical illness and anemia [68], which in turn prompts transfusion [68].

Some could be alternate reasons for a PTAE: advanced cancer [81], liver disease [82], and infection [83].

Others could be a PTAE or sequelae of PTAEs: mechanical ventilation, which is a known consequence of TAEs [84,85]; pneumomediastinum, which could be caused by mechanical ventilation [79]; a tracheostomy tube, which is placed when long-term mechanical ventilation is anticipated [86]; acute respiratory distress syndrome, which shares features (noncardiogenic pulmonary edema and hypoxia) with TRALI [84] and is also known as acute lung injury and is treated with noninvasive or invasive ventilation [87]; and permanent hemodialysis indicating permanent kidney injury [88], which can result from hemolytic transfusion reactions [89] and is associated with volume overload [90], which is part of TACO [66].

The high number of topics per document reflects the complexity of patients in the critical care unit. Multiple topics covering illnesses and procedures were expected for critically ill patients and were the norm for the vast majority of documents. The documents with 13 and 14 significant topics described many complex clinical problems consistent with the need for critical care. Several of the documents had a variety of PTAEs in more than one category, suggesting the importance of checking the documents with multiple nontrivial topics for PTAEs.

We then divided the study population into three cohorts: admissions starting between July 1, 2001, and June 30, 2006 (period 1; 14,410 documents); July 1, 2006, to June 30, 2007 (period 2; 3581 documents), and July 1, 2007, to June 30, 2008 (period 3; 3296 documents).

To focus on new or increasing AEs, we reduced the number of words to analyze by filtering by whether they were unusual and increasing (or new) in period 3 compared to period 2 (Figures 1C,D and 6A). We adopted two parallel approaches, as shown in Figure 6: (1) binary classification of the notes and (2) analysis of term frequency between periods 3 and 2.

For the binary classification, we fit two classification models: LR with L2/ridge regularization [61] and multinomial NB [59,60]. Model evaluation found LR outperformed NB (with a weighted average F1 score of 0.76 compared to NB’s weighted average F1 of 0.69), but that NB more effectively identified completely new terms in the target time period.

After evaluating the models, we refit both models without a train-test split on the entire 24-month data set and combined the top 5000 features from LR (those with the highest positive coefficient associated with the positive target class) and the top 5000 features from NB (those with the lowest log probability ratio). Combining the lists resulted in a set of 9896 terms.

We used frequency analysis to find emerging rare clinical events. We identified two groups of terms: (1) those which appeared in fewer than 10% of documents in period 2 and saw a 30% increase in raw frequency in period 3, and (2) any terms that never appeared in period 2 and did appear in period 3. For those new terms appearing in period 3, we filtered out digit-only terms (a large number of terms in this group).

For the final feature set, we took the intersection of terms identified from the binary classification and frequency analysis processes. This resulted in 6122 significant terms identified from the initial 117,049 unique terms in the documents from period 3 (5.2% of terms). We revectorized (Figure 1E) the 12-month corpus from period 3 using the combined feature list as our vocabulary (which has the effect of filtering the notes to only include terms in the vocabulary).

The co-occurrence of words in documents in the last time period was analyzed with LDA topic analysis [65]. We chose the final number of topics (n=20) based on a balance of large and small topics and at least one topic with no substantive words. We used the words with the highest scores of their relationship to topics (Figure 1F), as well as the topic document scores that indicate the probability of the topic fit for a document (Figure 1G), to explore topic meanings. We manually read the three top-scoring documents for each topic (Figure 1H).

Documents from selected individual admissions, as well as summary data from July 2001 to June 2008, were used to evaluate whether any topics formed around AEs. Most topics inspired time plots of selected words, diagnosis codes, or procedure codes (see criteria in Table S4, Multimedia Appendix 1) through periods 1, 2, and 3. Slopes were analyzed for changes [92,93].

For this report, out of concern for patient privacy, we substituted generic words (such as “condition01,” “condition02,” etc) for rare conditions, drugs, events, and languages since the year of admission is being presented. Related substitute words (eg, “condition09a,” “condition09b”) were used as synonyms.

For the most common topics, the admissions with the top three topic match scores are summarized in Table S5 (Multimedia Appendix 1). For the topics with words that suggested an interpretation, the records supported the interpretations. For the other topics, the records suggested interpretations that were consistent with the top words. Each of the three top-scoring admissions within a topic were quite similar to each other (an indication that the topics were coherent and the model was working correctly, with the exception of the third admission in topic 3).

The three top-scoring documents for topic 18 described long complex stays, which included large numbers of notes. The general words in the topic (“for,” “hr,” “plan,” “cont,” “today,” “skin,” and “are”) were nearly ubiquitous in periods 2 and 3. The words indicating mechanical ventilation (“vent,” “intubated,” and “trach”) were present in between 51% and 58% of the admissions per quarter in periods 2 and 3, with a slight, clinically insignificant increase for period 3. The lengths of stay and numbers of notes also did not vary between periods 2 and 3.

We noticed that among the five records in Table S5 (Multimedia Appendix 1) that mentioned cardiac catheterization, all mentioned explicit or implied dosing with heparin followed the same day with hypotension that required treatment (heparin is generally part of cardiovascular procedures) [94].

Topics 3 and 7 both have cardiac catheterization for heart problems in common; for 5 out of 6 instances, the procedure or heparin administration was followed by hypotension (4 instances) that needed to be treated or heart rhythm deterioration (1 instance). To investigate whether these potential heparin AEs were increasing between July 2001 and June 2008, we plotted two measures of exposure (an invasive cardiac procedure code and “heparin”) and a measure of AE (“hypotension”). The proportion of admissions that had invasive cardiovascular procedure codes (Figure 7A,B) declined overall (Figure 7A), but had a local increase in period 3 compared to period 2. In contrast to the procedures, the words “heparin” and “hypotension” showed an overall rough increase over the entire time frame. We also noticed that the proportion of admissions with invasive cardiology codes that had the word “hypotension” increased gradually over time (Figures 7A,B), followed by a drop in the last quarter; the pattern was similar and weaker for the proportion of admissions with “heparin” that also had “hypotension.” There was a decrease in “hypotension” in the last quarter, both as a proportion of all admissions, and as a proportion of either indicator of having been exposed to heparin.

Topic 19 (and 13) corresponded with trauma. Figure 8 shows that trauma diagnosis and procedure codes increased steadily over time through periods 1 to 3.

The brain topic (1 and 17, combined) was centered around admissions for brain injury (ie, bleeding, ischemia, or trauma). Figure 9A-C shows that there were local increases in codes for bleeding and ischemia for period 3 compared to period 2. There were slight increases in the codes for all three types of brain injuries overall. The text words indicating these conditions showed similar trends.

Topic 4 describes prolonged drainage after abdominal surgery. The index surgeries were performed before admission for 2 instances and during hospitalization for the third. Figure 10 shows that codes for wounds were quite infrequent. However, long patient stays with words for leaky surgical wound or catheter were more common, rose gradually over time, and had a local increase in period 3, compared to period 2.

Condition01 was the subject of the three admissions with the top match scores for topic 12. The codes and words were generally rare for the three periods and showed a local increase between periods 2 and 3.

Summaries of admissions with topic matching scores for the less common topics are shown in Table S6 (Multimedia Appendix 1). We examined the top-scoring admissions matched to topic 11 and all admissions matched to the others. All admissions in this table had topic match scores for the index topic of <0.15 (column 2). Despite each admission in Table S6 (Multimedia Appendix 1) having at least one strong topic match score for at least one of the strong topics in Table S5 (Multimedia Appendix 1), the topics in Table S6 are distinct from those in Table S5. Some of the topics have admissions that have common aspects (topics 11, 10, 2, 9).

A total of 14 PAEs evident in the notes were distributed among the less common topics: 13 related to medical therapy (6 medications, 3 medical devices, 2 procedures, and 2 combinations) and 2 were nonmedical. Five drug and all of the medical device PAEs were published in the product labels and/or in the medical literature. Of the PAEs, 9 occurred outside the hospital and were related to the reason for admission. The diagnosis and procedure codes generally did not give enough information to understand the specific cause and associated PAE. Figure 11 shows that while the proportions over the 7 years of admissions with allergy and anaphylaxis words steadily decreased, the diagnosis codes for drug AEs and for surgical or procedure-related AEs increased slightly over time.

The other rare and infrequent terms, related diagnosis or procedure codes, and foreign language sentences were rare throughout all three time periods and increased during period 3.