This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIRx Med, is properly cited. The complete bibliographic information, a link to the original publication on https://med.jmirx.org/, as well as this copyright and license information must be included.
This paper [
In addition to responding to the reviewers’ comments [
We believe our title matches the study contents. We do not understand how the results of using a new method, applied in a new area (blood transfusion adverse events [AEs]), are “self-evident.” We prefer to keep the title unchanged.
Please see the new subsection “Comparison of the Shakespeare Method to Other Applications of LDA Topic Modeling” at the end of the
“We were unable to find published instances of LDA topic modeling applications for adverse event detection. Furthermore, we found none that apply LDA topic modeling to words or phrases in documents in the group of interest that are filtered to terms that most significantly distinguished a patient group of interest from a comparison group. This filtering process was essential for identifying topics describing the unique qualities of transfused vs nontransfused groups. Also, to our knowledge, we are the first to check the interpretation of documents with large numbers of topics with nontrivial scores.”
Please see the new subsection “Comparison of the Shakespeare Method to Other Applications of LDA Topic Modeling” at the end of the
We agree that natural language processing (NLP) is indispensable to finding potential AEs in unstructured text. Please see the new subsection “Comparison of the Shakespeare Method to Other Applications of LDA Topic Modeling” at the end of the
“LDA topic modeling has been used for a variety of NLP tasks [6,7] (although it can also be used on other high-dimension data) such as text classification and filtering [8].”
We state in the
As described, LDA topic modeling is one step in the Shakespeare method.
In the
“We were unable to find published instances of LDA topic modeling applications for adverse event detection. Furthermore, we found none that apply LDA topic modeling to words or phrases in documents in the group of interest that are filtered to terms that most significantly distinguished a patient group of interest from a comparison group. This filtering process was essential for identifying topics describing the unique qualities of transfused vs nontransfused groups. Also, to our knowledge, we are the first to check the interpretation of documents with large numbers of topics with nontrivial scores.”
Thank you for pointing out this error. We have made the correction to five steps.
We have clarified this sentence in the
“Many methods for finding AEs in text [9-34] rely on predefining possible AEs before searching for prespecified words and phrases or manual labeling (standardization) by investigators. Crucially, events described in text may not necessarily be attributed to AEs [19,35,36]. We wanted to develop a method to identify possible AEs, even if unknown or unattributed, without any prespecifications or standardization of notes.”
We have clarified our statements in the
We are in the process of publishing the code and expect to have a permanent citation in a few weeks. We now cite it as reference 54 in the
The details are in another paper we cited (reference 57).
We added some explanation to the
“An important consideration for LDA is that the number of topics must be selected a priori. The results of topic modeling change depending on the number of topics assigned to a corpus—this is an iterative (hyperparameter tuning) process that requires human judgment to interpret the topics (based on the top terms in each topic) and determine which number of topics best fits the corpus. With too few topics assigned, topics are not cohesive and do not add any clarity or information to an analysis. With too many topics assigned, “incoherent” topics that do not capture terms common to the member documents proliferate; also, useful topics are likely split among smaller, more specific topics, although that does not limit the ability to analyze true clusters in the corpus.
To tune the hyperparameters of the LDA model, we calculated models with the following numbers of topics: 25, 35, 45, 55, 65, 75, 85. We observed (data not shown):
In the
“The chosen number of topics was effective for identifying a range of PTAEs. Evaluation of the overlap of topics and contents of documents identified for varying numbers of topics has not been reported in the literature. Our iterative approach to evaluating different hyperparameters demonstrated to our satisfaction the relative stability of PTAEs indicated by topics.
We determined the number of topics based on our experience of tuning the hyperparameters, the number of TAEs reported in the literature, and the complexities of critical care patients. We were satisfied with the number because there was both overlap of topics that simultaneously had high word and document scores and some incoherent topics with low scores. As the number of topics gets too large, additional topics are uninterpretable, and that as data set size increases, more robust topics are generated [37].”
In the
“Systematic evaluation of the number of topics and other hyperparameters is always necessary for LDA topic modeling in a new setting.”
In the
“Topic modeling is an unsupervised method commonly used in NLP to extract the most relevant terms for each topic (cluster) of similar documents [6,7]. We chose latent Dirichlet allocation (LDA) [8] to accomplish topic modeling of the T documents. LDA is a generative probabilistic model that results in interpretable dimensionality reduction, which means that we reduced 41,664 terms to 45 topics for our data. A topic is a multimodal distribution of terms over an entire vocabulary (in our case, all the filtered terms). A topic consists of co-occurring terms in this corpus of T documents. Each document can have a mixture of these topics. Each topic contribution in a document is a probability (we refer to this as a document topic score); thus, the scores of all topics for a document sum to 1 (see Figure 3D).”
In the new
“As noted before, we were initially surprised that primarily unigrams (and not the longer sequences) appeared to play a significant role in distinguishing transfusion from control texts. We believe it is possible that enough unigrams that were part of meaningful phrases were also in other phrases or were significant on their own to result in relatively higher scores. For example, although “mechanical ventilation” conveys more meaning than just “mechanical” or “ventilation,” each word occurs singly or in phrases other than “mechanical ventilation.” Because bigrams and phrases were important in other LDA studies [38,39], we do not conclude that our unigram finding is necessarily applicable to other study settings. In this data set and blood transfusion situation, including only unigrams would not be expected to have changed the particular unigrams selected during the ensemble classification step. In other studies, it might be important to include n-grams where n>1.”
In the new
“In this data set and blood transfusion situation, including only unigrams would not be expected to have changed the particular unigrams selected during the ensemble classification step. In other studies, it might be important to include n-grams where n>1.”
In the new
“Because bigrams and phrases were important in other LDA studies [38,39], we do not conclude that our unigram finding is necessarily applicable to other study settings.”
We agree. In the
“We present our use of the Shakespeare method for a different surveillance question elsewhere [40].”
The renamed
In the new
“Filtering the vectors to only terms that were important for focusing the topics on clinical conditions specific to transfusion, including reasons for and consequences of transfusion, was important for identifying PTAEs.”
We simplified the statement to:
“We chose the case of transfusion adverse events (TAEs) and potential TAEs (PTAEs) because new TAE types were becoming recognized during the study data period, so we anticipated an opportunity to find unattributed TAEs in the notes.”
Thank you for finding this mistake, which we corrected to “five steps.”
Thank you for finding this typo in the
We finalized the citation for the Shakespeare method software in reference 54, and submitted manuscripts with and without tracked changes that show our changes.
We believe we addressed the reviewer’s [
We disagree that the Shakespeare method is an alternative to NLP, because we leverage NLP, which includes many methods. As part of the Shakespeare method, we used the following NLP methods: n-gram formation, count vectorization, supervised learning, and LDA topic modeling. We mentioned another NLP method, word/phrase searches, in the
The reviewer seems to agree that the dictionary method relies on predefined possible AEs, which could rely on, for example, the Unified Medical Language System vocabulary list and could miss important terms. We are proposing an alternative method to find both expected and unexpected possible AEs, as we state in the
We agree and state in the
The application of the Shakespeare method to blood transfusion is a use scenario, so we do not understand why the reviewer thinks a potential use scenario needs to be included; however, we did include reference 107 as an additional scenario. We do not understand why or how manual review is an example of a potential use scenario. We reported our manual review of the results, so we do not understand what the reviewer means by asking “will more manual reviews be needed for the results.”
We changed the beginning of the sentence to “We examined whether.”
Thank you!
We already stated some limitations. In the subsection “Discussion of Time Periods Case,” we pointed out that removing numerals from alphanumeric words had resulted in the creation of a “junk” topic that we would not recommend doing again. Additionally, in the
We appreciate the reviewer’s praise and hope we have satisfied the concerns.
We agree that it would be great to know the accuracy of the Shakespeare method. Please see “Top-Scoring Documents for Each Transfusion Topic,” where we reviewed a random selection of transfusion admissions and compared them to the transfusion documents with high topic scores.
We trimmed the list of keywords.
We are satisfied with the current state of the
adverse event
electronic health record
latent Dirichlet allocation
natural language processing
None declared.