Published on in Vol 2, No 4 (2021): Oct-Dec

Preprints (earlier versions) of this paper are available at https://preprints.jmir.org/preprint/33199, first published .
Authors’ Response to Peer Reviews of “A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study”

Authors’ Response to Peer Reviews of “A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study”

Authors’ Response to Peer Reviews of “A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study”

Authors’ Response to Peer Reviews

1Quokka, Palo Alto, CA, United States

2Department of Computer Science, University of California, Berkeley, Berkeley, CA, United States

3Department of Cognitive Science, University of California, Berkeley, Berkeley, CA, United States

4Department of Bioengineering, Stanford University, Stanford, CA, United States

Corresponding Author:

Peter Washington, BA, MS

Department of Bioengineering

Stanford University

1265 Welch Rd

Stanford, CA, 94305

United States

Phone: 1 5126800926

Email: peterwashington@stanford.edu


Related ArticlesPreprint: https://preprints.jmir.org/preprint/24972
Peer-Review Report by Anonymous: https://med.jmirx.org/2021/4/e33923/
Peer-Review Report by Maria da Graça Pimentel (Reviewer J): https://med.jmirx.org/2021/4/e33925/
Peer-Review Report by Ziyou Ren (Reviewer K): https://med.jmirx.org/2021/4/e33928/
Peer-Review Report by Ashwag Alasmari (Reviewer L): https://med.jmirx.org/2021/4/e33930/
Peer-Review Report by Hamed Mehdizadeh (Reviewer M): https://med.jmirx.org/2021/4/e33931/
Peer-Review Report by Anonymous: https://med.jmirx.org/2021/4/e33932/
Published Article: https://med.jmirx.org/2021/4/e24972/
JMIRx Med 2021;2(4):e33199

doi:10.2196/33199

Keywords


This is the authors’ response to peer-review reports for “A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study.”


We would like to thank the Editor and Reviewers for excellent comments that have vastly improved the quality of our manuscript [1]. Below, we provide a point-by-point response to reviewers, and we address all points made by all reviewers.

Anonymous Reviewer [2]

1. We thank the anonymous reviewer for pointing out this omission. We have added an entire subsection to the Methods section, titled Quokka System, which explicitly describes the components of the Quokka system in more detail.

2. We apologize for this confusion, and we should have been more careful when originally reporting our results. We meant to convey that we reject the NULL hypothesis that similar proportions of users would participate in local and remote activities during the challenges. Our alternate hypothesis, that users prefer local over remote activities, was therefore confirmed. We have clarified this language throughout the manuscript and have explicitly referred to the hypotheses as H1, H2, and H3.

3. We thank the anonymous reviewer for finding this lack of clarity. We have added a new section to the Methods, called Statistical Tests, which provides details about how we tested for significance. In particular, we added: “To perform statistical testing for H1-H3, we conducted a binomial proportion test, where we used the proportion of local (H1), social (H2), and new (H3) self-reported activities per week. The null hypothesis is that the proportion is 0.5 (equal number of local and remote, social and individual, as well as new and familiar activities). The goal was to determine if the increased rates of local, social, and new self-reported activities was statistically significant. We calculated a Clopper-Pearson Binomial proportion confidence interval for one test, a method which leverages the cumulative probabilities of the binomial distribution.”

4. We have clarified these points throughout the manuscript. We chose the local versus social, remote versus in-person, and familiar versus new categories because these were the unique aspects of Quokka that we wanted to test. Other health and wellness digital health tools do not focus on the social, in-person, and novel experience aspects of well-being, and we wanted to ensure that Quokka would feasibly promote increased activity of these kinds. In order to code these categories from participant questionnaire responses, we hired 3 independent raters recruited on Upwork, a popular web-based freelancing platform that connects workers to job providers. To reach the final category, a majority-rules consensus was taken for the categorical labels provided by raters. In cases where all 3 raters disagreed, the authors provided the final rating. Protected user data were anonymized when provided to Upwork workers. We have updated the manuscript with these details.

5. We have added a new section to our manuscript, called Limitations: Study Design, which highlights the limitation of the analysis consisting of quantitative analysis of coded qualitative data.

6. Because this research was conducted outside of our capacity as students at a University, and our startup company does not have an ethics review committee, no formal University ethics review was sought for this study. However, all users provided informed consent for participation. In particular, all participants consented on signup to a Privacy Policy and a Terms and Conditions that walked through how their data would be used. We ensured that all user data were anonymized during our analyses, and we do not reveal any protected health information or any other identifiable information in the manuscript. We believe that we fulfill JMIR’s requirements for ethics approval [3]. We have added to the manuscript that the research was conducted in accordance with the Declaration of Helsinki.

We are happy to work with the Editor to ensure that this manuscript can fulfill the ethics review requirements on JMIR. Please let us know if there are any additional steps we should take to address this concern.

Reviewer J

1. We thank Reviewer J [4] for pointing out this lack of clarity. The challenges were part of the interventional program that makes use of the established success of community-based social programs for behavior change, in particular through a new community-based social network, Quokka, which was the embodiment of the interventional paradigm we were exploring the feasibility of. We have updated the manuscript to reflect this more clearly.

2. The program focused on one habit per week, although participants were encouraged to stick to whichever habits they found most effective both through the duration of the program and afterwards. A final survey was sent to participants at the end of each program to collect input and feedback from participants. Respondents cited which habits they had continued and were planning to continue from then on, although this was not further assessed after the program completion.

3. We reached out to over 15 US colleges and universities and met with several administrative health services and student health club staff in order to discuss the possibility of running a program on their campuses. Since this was an early pilot, we chose a small subset of schools to coordinate programs based on their overall interest and availability in dedicating time and effort towards participating. We have clarified this point in the manuscript.

4. We believe that the confirmation of the feasibility of Quokka to promote local, social, and novel experiences can be generalized to other universities, particularly given the differences in student body size and demographics between the schools.

Typical demographics of college students in the United States follow an approximately equal split of women and men (with women holding a slight majority). The vast majority of these students are between 18 and 24 years old (87.5% in 2017, when this study took place). By ethnicity, the undergraduate college student population in 2017 was approximately 53% non-Hispanic White, 21% Hispanic, 15% Black, 8% Asian, and 3% non-Hispanic “Other” [5].

For the four universities included in this study, the typical demographics of their undergraduate college students followed a similar pattern to the national statistics: approximately equal split between women/men, primarily between 18 and 24 years old, and predominantly identifying as non-Hispanic White with varying distributions of students identifying as Hispanic, Black, Asian, or “Other.”

We have included this information in the manuscript.

5. We reached out to administrative health services and student health clubs on each campus, meeting with prospective candidates who would be interested in volunteering. Typically, one student health club or group per campus would become the designated “host” while working with other school resources and groups to customize their programs. We have added this information to the manuscript.

6. Typical demographics of college students in the United States follow an approximately equal split of women and men (with women holding a slight majority). The vast majority of these students are between 18 and 24 years old (87.5% in 2017, when this study took place). By ethnicity, the undergraduate college student population in 2017 was approximately 53% non-Hispanic White, 21% Hispanic, 15% Black, 8% Asian, and 3% non-Hispanic “Other” [5].

For the four universities included in this study, the typical demographics of their undergraduate college students followed a similar pattern to the national statistics: approximately equal split between women/men, primarily between 18 and 24 years old, and predominantly identifying as non-Hispanic White with varying distributions of students identifying as Hispanic, Black, Asian, or “Other.”

We have included this information in the manuscript.

7. We thank Reviewer J for identifying these areas for improvement. We have clarified all of these issues throughout the manuscript, and we refer to explicit hypotheses throughout.

8. We have reviewed the manuscript with an eye towards substituting imprecise terms throughout the manuscript.

9. We have now added section headers for the challenges list in the Challenges Themes section.

Reviewer K

1. We agree with this point [6], and we concur that an ideal study would have included outcome measures relating to well-being. Because such an analysis was missing, we have limited the scope of this manuscript to be an “exploratory feasibility study” with the aim of evaluating the Quokka system’s potential for promotion of local, social, and unfamiliar activities as they pertain to healthy habits. A larger, more controlled study with outcome measure tracking will be required to claim anything beyond an exploratory feasibility study.

2. The Quokka Challenge was designed as a new program in the fall academic quarter/semester of 2017 to promote healthier habits in the university setting. The program design and implementation were influenced by prior research in the field, although it was uniquely created for the university setting. This manuscript highlights Quokka’s first pilot programs, evaluating its framework’s potential for increasing participation in healthy habits.

3. The Quokka platform was designed by a team of engineers, including the authors of this manuscript, who designed every aspect of the platform and challenge. Additional names are listed in the manuscript’s Acknowledgments section.

4. We do not have an explanation for the low retention, other than that our program had higher retention than the average well-being digital application [7]. We have included in the manuscript a citation to this study about the user retention of digital wellbeing apps.

5. We did not employ a control group for this study. We agree that this is a limitation which prevents us from describing the presented study as anything beyond an “exploratory feasibility study.” We have emphasized throughout the manuscript that the presented study is only exploratory in nature, and we have emphasized in the Limitations that we do not know whether Quokka increases social, local, or new activities. We only know that when participating in the Quokka challenge, users are more likely to conduct social, local, or new activities than individual, remote, or familiar activities, by a large and statistically significant margin.

6. We thank Reviewer K for pointing out this omission, which we have corrected in the manuscript.

7. Because this research was conducted outside of our capacity as students at a University, and our startup company does not have an ethics review committee, no formal University ethics review was sought for this study. However, all users provided informed consent for participation. In particular, all participants consented on signup to a Privacy Policy and a Terms and Conditions which walked through how their data would be used. We ensured that all user data were anonymized during our analyses, and we do not reveal any protected health information or any other identifiable information in the manuscript. We believe that we fulfill JMIR’s requirements for ethics approval as outlined here [3]. We have added to the manuscript that the research was conducted in accordance with the Declaration of Helsinki.

We are happy to work with the Editor to ensure that this manuscript can fulfill JMIR’s ethics review requirements. Please let us know if there are any additional steps we should take to address this concern.

8. The program was conducted during the fall academic quarter/semester of 2017, and this detail has been added to the manuscript introduction.

Reviewer L

1. We thank Reviewer L [8] for bringing up this point. The Quokka Challenge was designed as a new program in the fall academic quarter/semester of 2017 to promote healthier habits in the university setting. The program design and implementation were influenced by prior research in the field, although it was uniquely created for the university setting. This manuscript now highlights Quokka’s first pilot programs, evaluating its framework’s potential for increasing participation in healthy habits. We have added this information to the manuscript.

2. We thank Reviewer L for pointing out this omission in the manuscript. We have created an explicit Related Work subsection in the Introduction which discusses the present study in the context of prior works.


Anonymous Reviewer [2]

1. We appreciate this response from the anonymous reviewer, and we have extensively proofread our manuscript to minimize any writing errors. We have replaced all instances of “patient” with “user.”

Reviewer J

1. We thank Reviewer J for this suggestion, and we have removed the Related Work section and integrated this material into the Introduction section.

2. We thank Reviewer J for pointing us to the JMIR Instructions for Authors, which we have read and more carefully adhered to in the currently submitted revision, including adding a Comparison With Prior Work subsection in the Discussion section.

3. We have identified prior studies discussed here and in the Introduction; we searched for “digital mental health intervention local community”, “digital mental health intervention online community”, “mental health social network”, and “digital mental health intervention local social network” on Google Scholar as well as the Journal of Medical Internet Research website. We have added this information to the manuscript.

4. We have selected the work currently in the Related Work section in the same way we identified works in the Comparison With Prior Work section, as described above. We have clarified this methodology in the manuscript.

5. We have updated the title to be “Exploratory Feasibility Study of Quokka: A Local Community-Based Social Network for Mental Health and Wellbeing.” This title now reflects the major mental health component of this work.

Reviewer K

1. We greatly thank Reviewer K for the kind review.

Reviewer M

1. We sincerely apologize for our omission of Reviewer M’s comments [9]. This was completely unintentional on our end and was due to a copying error. We are embarrassed by this mistake and we have ensured that all reviewer comments are properly addressed for this revision round. We thank Reviewer M and the Editor for allowing us to address these issues here.

2. We have checked similar papers published in JMIR and we believe that the updated manuscript adheres to the journal style.

3. We thank Reviewer M for pointing out these missing details in the Introduction. We have added a new introductory paragraph that motivates the problem that Quokka attempts to address, namely, the large mental health burden globally. We also discuss the current solutions to mental health care in the second paragraph, including the limitations of these approaches and why a guided yet remote digital health intervention could address these issues.

4. We agree, and we have moved this section down to the Discussion section.

5. We agree that this formatting was not ideal. To address this concern, we have incorporated the old Related Work section into the Introduction section, which now provides several references in the Introduction providing motivation for the work. In addition, we have significantly bolstered the citations when describing the motivation for Quokka and the current solutions which exist today.

6. We have moved these paragraphs to other sections of the paper and we have restructured the Introduction as requested by Reviewer M.

7. We thank Reviewer M for informing us about behavior changing theory (BCT) literature. We have added a section about BCT and prior related studies in the Introduction section (the fourth paragraph in the revised manuscript).

8. We thank Reviewer M for pointing out this omission. We have added a list of manuscripts and clarified that the prior work we are comparing against, and in fact all prior work in this space does not leverage both local health opportunities and community-based programming to drive behavior change in a single social network.

9. We have added an explicit description of the aim of this study in the second to last paragraph of the Introduction section.

10. We sincerely apologize for our omission of Reviewer M’s comments. This was completely unintentional on our end and was due to a copying error. We are embarrassed by this mistake and we have ensured that all reviewer comments are properly addressed for this revision round. We thank Reviewer M and the Editor for allowing us to address these issues here.

11. We have separated what was previously the Methods section into two sections. The first section, called The Quokka Platform and Challenge, describes the Quokka system and program, and the Methods section now starts with the Recruitment subsection.

12. We thank Reviewer M for pointing out our lack of clarity regarding the exact study design and the purpose of the study. We note that the primary goal of this study was not to provide a controlled trial or to claim that Quokka is an intervention. Our goal was to test the feasibility of such a system by verifying that study participants engage in the behaviors suggested by Quokka for the duration of the program. We have clarified in the manuscript all questions asked by Reviewer M. In particular:

We designed the study as a feasibility study with three central hypotheses: (1) users will spend more time on local over remote activities, (2) users will spend more time on novel over familiar experiences, and (3) users will spend more time on social over individual challenges.

We acquired users through several recruitment strategies (eg, emails, posters) at four separate universities.

We did not include any exclusion criteria besides being a student at one of the four target universities.

All users signed a Terms of Service and Privacy Policy providing study consent. All user data is fully anonymized.

Data were collected by a weekly check-in which consisted of a weekly email reminder to fill in a survey form embedded on the Quokka website. All responses were free-form, and we categorized the responses after the study. Details are provided in the Methods section.

The study took place during the fall quarter/semester of 2017 at the respective universities.

13. We appreciate this response, and we have added a paragraph (fourth paragraph in the Introduction) discussing the theories of behavior change that Quokka is based on. Theories of behavior change suggest that intervention effectiveness may be increased through the incorporation of social and cultural factors that also influence behavior [10,11,12]. These theories targeting the lifestyle focus on learning and decision-making to drive action and reflection, but understanding other factors such as individual beliefs, motivations, and the environment, are important for continued maintenance of health as well [13,14]. Examples of behavior change theories that examine these additional factors as applied to health outcomes include the health belief model (ie, behavior change is posited on barriers, benefits, self-efficacy, and threat) and the theory of planned behavior (ie, actions are driven by behavioral intent, subjective norms, and perceived behavioral control) [14,15,16]. Several of these have been in the university setting, which is especially pertinent given the Quokka setting. Quokka builds upon prior works by incorporating social, cultural, and local environmental elements into its framework and examining the effects of these community factors on individual action and reflection. Furthermore, Quokka utilizes several digital intervention techniques (including option-based, attribute-based, and goal-based techniques) that build upon these theories to drive further habit formation and maintenance [11].

14. We have ensured that the paper has a clear and distinct Results section. This Results section include the following subsections: User Statistics (quantitative), Evaluation Outcomes (quantitative), Participation Due to Localized Social Influence (qualitative), Shared Experiences (qualitative), Local Community-Supported Resources (qualitative), and User Reflection (qualitative).

15. We thank Reviewer M for pointing out this omission. We have added an explicit section comparing against prior work and have clarified that the prior work we are comparing against, and in fact all prior work in this space, does not leverage both local health opportunities and community-based programming to drive behavior change in a single social network.

16. We have ensured that the Limitations sections are succinct and discuss the points brought up by Reviewer M.

17. We thank Reviewer M for finding this error in the reference format. We have updated the references according to the provided suggestions and have ensured that our reference format matches that of papers previously published in JMIR.

18. We thank Reviewer M for finding this error in the reference format. We have updated the references according to the provided suggestions.

19. We thank Reviewer M for pointing this out. We have included “et al” for all papers with more than 6 people.

  1. Shih C, Pudipeddi R, Uthayakumar A, Washington P. A local community-based social network for mental health and well-being (Quokka): exploratory feasibility study. JMIRx Med 2021;2(4):e24972 [FREE Full text] [CrossRef]
  2. Anonymous. Peer review of "A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study". JMIRx Med 2021;2(4):e33923 [FREE Full text] [CrossRef]
  3. Do I need ethics approval for social media research? JMIR Publications Knowledge Base and Help Center.   URL: https:/​/support.​jmir.org/​hc/​en-us/​articles/​115001620728-Do-I-need-ethics-approval-for-social-media-research- [accessed 2021-10-08]
  4. da Graça Campos Pimentel M. Peer review of "A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study". JMIRx Med 2021;2(4):e33925 [FREE Full text] [CrossRef]
  5. More Than 76 Million Students Enrolled in U.S. Schools, Census Bureau Reports. United States Census Bureau.   URL: https://www.census.gov/newsroom/press-releases/2018/school-enrollment.html [accessed 2021-10-10]
  6. Ren Z. Peer review of "A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study". JMIRx Med 2021;2(4):e33928 [FREE Full text] [CrossRef]
  7. Baumel A, Muench F, Edan S, Kane JM. Objective user engagement with mental health apps: systematic search and panel-based usage analysis. J Med Internet Res 2019 Sep 25;21(9):e14567 [FREE Full text] [CrossRef] [Medline]
  8. Alasmari A. Peer review of "A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study". JMIRx Med 2021;2(4):e33930 [FREE Full text] [CrossRef]
  9. Mehdizadeh H. Peer review of "A Local Community-Based Social Network for Mental Health and Well-being (Quokka): Exploratory Feasibility Study". JMIRx Med 2021;2(4):e33931 [FREE Full text] [CrossRef]
  10. Davis R, Campbell R, Hildon Z, Hobbs L, Michie S. Theories of behaviour and behaviour change across the social and behavioural sciences: a scoping review. Health Psychol Rev 2015;9(3):323-344 [FREE Full text] [CrossRef] [Medline]
  11. Pellmar TC, Brandt EN, Baird MA. Health and behavior: the interplay of biological, behavioral, and social influences: summary of an Institute of Medicine report. Am J Health Promot 2002;16(4):206-219. [CrossRef] [Medline]
  12. Glanz K, Rimer BK, Viswanath K. Theory, research, and practice in health behavior and health education. In: Health Behavior and Health Education: Theory, Research, and Practice. San Francisco, CA: Jossey-Bass; 2008:23-40.
  13. Zhang C, Lakens D, IJsselsteijn WA. Theory integration for lifestyle behavior change in the digital age: an adaptive decision-making framework. J Med Internet Res 2021 Apr 09;23(4):e17127 [FREE Full text] [CrossRef] [Medline]
  14. Britt RK, Collins WB, Wilson K, Linnemeier G, Englebert AM. eHealth literacy and health behaviors affecting modern college students: a pilot study of issues identified by the American College Health Association. J Med Internet Res 2017 Dec 19;19(12):e392 [FREE Full text] [CrossRef] [Medline]
  15. Janz NK, Becker MH. The Health Belief Model: a decade later. Health Educ Q 1984;11(1):1-47. [CrossRef] [Medline]
  16. Nejad LM, Wertheim EH, Greenwood K. Comparison of the Health Belief Model and the Theory of Planned Behavior in the prediction of dieting and fasting behavior. E-JAP 2005 Jul 08;1(1):63-74. [CrossRef]


BCT: behavior changing theory


Edited by E Meinert; This is a non–peer-reviewed article. submitted 29.09.21; accepted 29.09.21; published 27.10.21

Copyright

©Cynthia Shih, Ruhi Pudipeddi, Arany Uthayakumar, Peter Washington. Originally published in JMIRx Med (https://med.jmirx.org), 27.10.2021.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIRx Med, is properly cited. The complete bibliographic information, a link to the original publication on https://med.jmirx.org/, as well as this copyright and license information must be included.