Development and Evaluation of a Mental Health Chatbot Using ChatGPT 4.0: Mixed Methods User Experience Study With Korean Users

There is no author summary for this article yet. Authors can add summaries to their articles on ScienceOpen to make them more accessible to a non-specialist audience.

Abstract

Background

Mental health chatbots have emerged as a promising tool for providing accessible and convenient support to individuals in need. Building on our previous research on digital interventions for loneliness and depression among Korean college students, this study addresses the limitations identified and explores more advanced artificial intelligence–driven solutions.

Objective

This study aimed to develop and evaluate the performance of HoMemeTown Dr. CareSam, an advanced cross-lingual chatbot using ChatGPT 4.0 (OpenAI) to provide seamless support in both English and Korean contexts. The chatbot was designed to address the need for more personalized and culturally sensitive mental health support identified in our previous work while providing an accessible and user-friendly interface for Korean young adults.

Methods

We conducted a mixed methods pilot study with 20 Korean young adults aged 18 to 27 (mean 23.3, SD 1.96) years. The HoMemeTown Dr CareSam chatbot was developed using the GPT application programming interface, incorporating features such as a gratitude journal and risk detection. User satisfaction and chatbot performance were evaluated using quantitative surveys and qualitative feedback, with triangulation used to ensure the validity and robustness of findings through cross-verification of data sources. Comparative analyses were conducted with other large language models chatbots and existing digital therapy tools (Woebot [Woebot Health Inc] and Happify [Twill Inc]).

Results

Users generally expressed positive views towards the chatbot, with positivity and support receiving the highest score on a 10-point scale (mean 9.0, SD 1.2), followed by empathy (mean 8.7, SD 1.6) and active listening (mean 8.0, SD 1.8). However, areas for improvement were noted in professionalism (mean 7.0, SD 2.0), complexity of content (mean 7.4, SD 2.0), and personalization (mean 7.4, SD 2.4). The chatbot demonstrated statistically significant performance differences compared with other large language models chatbots ( F=3.27; P=.047), with more pronounced differences compared with Woebot and Happify ( F=12.94; P<.001). Qualitative feedback highlighted the chatbot’s strengths in providing empathetic responses and a user-friendly interface, while areas for improvement included response speed and the naturalness of Korean language responses.

Conclusions

The HoMemeTown Dr CareSam chatbot shows potential as a cross-lingual mental health support tool, achieving high user satisfaction and demonstrating comparative advantages over existing digital interventions. However, the study’s limited sample size and short-term nature necessitate further research. Future studies should include larger-scale clinical trials, enhanced risk detection features, and integration with existing health care systems to fully realize its potential in supporting mental well-being across different linguistic and cultural contexts.

Related collections

Most cited references 40

Record: found
Abstract: found
Article: not found

Language Models are Few-Shot Learners

Tom B. Brown, Benjamin Mann, Nick Ryder … (2020)

Recent work has demonstrated substantial gains on many NLP tasks and benchmarks by pre-training on a large corpus of text followed by fine-tuning on a specific task. While typically task-agnostic in architecture, this method still requires task-specific fine-tuning datasets of thousands or tens of thousands of examples. By contrast, humans can generally perform a new language task from only a few examples or from simple instructions - something which current NLP systems still largely struggle to do. Here we show that scaling up language models greatly improves task-agnostic, few-shot performance, sometimes even reaching competitiveness with prior state-of-the-art fine-tuning approaches. Specifically, we train GPT-3, an autoregressive language model with 175 billion parameters, 10x more than any previous non-sparse language model, and test its performance in the few-shot setting. For all tasks, GPT-3 is applied without any gradient updates or fine-tuning, with tasks and few-shot demonstrations specified purely via text interaction with the model. GPT-3 achieves strong performance on many NLP datasets, including translation, question-answering, and cloze tasks, as well as several tasks that require on-the-fly reasoning or domain adaptation, such as unscrambling words, using a novel word in a sentence, or performing 3-digit arithmetic. At the same time, we also identify some datasets where GPT-3's few-shot learning still struggles, as well as some datasets where GPT-3 faces methodological issues related to training on large web corpora. Finally, we find that GPT-3 can generate samples of news articles which human evaluators have difficulty distinguishing from articles written by humans. We discuss broader societal impacts of this finding and of GPT-3 in general. 40+32 pages

0 comments Cited 1011 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Positive psychology progress: empirical validation of interventions.

Martin Seligman, Tracy A Steen, Nansook Park … (2005)

Positive psychology has flourished in the last 5 years. The authors review recent developments in the field, including books, meetings, courses, and conferences. They also discuss the newly created classification of character strengths and virtues, a positive complement to the various editions of the Diagnostic and Statistical Manual of Mental Disorders (e. g., American Psychiatric Association, 1994), and present some cross-cultural findings that suggest a surprising ubiquity of strengths and virtues. Finally, the authors focus on psychological interventions that increase individual happiness. In a 6-group, random-assignment, placebo-controlled Internet study, the authors tested 5 purported happiness interventions and 1 plausible control exercise. They found that 3 of the interventions lastingly increased happiness and decreased depressive symptoms. Positive interventions can supplement traditional interventions that relieve suffering and may someday be the practical legacy of positive psychology.

0 comments Cited 608 times – based on 0 reviews      Review now

Bookmark

Record: found
Abstract: found
Article: not found

Advantages and limitations of Internet-based interventions for common mental disorders.

Gerhard Andersson, Nickolai Titov (2014)

Several Internet interventions have been developed and tested for common mental disorders, and the evidence to date shows that these treatments often result in similar outcomes as in face-to-face psychotherapy and that they are cost-effective. In this paper, we first review the pros and cons of how participants in Internet treatment trials have been recruited. We then comment on the assessment procedures often involved in Internet interventions and conclude that, while online questionnaires yield robust results, diagnoses cannot be determined without any contact with the patient. We then review the role of the therapist and conclude that, although treatments including guidance seem to lead to better outcomes than unguided treatments, this guidance can be mainly practical and supportive rather than explicitly therapeutic in orientation. Then we briefly describe the advantages and disadvantages of treatments for mood and anxiety disorders and comment on ways to handle comorbidity often associated with these disorders. Finally we discuss challenges when disseminating Internet interventions. In conclusion, there is now a large body of evidence suggesting that Internet interventions work. Several research questions remain open, including how Internet interventions can be blended with traditional forms of care. Copyright © 2014 World Psychiatric Association.

0 comments Cited 459 times – based on 0 reviews      Review now

Bookmark

All references

Author and article information

Contributors

Boyoung Kang:

ORCID: https://orcid.org/0009-0000-4338-8728

Sungkyunkwan University25-2, Sungkyunkwan-Ro, Jongno-guSeoul, 03063Republic of Korea82 027401770bykang2015@gmail.com

Journal

Journal ID (nlm-ta): JMIR Med Inform

Journal ID (iso-abbrev): JMIR Med Inform

Journal ID (publisher-id): JMI

Title: JMIR Medical Informatics

Publisher: JMIR Publications (Toronto, Canada )

ISSN (Electronic): 2291-9694

Publication date Collection: 2025

Publication date (Electronic): 3 January 2025

Volume: 13

Electronic Location Identifier: e63538

Affiliations

[1 ] Sungkyunkwan University Seoul Republic of Korea

Author notes

Corresponding Author: Boyoung Kang bykang2015@ 123456gmail.com

Author information

Boyoung Kang https://orcid.org/0009-0000-4338-8728

Munpyo Hong https://orcid.org/0009-0008-4795-7500

Article

Publisher ID: v13i1e63538

DOI: 10.2196/63538

PMC ID: 11748427

PubMed ID: 39752663

SO-VID: 06f66cff-df87-4ff8-99bf-4d850d6b6c77

License:

This is an open-access article distributed under the terms of the Creative Commons Attribution License ( https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Medical Informatics, is properly cited. The complete bibliographic information, a link to the original publication on https://medinform.jmir.org/, as well as this copyright and license information must be included.

History

Date received : 23 June 2024

Date revision requested : 4 November 2024

Date revision received : 12 November 2024

Date accepted : 16 November 2024

Comments

Comment on this article

scite_

Smart Citations

Citing PublicationsSupportingMentioningContrasting

View Citations

See how this article has been cited at scite.ai

scite shows how a scientific paper has been cited by providing the context of the citation, a classification describing whether it supports, mentions, or contrasts the cited claim, and a label indicating in which section the citation was made.

Most referenced authors 560

See all reference authors

Submit your digital health research with an established publisher
- celebrating 25 years of open access