Mondal, Himel, Dhanvijay, Anup Kumar, Juhi, Ayesha, Singh, Amita, Pinjar, Mohammed Jaffer, Kumari, Anita, Mittal, Swati, Kumari, Amita, Mondal, Shaikat. (1402). Assessment of the Capability of ChatGPT-3.5 in Medical Physiology Examination in an Indian Medical School. سامانه مدیریت نشریات علمی, 14(4), 311-317. doi: 10.30476/ijvlms.2023.98496.1221
Himel Mondal; Anup Kumar Dhanvijay; Ayesha Juhi; Amita Singh; Mohammed Jaffer Pinjar; Anita Kumari; Swati Mittal; Amita Kumari; Shaikat Mondal. "Assessment of the Capability of ChatGPT-3.5 in Medical Physiology Examination in an Indian Medical School". سامانه مدیریت نشریات علمی, 14, 4, 1402, 311-317. doi: 10.30476/ijvlms.2023.98496.1221
Mondal, Himel, Dhanvijay, Anup Kumar, Juhi, Ayesha, Singh, Amita, Pinjar, Mohammed Jaffer, Kumari, Anita, Mittal, Swati, Kumari, Amita, Mondal, Shaikat. (1402). 'Assessment of the Capability of ChatGPT-3.5 in Medical Physiology Examination in an Indian Medical School', سامانه مدیریت نشریات علمی, 14(4), pp. 311-317. doi: 10.30476/ijvlms.2023.98496.1221
Mondal, Himel, Dhanvijay, Anup Kumar, Juhi, Ayesha, Singh, Amita, Pinjar, Mohammed Jaffer, Kumari, Anita, Mittal, Swati, Kumari, Amita, Mondal, Shaikat. Assessment of the Capability of ChatGPT-3.5 in Medical Physiology Examination in an Indian Medical School. سامانه مدیریت نشریات علمی, 1402; 14(4): 311-317. doi: 10.30476/ijvlms.2023.98496.1221
Assessment of the Capability of ChatGPT-3.5 in Medical Physiology Examination in an Indian Medical School
Interdisciplinary Journal of Virtual Learning in Medical Sciences
1Department of Physiology, All India Institute of Medical Sciences, Deoghar, Jharkhand, India
2Department of Physiology, Kalyan Singh Government Medical College Bulandshahr, Uttar Pradesh, India
3Department of Physiology, Raiganj Government Medical College and Hospital, West Bengal, India
چکیده
Background: There has been increasing interest in exploring the capabilities of artificial intelligence (AI) in various fields, including education. Medical education is an area where AI can potentially have a significant impact, especially in helping students answer their customized questions. In this study, we aimed to investigate the capability of ChatGPT, a conversational AI model in generating answers to medical physiology exam questions in an Indian medical school. Methods: This cross-sectional study was conducted in March 2023 in an Indian Medical School, Deoghar, Jharkhand, India. The first mid-semester physiology examination was taken as the reference examination. There were two long essays, five short essay questions (total mark 40), and 20 multiple-choice questions (MCQ) (total mark 10). We generated the response from ChatGPT (in March 13 version) for both essay and MCQ questions. The essay-type answer sheet was evaluated by five faculties, and the average was taken as the final score. The score of 125 students (all first-year medical students) in the examination was obtained from the departmental registery. The median score of the 125 students was compared with the score of ChatGPT using Mann-Whitney U test. Results: The median score of 125 students in essay-type questions was 20.5 (Q1-Q3: 18-23.5) which corresponds to a median percentage of 51.25% (Q1-Q3: 45-58.75) (P=0.147). The answer generated by ChatGPT scored 21.5 (Q1-Q3: 21.5-22), which corresponds to 53.75% (Q1-Q3: 53.75-55) (P=0.125). Hence, ChatGPT scored like that of the students (P=0.4) in essay-type questions. In MCQ-type questions, ChatGPT answered 19 correctly in 20 questions (score=9.5), and this was higher than the median score of students (6) (Q1-Q3: 5-6.5) (P<0.001). Conclusion: ChatGPT has the potential to generate answers to medical physiology examination questions. It has a higher capability to solve MCQ questions than essay-type ones. Although ChatGPT was able to provide answers that had the quality to pass the examination, the capability of generating high-quality answers for educational purposes is yet to be achieved. Hence, its usage in medical education for teaching and learning purposes is yet to be explored.
Civaner MM, Uncu Y, Bulut F, Chalil EG, Tatli A. Artificial intelligence in medical education: a cross-sectional needs assessment. BMC Med Educ. 2022;22:772. doi: 10.1186/s12909-022-03852-3. PubMed PMID: 36352431; PubMed Central PMCID: PMC9646274.
Tlili A, Shehata B, Adarkwah MA, Bozkurt A, Hickey DT, Huang R, et al. What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education. Smart Learn Environ. 2023;10:15. doi: 10.1186/s40561-023-00237-x.
Taecharungroj V. “What Can ChatGPT Do?” Analyzing Early Reactions to the Innovative AI Chatbot on Twitter. Big Data and Cognitive Computing. 2023; 7(1):35. doi: 10.3390/bdcc7010035
Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How Does ChatGPT Perform on the United States Medical Licensing Examination? The Implications of Large Language Models for Medical Education and Knowledge Assessment. JMIR Med Educ. 2023;9:e45312. doi: 10.2196/45312. PubMed PMID: 36753318; PubMed Central PMCID: PMC994776
Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLOS Digit Health 2023;2:e0000198. doi: 10.1371/journal.pdig.0000198. PubMed PMID: 36812645; PubMed Central PMCID: PMC9931
ChatGPT fails to clear the prestigious Civil Service Examination. IndiaAI. 2023 March 07. [Cited 2023 March 22].Available from: https://indiaai.gov.in/news/chatgpt-fails-to-clear-the-prestigious-civil-service-examination.
Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT - Reshaping medical education and clinical management. Pak J Med Sci. 2023;39:605-7. doi: 10.12669/pjms.39.2.7653. PubMed PMID: 36950398; PubMed Central PMCID: PMC1002569
Sinha RK, Deb Roy A, Kumar N, Mondal H. Applicability of ChatGPT in Assisting to Solve Higher Order Problems in Pathology. Cureus. 2023;15:e35237. doi: 10.7759/cureus.35237. PubMed PMID: 36968864; PubMed Central PMCID: PMC1003369
Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595. doi: 10.3389/frai.2023.1169595. PubMed PMID: 37215063; PubMed Central PMCID: PMC1019286
Newton PM. ChatGPT performance on MCQ-based exams. 2023; Preprint. doi:10.35542/osf.io/sytu3.
Huynh LM, Bonebrake BT, Schultis K, Quach A, Deibert CM. New Artificial Intelligence ChatGPT Performs Poorly on the 2022 Self-assessment Study Program for Urology. Urol Pract. 2023;10:409-15. doi: 10.1097/UPJ.0000000000000406. PubMed PMID: 3727637
Pepple DJ, Young LE, Carroll RG. A comparison of student performance in multiple-choice and long essay questions in the MBBS stage I physiology examination at the University of the West Indies (Mona Campus). Adv Physiol Educ. 2010;34:86-9. doi: 10.1152/advan.00087.2009. PubMed PMID: 20522902.
Arif TB, Munaf U, Ul-Haque I. The future of medical education and research: Is ChatGPT a blessing or blight in disguise? Med Educ Online. 2023;28:2181052. doi: 10.1080/10872981.2023.2181052. PubMed PMID: 36809073; PubMed Central PMCID: PMC994629
Das D, Kumar N, Longjam L, Sinha R, Deb Roy A, Mondal H, et al. Assessing the Capability of ChatGPT in Answering First- and Second-Order Knowledge Questions on Microbiology as per Competency-Based Medical Education Curriculum. Cureus 2023;15:e36034. doi: 10.7759/cureus.36034. PubMed PMID: 37056538; PubMed Central PMCID: PMC10086829.
Quintans-Júnior LJ, Gurgel RQ, Araújo AAS, Correia D, Martins-Filho PR. ChatGPT: the new panacea of the academic world. Rev Soc Bras Med Trop. 2023;56:e0060. doi: 10.1590/0037-8682-0060-2023. PubMed PMID: 36888781; PubMed Central PMCID: PMC999110
