Evaluation of three artificial intelligence chatbots for generating clinical hematology multiple choice questions for medical students.

Save

The integration of artificial intelligence (AI) into medical education has shown promise in streamlining content creation, yet the reliability and validity of AI-generated assessments remain critical concerns. This study evaluates three AI models-ChatGPT, Perplexity, and DeepSeek-in generating hematology multiple-choice questions (MCQs), focusing on their alignment with clinical guidelines, cognitive complexity, and expert acceptability, to determine their practical utility in medical education. To quantitatively evaluate and compare the performance of three AI models-ChatGPT, Perplexity, and DeepSeek-in generating multiple-choice questions (MCQs) relevant to hematology, with a focus on content validity, cognitive level alignment, and expert acceptance.

In this study, each AI model was prompted to generate 50 MCQs across five key hematology topics, following standardized instructions emphasizing guideline alignment and cognitive diversity. Three hematology experts, blinded to question source, independently rated all 150 MCQs on criteria including accuracy, clinical relevance, clarity, distractor plausibility, and overall quality, using a structured rubric. Scores were averaged per model, and questions were categorized by Bloom’s taxonomy level.

Acceptance was defined as a total score ≥ 15 out of 25. DeepSeek achieved the highest scores for accuracy (4.7 ± 0.4), clinical relevance (4.8 ± 0.3), and distractor plausibility (4.7 ± 0.4), with a perfect acceptance rate (100%) and no need for revision. Perplexity and ChatGPT also produced clinically relevant questions but required minor revisions (acceptance rates: 96% and 90%, respectively).

All models favored higher-order cognitive questions. Knowledge and comprehension questions were limited across all models. AI models, particularly DeepSeek, can efficiently generate high-quality, clinically relevant hematology MCQs suitable for medical education and assessment.

While DeepSeek demonstrated superior reliability and required minimal expert revision, all models underrepresented foundational knowledge questions and lacked autonomous image-based item generation. Hybrid human-AI workflows and targeted prompt engineering are recommended to optimize cognitive coverage and ensure educational rigor.

Supplementary Information

The online version contains supplementary material available at 10.1038/s41598-026-36839-x.

Background

Multiple-choice questions (MCQs) have become one of the most widely used tools for assessing knowledge in medical education1. Their popularity stems from their ability to cover a broad range of curriculum content, standardize assessments, and provide valuable statistical data for quality control of institutional question banks1. However, crafting high-quality MCQs remains a significant pedagogical challenge: poorly constructed questions can distort assessment outcomes, undermine validity, and negatively impact student performance2.

Effective MCQ development requires a rigorous methodology grounded in clearly defined educational objectives, identification of targeted learning levels, and adherence to established item-writing principles3,4. Objectives must be stated in observable and measurable terms to ensure each question assesses a specific, relevant knowledge area or skill essential for medical practice. This approach is supported by recent guidelines3,4.

Incorporating test blueprinting, systematic mapping of questions to curriculum content, ensure balanced coverage and enhances content validity5.

Artificial intelligence (AI) is increasingly transforming medical education by enhancing content creation, adaptive learning, and assessment design. AI-powered chatbots such as ChatGPT, Perplexity, and DeepSeek have demonstrated the ability to generate medical MCQs, which offers new opportunities to streamline quiz development, personalize learning, and boost student engagement in self-learning. AI enables the rapid creation of tailored and contextually relevant questions, allowing educators to efficiently expand their question banks and provide adaptive assessments that address individual learners’ knowledge gaps6–8.

Raising awareness among medical educators and students about AI tools is a foundational step for the successful development and implementation of AI-based solutions in medical education. Recent literature highlights that while the integration of AI offers significant benefits, such as personalized learning, streamlined content creation, and enhanced assessment, its adoption is often hindered by a lack of awareness, digital literacy, and formal training among both faculty and learners9–11.

To overcome these barriers, targeted educational strategies are essential. These include workshops, seminars, and interdisciplinary courses designed to introduce AI fundamentals, demonstrate real-world applications in healthcare and education, and encourage critical reflection on ethical considerations12. This study aims to address this gap by evaluating ChatGPT, Perplexity, and DeepSeek, three AI-driven chatbots, on their ability to generate hematology-focused MCQs for medical students by comparing the validity, difficulty level, and reasoning complexity of their outputs.

Methods

This study conducted over two months, from March to April 2025, was aiming to evaluate the performance of three AI models, ChatGPT, Perplexity, and DeepSeek, in generating pertinent MCQs relevant to hematology based on experts’ quantitative evaluation. It was composed of four main phases: (i) topic selection; (ii) prompt formulation; (iii) MQCs generation with each of the tested AI models; and (iv) experts’ evaluation (Fig. 1).

Fig. 1Flowchart of the study design.

Topic selection and prompt generation

Five hematology topics were selected because they are commonly assessed in written examinations for medical students and they require both foundational knowledge and clinical reasoning and based on their clinical importance and educational relevance: pancytopenia, anemia, thrombopenia, myeloproliferative syndromes, and lymphoproliferative syndromes. Each AI model was asked to generate 10 MCQs per topic, resulting in a total of 50 questions per model and 150 questions overall.

The prompt was the same given to each AI model. Instructions emphasized alignment with current hematology guidelines and the need to assess various cognitive levels, including recall, application, and clinical reasoning (Fig. 2).

Fig. 2The prompt formulation.

Expert review process13–15

Expert in hematology: a specialist with advanced knowledge and experienced in the relevant field. The selection criteria were based on academic qualification with medical degree and experience in hematology education, especially in MCQs creation and assessment. According to these criteria selected experts were professors and associate professors with a minimal of 4 years of experience in hematology education.

Review process (Appendix 1): Three hematology experts evaluate all 150 AI-generated MCQs while being blind to their source (which AI model generated each question). Experts rate each question using a standardized scoring rubric (e.g., 1–5 scale for validity, clarity, and relevance) to quantify their assessments with 2 Yes/No questions13–15. The review process consists of independent assessments based on the 8 following predefined criteria.

These criteria were selected based on established principles of medical assessment design to ensure a comprehensive evaluation of AI-generated MCQs. Specifically, they address scientific validity, educational relevance, clarity, and assessment quality, which are core components of high-quality MCQ construction. Alignment with curriculum objectives was assessed using Bloom’s taxonomy to determine the targeted cognitive level.Accuracy and Scientific Validity: Rate the factual correctness of content.

Higher scores indicate high accuracy.Clinical Relevance: Assess how well the question reflects real-world hematology scenarios and clinical decision-making.Alignment with Curriculum/Objectives: Evaluate the question’s relevance to the intended learning outcomes and categorize questions based on Bloom’s taxonomy (remembering, understanding, applying, analyzing).Clarity and Unambiguity: Judge the wording, structure, and overall clarity of the question stem and options.Plausibility of Distractors: Rate how well distractors resemble realistic errors or misconceptions in hematology.Overall Quality: An overall rating summarizing the question’s validity, relevance, and clarity.Difficulty level: Is the question appropriately challenging for the target audience?Discrimination power: Does the question differentiate between strong and weak students?

Reported difficulty level and discrimination power were based solely on expert opinion using a binary (Yes/No) rubric and were not derived from formal psychometric analysis or student performance data.

Scoring rubric

Step 1: Each expert independently reviews every MCQ without influence from others.Step 2: For each MCQ, the expert assigns a score from 1 (poor) to 5 (excellent) on each evaluation criterion (e.g., accuracy, clinical relevance, clarity, distractor plausibility, overall quality).Step 3: The scores across all criteria are summed to calculate a total score for each question.Step 4: Based on the total score, decisions are made regarding the question’s suitability.High scores20–25: Accept the question as is or perform minimal revisions.Moderate scores15–19: Consider revising the question to improve quality.Low scores (below 15): Discard the question or undertake significant revision

This structured rubric ensures that expert evaluations are transparent, consistent, and efficient, facilitating objective decision-making in the review and selection of high-quality MCQs13–15.

Data analysis

Scores for validity, relevance, difficulty and all the criteria mentioned above were averaged per AI model and expert.Qualitative feedback from experts regarding common strengths and weaknesses of each AI model’s questions was also collected for thematic analysis.

Ethical considerations and statement

As this study involved expert review of AI-generated educational content without human subjects, formal ethical approval was not required, and an informed consent was obtained from all the experts. Confidentiality of expert identities and anonymity of question sources were maintained throughout. The author has no conflict of interest to declare as the three tested chatbots are available for free.

All methods were carried out in accordance with relevant guidelines and regulations. This study did not involve human participants, human tissue samples, or identifiable personal data. Therefore, ethical approval from an institutional or licensing committee and informed consent were not required.

Results

A total of 150 MCQs was generated in less than 30 s per AI chatbot.

AI performance in MCQs generation

Regarding accuracy and scientific validity, DeepSeek received the highest rate (4.5 ± 0.6) indicating a strong factual correctness and alignment with hematologic guidelines. The three tested AI models were able to generate clinically relevant questions with strong alignment to practice with a highest score for DeepSeek (4.8 ± 0.3) (Table 1).

For plausibility of distractors, ChatGPT received the lowest score (4.1 ± 0.8), indicating that its distractors might be less convincing or too easy to eliminate. DeepSeek had the highest overall quality score (4.5 ± 0.4) confirming its strength in generating coherent and well-structured useful MCQs in the study context (Table 1).

Table 1Average scores for evaluation criteria by AI models.Evaluation CriterionChatGPT(Average Score ± SD)DeepSeek (Average Score ± SD)Perplexity (Average Score ± SD)pAccuracy & Scientific Validity4.5 ± 0.64.7 ± 0.44.6 ± 0.60.03Clinical Relevance4.5 ± 0.54.8 ± 0.34.6 ± 0.50.01Alignment with Curriculum4.3 ± 0.84.1 ± 0.64.4 ± 0.60.5Clarity & Unambiguity4.4 ± 0.84.6 ± 0.44.6 ± 0.50.25Plausibility of Distractors4.1 ± 0.84.7 ± 0.44.3 ± 0.70.005Overall Quality4.1 ± 0.74.5 ± 0.44.3 ± 0.70.02Difficulty Level(Yes %)88%94%88%0.2Discrimination Power (Yes %)74%86%74%0.04The Discrimination Power and the Difficulty level are presented as the percentage of questions deemed to have adequate discrimination by the experts (based on their Yes/No responses). SD Standard Deviation, AI Artificial Intelligence.

Among the generated MCQ, are shown in Table 2, the percentage of those with two or more correct answers for each of the tested AI chatbot.

Table 2Percentage of multiple correct answers for each MCQ.Correct answersChatGPTN/50 (%)DeepSeekN/50 (%)PerplexityN/50 (%)≥ 2 answers9 (18%)1 (2%)15 (30%)MCQ Multi-choice question.

Bloom’s taxonomy and cognitive level alignment

Using a dichotomized classification of Bloom’s Taxonomy, MCQs were grouped into lower-order cognitive levels (Knowledge/Comprehension) and higher-order cognitive levels (Application, Analysis, Synthesis, and Evaluation). Across All three AI models, the generated MCQs belonged to the higher-order cognitive levels, accounting for 78% of items generated by ChatGPT, 92% by DeepSeek, and 80% by Perplexity. In contrast, Knowledge/Comprehension-level questions represented a smaller proportion, comprising 22% of ChatGPT questions, 8% of DeepSeek questions, and 20% of Perplexity questions.

Although DeepSeek produced the highest proportion of higher-order cognitive questions and the lowest proportion of Knowledge/Comprehension items, the differences between the three models did not reach statistical significance (p = 0.08 for higher-order levels; p = 0.06 for Knowledge/Comprehension).

These data are detailed in Table 3.

Table 3Distribution of MCQs across bloom’s taxonomy levels.Bloom’sTaxonomy LevelsChatGPTN/50 (%)DeepSeekN/50 (%)PerplexityN/50 (%)pKnowledge/Comprehension11 (22%)4 (8%)10 (20%)0.06Application \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\quad \Downarrow$$\end{document}Synthesis 39 (78%)46 (92%)40 (80%)0.08MCQ Multi-choice questions.

Expert validation and acceptance rates

DeepSeek demonstrated a perfect acceptance rate (100%), meaning all MCQs met the required quality threshold with no revision needed. Both ChatGPT and Perplexity had some questions that needed revision (Table 4).

Table 4Percentage of MCQs meeting acceptance threshold (Total Score ≥ 15).AI ModelNumber of Accepted MCQsN/50 (%)Number of Revised MCQsN/50 (%)ChatGPT45 (90%)5 (10%)DeepSeek50 (100%)0 (0%)Perplexity48 (96%)2 (4%)AI Artificial intelligence, MCQs Multiple-choice questions.

Here are some examples of questions that require minor revisions:

For ChatGPT

For perplexity

Discussion

The aim of our study was to evaluate the ability of AI models (ChatGPT, Perplexity and DeepSeek) to generate hematology MCQs and to compare the models’ performance based on predefined criteria such as validity, difficulty and reasoning ability required to answer them.

AI performance in MCQs generation

Our study demonstrates that advanced AI models such as DeepSeek, Perplexity, and ChatGPT can efficiently generate high-quality, clinically relevant hematology MCQs, and among these three models, DeepSeek demonstrated higher accuracy and distractor plausibility making it the most promising model for generating high-quality MCQs in this study context, while clearly emphasizing that formal psychometric validation with student data is required to confirm discrimination power and suitability for high-stakes assessment. These results concur with recent studies that prove that AI-automated MCQs can ideally match or even exceed human-designed questions in terms of validity, comprehension, and efficacy, reducing time and effort devoted to question preparation14. However, like previous studies, the AI-generated test questions in this study also tended to be less difficult and mainly focused on lower- to mid-level cognitive abilities, potentially that they may not be particularly good at assessing higher-level clinical reasoning unless specifically prompted14,16.

Despite high rates of acceptability, expert rigorous examination is still needed since occasionally the AI models can be inclined to produce factual inaccuracies or unrealistic distractors, something also highlighted in the wider literature14,15.

Of note, when the study began, none of the chatbots created or introduced spontaneously an image-based question, which can be considered as a study limitation. Nevertheless, when we directly demanded to create, both DeepSeek and Perplexity expressed directly that they cannot provide scientific image-based questions, only ChatGPT essayed to have more time and asked a lot of specific questions to generate finally an image-based question of bad quality.

Moreover, the fact that current models cannot generate image-based or multimodal questions on their own points to a limitation for specialties like hematology, where visual interpretation is paramount. Overall, while AI can potentially enable efficient MCQ generation and help in standardized assessment development, optimal results require hybrid workflows that combine AI efficiency with human vetting to ensure educational rigor and clinical relevance14,15.

Bloom’s taxonomy and cognitive level alignment

The breakdown of questions by Bloom’s taxonomy levels provides important insight into how AI models address cognitive complexity in medical education. Our findings demonstrate that all AI models predominantly generated MCQs targeting high-order cognitive levels. These questions accounted for 78% of items produced by ChatGPT, 92% by DeepSeek, and 80% by Perplexity, whereas Knowledge/Comprehension-level questions represented a smaller proportion (22%, 8%, and 20%, respectively).

This distribution suggests that contemporary large language models are inherently oriented toward generating clinically contextualized and reasoning-based questions rather than items focused on factual recall. This tendency aligns with the underlying architecture of transformer-based large language models, which are trained on large corpora emphasizing semantic relationships, pattern recognition, and contextual inference rather than rote memorization. As a result, AI-generated questions often embed clinical scenarios and decision-making elements that naturally correspond to higher-order cognitive processes17,18.

Studies evaluating ChatGPT and other AI chatbots in medical education found that, although these tools can efficiently generate questions aligned with higher levels of Bloom’s taxonomy, they tend to produce fewer items that assess basic recall and understanding, mirroring the pattern seen in this study19. This could be considered from another point of view as a strength of these AI tools as the task of conceiving MCQ with higher cognitive level is more complex compared to generating remembering-based questions.

The observed imbalance highlights the urgent need for targeted and precise prompt engineering to explicitly instruct AI models to generate questions across the entire range of cognitive levels, ensuring comprehensive assessment coverage. Our research guides this process indicating that while AI excels at producing higher-order questions efficiently, human oversight remains critical to ensure that assessments adequately address foundational levels of cognitive skills and account for contextual nuances that are unique to medical education14,20,21. Through this precious cooperation, we contribute to generating a well-organized and pedagogically sound responses that enhance the accuracy and utility of AI-generated assessments in medical training22,23.

Expert validation and acceptance rates

The expert validation results indicate significant MCQ acceptance rate differences between AI models, in which DeepSeek obtained 100% (no need for revision), in contrast to ChatGPT (90%) and Perplexity (96%). The findings agree with available research data that indicates AI-generated MCQs need to be reviewed by humans but can attain high validity with models14,22.

DeepSeek’s performance-namely increased distractor plausibility (4.7/5) and clinical utility (4.8/5) likely reflects optimized training on medical datasets, overcoming general AI vulnerabilities like factual inaccuracies and superficial reasoning seen in studies on general-purpose large language models like ChatGPT14,16.

While Perplexity’s 96% rate of acceptance suggests reliability for complex-reasoning multi-response tasks (e.g., 30% multi-answer questions), ChatGPT’s 10% rate of revision points to persistent problems in distractor design, as corroborated by findings that point to AI-distractors as being implausible unless strongly prompted14. This highlights the importance of encouraging multi-answer questions compared to simple-answer questions. This approach better assesses higher order clinical reasoning and aligns more with AI’s capability to synthetize correct information which will represent a pedagogically sound strategy to enhance assessment depth and validity in medical education24.

Moreover, AI-generated MCQs may be used, for medical students, as formative assessment tools, practice resources and support for clinical reasoning development, particularly when questions are carefully curated and reviewed by educators25.

Study strengths and limitations

This study presented a quantitative structured evaluation of AI-generated MCQs in hematology, offering valuable insights into their quality, relevance, and cognitive level alignment. To the best of our knowledge, this is the first study in the field of hematology.

It bears mentioning that the current study has strengths and limitations. The questions generated were evaluated by 3 experts in the field providing reliable data compared to other studies. For the study limitations, the potential influence of expert characteristics (training background, geographic context, type of clinical practice) and the reliance on expert judgment rather than psychometric validation.

The absence of live testing with medical students, which limits conclusions about real-world performance, discrimination and validity were not empirically measured, rather than being absent.

Conclusion

AI has increasingly become a valuable tool in medical education, revolutionizing various aspects, including content creation, adaptive learning, and assessment design. AI-powered chatbots such as ChatGPT, Perplexity, and DeepSeek have demonstrated their ability to generate medical MCQs, offering new opportunities to streamline quiz development, personalize learning, and enhance student engagement.

MCQs are widely used in medical education for their ability to efficiently assess knowledge recall, clinical reasoning, and decision-making skills. Traditionally, MCQs are designed by expert educators, a process that requires significant time, expertise, and validation to ensure their reliability, validity, and appropriate difficulty level. AI-generated MCQs provide a promising alternative by automating question creation, which generates initial MCQs drafts, including stems, distractors, and correct answers with the intention of reducing faculty workload26.

However, their effectiveness in producing clinically relevant and pedagogically sound assessments remains uncertain.

This study aims to address this gap by systematically evaluating ChatGPT, Perplexity, and DeepSeek, three AI-driven chatbots, on their ability to generate hematology-focused MCQs for medical students. By comparing the validity, difficulty level, and reasoning complexity of their outputs, this research seeks to determine whether AI-generated assessments can match or complement traditionally crafted questions.

The integration of AI into medical education, as demonstrated in our examination of MCQ generation using ChatGPT, is both promising and complex. The key to harnessing AI’s capabilities lies in meticulous prompt engineering and a thorough review by individuals with deep medical content knowledge. With continued research, ongoing refinement, and collaboration between disciplines, the future holds a synergy where educators are better equipped, and students benefit from a more comprehensive and detailed learning experience15.

This study demonstrates that AI models like DeepSeek, Perplexity, and ChatGPT can generate clinically relevant MCQs with varying strengths: DeepSeek demonstrates higher levels in clinical precision and distractor plausibility, making it the most promising model for generating high-quality MCQs in this study context. Perplexity produces complex, multi-answer questions suited for case-based reasoning; and ChatGPT offers rapid drafting but requires human oversight for distractor refinement. While all models prioritize higher-order cognitive skills (application/analysis), they often underperform in foundational knowledge and synthesis-level questions, necessitating targeted prompting.

Expert validation confirms domain-specific AI’s potential to reduce faculty workload, though hybrid human-AI workflows remain critical to address limitations in multimodal content generation, rare disease coverage, and evolving guideline adherence. By strategically leveraging each model’s strengths-DeepSeek’s reliability, Perplexity’s complexity, and ChatGPT’s efficiency-educators can enhance MCQ development while maintaining academic rigor, provided continuous validation and ethical safeguards are implemented.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Related articles

PLOS

Traditional knowledge of childhood illnesses among the Sidama nation, clinical presentation and hematological findings

Background: Sidama possesses extensive indigenous knowledge regarding infant illnesses described as “feyancho”. Feyancho has a widespread recognition among Sidama people, though poorly documented in scientific literature. This study aimed to document the indigenous understanding of this infantile disease and examine associated clinical and biochemical profiles in the Sidama National Regional State, Ethiopia. Methods: A household-based cross-sectional study was conducted in nine districts selecte...

PubMed

The hematology workforce in the United States: past, present, and strategies for the future.

Abstract From 2017 to 2021, the American Society of Hematology (ASH) conducted the largest study to date of the hematology workforce in the United States. The study highlighted a deficit in the classical hematology (CH) workforce. Despite the very high CH clinical volumes encountered by practicing hematologists and medical oncologists, <5% of all hematology and medical oncology fellows reported having a career focus in CH. The reason underlying this deficit was a lack of mentorship in CH compar...