Introduction

During their medical education, students undergo training and internships in hospitals alongside theoretical courses. This combination allows them to acquire essential skills for their future careers as doctors. In hospitals, students observe how doctors interact with patients and perform various medical procedures. They also engage with patients, doctors, and hospital staff. Under the supervision of experienced doctors, students practice medical tasks, including diagnosing patients and developing treatment plans. Training and internships typically require the physical presence of students, patients, and doctors in the same hospital environment. However, numerous studies have demonstrated that virtual patients are highly effective in promoting clinical reasoning skills and significantly enhance learning outcomes [1,2,3,4,5]. Additionally, other research has shown that virtual cases are well-received by medical students [6, 7]. Several online platforms offer solutions for medical students and healthcare professionals, including virtual cases. For instance, platform [8] integrates five scenarios designed to train medical students and health professionals in treating COVID-19 patients. Platforms [9, 10] each incorporate many scenarios to improve the critical reasoning skills of medical students and healthcare professionals. Platform [11] provides a simulation experience, where a professor remotely shares the screen displaying a streaming video of a patient and the patient’s vital signs monitor. Students use this setup to address key medical objectives of the case. Figure 1 illustrates a flow diagram of the systematic search for finding the relevant papers and learning platforms reported above, while Table 1 makes a synthesis of the type of publications, the reported metrics and main results in those papers.

Fig. 1
figure 1

Flow diagram for the systematic process used to find the relevant papers and learning platforms

Full size image
Table 1 Type of publications, used metrics, and main results reported in the selected papers (VP: Virtual Patient, MCQ: multiple-choice questions)
Full size table

This article presents the results of using a web-based platform for learning clinical reasoning in cardiovascular diseases, aimed at enhancing the abilities of medical students and interns in this area. It also briefly describes the platform’s functionalities for both students and professors, and provides an overview of their satisfaction with the platform. The Virtual Patient platform allows medical students to virtually replicate, through clinically valid real-life scenarios, the processes of diagnosing patients and recommending treatments. Students interact with an avatar of the patient using natural language, either through speech or text-based interactions. The platform includes multiple scenarios aimed at training medical students in acute and chronic heart diseases. The platform detailed architecture and functionalities are reported in [12] and two screen shots of the interface are presented in Fig. 2.

The creation of the Virtual Patient platform began during the COVID-19 pandemic to compensate for the lack of physical interaction with real patients. It was further developed and continued to be used after the pandemic, in a partnership between the National University of Science and Technology POLITEHNICA Bucharest and the University of Medicine and Pharmacy “Carol Davila” Bucharest. The platform has proven to be an invaluable tool for complementing traditional training, as demonstrated by the results of our study. The platform addresses the open challenges to the current state of the art which are:

  • collection of medical data,

  • variation of medical cases that are presented to the students,

  • presence of evaluation in different stages of diagnostic, recommendations and prognostic,

  • user-friendly interface and different modalities,

  • extensibility to other medical cases.

The study reported here was conducted over a two-year period and involved 210 users: 178 medical students, 7 professors, and 25 engineering students, the last group called to evaluate the technical aspects of the platform. The evaluation by medical students was performed in four rounds, each round corresponding to successive enhancements of the platform functionalities and addition of new cases, with a total number of 1,098 evaluation sessions.

Fig. 2
figure 2

The Virtual Patient platform - student view (home page on the left, investigations on the right)

Full size image

Methods

Medical cases

The Virtual Patient platform allows students to engage with all aspects of clinical activities involved in diagnosing a patient and recommending appropriate treatments. It also includes evaluations by professors (doctors) of the students’ performance and provides the possibility to provide tutor’s feedback either during the interaction of the student with the virtual patient or afterward.

When interacting with the platform, students are presented with a medical case that includes the following stages sequentially:

  • Patient Description. A brief overview of the patient along with one or more images illustrating the general appearance of the virtual patient (such as posture, position, clothing, facial expressions).

  • Anamnesis. The student asks the patient specific questions about their medical history and receives corresponding answers.

  • Objective Examination. This varies depending on the patient’s symptoms. The student must observe and note external signs (e.g., weight, scars, spinal deformities), auscultate the heart and respiratory system, and possibly investigate other systems and organs (e.g., digestive system) as needed.

  • Presumptive Diagnosis. The student provides an initial diagnosis based on the information collected and analyzed up to that point, before requesting any further health investigations.

  • Health Investigations. The student requests relevant health investigations specific to the patient’s symptoms, which may include blood analysis, electrocardiogram (ECG), radiography, ultrasound, computed tomography (CT), or angiography.

  • Treatment Plan. Based on the gathered information, the student recommends the main elements of an appropriate treatment plan, which can include dietetic-hygiene measures, pharmacological treatments, interventional procedures, and/or surgical treatments.

  • Complications. The student suggests possible complications that may arise during the illness’s progression and treatment.

  • Differential and Final Diagnoses. The student provides the main points of the differential diagnosis (possible diseases the patient might or might not have, with justifications) and the final diagnosis for the case.

  • Prognosis. The student predicts the likely trajectory/evolution of the patient’s illness.

The platform contains two types of medical cases (patient scenarios): simple scenarios for patients with one pathology and complex scenarios for patients with several related pathologies. These scenarios can be either automatically generated by an algorithm that ensures the validity of combined data or entirely constructed by a professor, which may not necessarily follow the same logic as the automated scenarios (e.g., not requesting a specific medical investigation based on patient symptoms).

Additionally, the platform includes a library of extremely rare medical cases collected from literature, which may never be encountered by a doctor under normal circumstances. This resource allows medical students to consult and analyze these rare cases for educational purposes.

Data collection

The data integrated into the Virtual Patient platform was collected, annotated, and anonymized by a team of health professionals from the University of Medicine and Pharmacy “Carol Davila” Bucharest.

Patient’s medical history and related information: for each pathology the team elaborated a document in which they introduced various medical information about patients (including medical history information). The information was extracted from the data that were available in the Bucharest Clinical Emergency Hospital (SCUB) but also from analyzing the specialized literature.

The anamnesis section: For each pathology, the team developed a document in which they listed the clinically relevant questions for this section along with their categories (such as: must {golden-set} or optional). For each question, doctors listed the possible answers of the patient along with along with an associated probability. The answers and their probabilities are correlated to the severity of the patient illness and respect the reported probability of their occurrence in the specialized literature and the European guidelines.

The objective examination: For each pathology, the team collected anterior and posterior pulmonary auscultations as well as cardiac auscultation. The collected auscultations are stored in mp3 formats and belong to patients that were hospitalized in the Cardiology Department of SCUB.

The medical investigations: For each pathology, the team collected different medical investigations. Similar to the collected auscultations, the collected medical investigations belong to patients that were hospitalized in SCUB. The investigations are stored in different formats such as xls format (tabular format stored in an excel file) for blood tests; pdf or jpeg formats for ECG; jpeg, png or tif formats for CT and chest X-rays; jpg, png, tif, mp4 or avi formats for ultrasounds; and mp4 or avi formats for angiographies and computed tomography angiography (CTA). The collected date-set is illustrated in Table 2.

Before uploading the data to the platform, the team anonymized the entire data-set. The anonymization made impossible to trace the data to their original patients: any information that can link the data to its original patient was removed. After uploading the data, the team annotated the data appropriately.

Table 2 The total volume of collected data-set. Complex case 1: acute inferior myocardial infarction + severe mitral regurgitation + heart failure; Complex case 2: heart failure + mitral stenosis
Full size table

Medical case generation

The medical data uploaded to the Virtual Patient platform are clinically valid. For building the scenarios, each element of a medical case is labeled by the disease it represents and the severity of its manifestation. Additionally, each element is tagged with supplementary useful information (such as specific associations). This labeling ensures that the generated scenarios are consistent and clinically valid.

The generation engine of the Virtual Patient assembles different elements of a scenario from the uploaded data based on clinically allowed associations between their labels, adhering to any specified restrictions. These restrictions are exclusively defined by the professors. The generated scenarios are ensured to be clinically valid and similar to real-life cases, with varying degrees of difficulty and peculiarities.

The generation engine operates in three modes:

  • Automated Mode. The engine takes the name of the disease and its severity as input and generates a scenario by associating the labels of different elements as described.

  • Semi-Automated Mode. In addition to the disease name and severity, the professor can input specific elements to be integrated into the scenario by uploading new data or selecting from the data existing on the platform. The engine then completes the scenario by associating labels for the missing elements.

  • Manual Mode. The professor provides all the elements of the scenario, either by uploading new data or selecting from the existing data. The engine integrates all the provided elements to generate the scenario.

The platform also includes a component to generate synthetic data based on generative artificial intelligence techniques. This component is still in an early stage of implementation. Currently, it can generate results for blood tests in tabular format.

For blood tests, an accepted value range for each blood component is defined for each pathology, considering the severity of the illness, as well as the sex and age of the patient. Some of the results are illustrated in Table 3. Since the synthetic data generation component is at an early stage, the generated data must be validated by a professor before being used in a scenario.

Table 3 Some synthetic generated data for blood tests results - Pulmonary Thromboembolism
Full size table

Participants

The platform has been evaluated at various stages of its implementation by a diverse group of users. A total of 178 students from the University of Medicine and Pharmacy “Carol Davila” Bucharest participated in the evaluation. These students, spanning different academic years, included 63 females and 115 males, aged between 18 and 29 years.

Additionally, 7 professors from the same university, comprising 3 females and 4 males aged between 30 and 60 years, evaluated the platform in its current stage.

In total, 185 users from the University of Medicine and Pharmacy “Carol Davila” Bucharest evaluated the platform, including 66 females and 119 males.

Beyond these users, a group of 25 engineering students from the National University of Science and Technology POLITEHNICA also evaluated the platform’s interface and functionalities. This group consisted of 11 females and 14 males, aged between 18 and 24 years.

Thus, a total of 210 users have evaluated the platform, comprising 77 females and 133 males, as illustrated in Table 4.

For each evaluation round, the questionnaire was slightly adjusted with questions related to features and medical cases included in the current version of the platform. However, the core of the questionnaire remained the same and it was focused on user satisfaction and opinions regarding the platform’s usefulness, features, functionalities, and interaction modalities. After each evaluation of the Virtual Patient, participants filled in the questionnaire. At the end of the questionnaire, users were asked to provide their suggestions for improving the platform.

Table 4 The composition of the groups that have evaluated the platform
Full size table

Platform development

Functionalities

The Virtual Patient platform integrates various functionalities. The main functionality of the platform is to enable medical students to virtually reproduce, in clinically valid scenarios, the diagnostic process and treatment recommendation. It allows students to use natural language to interact with virtual patients through speech and text-based interactions, in order to reproduce the interaction between students and their patient that usually take place in hospitals. The second functionality of the platform is allowing professors to visualize the performance of their students, to provide feedbacks for students and to score their performances. These actions can be done by professors in real time during the student’s session or after the end of the sessions whenever they prefer. Allowing professors to upload, manage and annotate health data is the third functionality of the platform. The fourth functionality of the platform is the automatic computer-based generated feedback that is provided after each session. It includes an automated awarded score and also takes into account the feedback provided by the professor, if any. The professor has the possibility to modify the score that was awarded by the platform as well as to modify the generated feedback. The user classification system is the fifth functionality of the platform. The sixth functionality of the platform is the generation of new clinically valid scenarios. The seventh and last functionality of the platform is the possibility to synthetically generate medical data.

The platform integrates a dialog module that allows interactions in natural language between the student and the virtual patient. The student asks questions to the virtual patient using speech or/and text-based commands and he/she will read the answers of the virtual patient on the graphical user interface (GUI) and will hear them through the phonetic output of the system (unless this option is disabled).

The platform supports English and Romanian languages with the possibility to extend the support to new languages. It integrates different features that can be customized by the user such as:

  • Enabling or disabling the apparition of meaningful icons on the main buttons, before the text associated to the button. This feature makes the identification of buttons faster for the user and enable the possibility to choose the type of menu. In addition, the user can customize the icon of each button.

  • Menu type: extended in which the buttons of the menu are displayed with the icon and text; or collapsed in which the buttons of the menu are displayed only with the icon. This feature reduces the space needed by the menu (by hiding the text of the buttons) and by consequence give more space to the other elements of the GUI.

  • Language of the platform: the user can choose the language to be used in the platform.

  • Speech voices: the user can choose the voice that should be used during the speech synthesis.

  • Sounds: the user can enable and disable the sounds.

  • Font and colours: the user can change the fonts and the dimensions of the fonts that are used across the platform, as well as the colours of the buttons.

Type of users

To manage access to the various functionalities of the platform, there are three types of users: student, professor, and administrator.

Administrator

Administrators have the highest level of permissions. They can overwrite information introduced by other users, add new data, and manage (edit and delete) existing data on the platform. Administrators can associate students or groups of students with a professor, create new scenarios and entries for new types of data (such as new diseases or medical investigations). They can also translate platform content into new languages, visualize and manage (edit, delete, and create) user accounts.

Professor

This user type is assigned to faculty members of medical schools. Professors can create new scenarios and assign them to specific students or groups of students. They can view students’ answers, evaluate their performance, and provide feedback during or after the sessions. Professors can modify performance scores automatically assigned by the platform after a session ends. They can add new health data, annotate, and manage (edit and delete) existing health data. Professors also validate scenarios and data generated by the platform’s generation module.

Student

This user type is assigned to medical students. At the beginning of each session, students should observe the virtual patient’s position (e.g., sitting down, Fowler’s position, supine position) as it provides useful diagnostic clues. During the anamnesis phase, students interact with the virtual patient through speech and text-based interactions to obtain answers. They can access the patient’s medical history, perform objective examinations, request and analyze medical investigations, and provide presumptive, differential, and final diagnoses. In the treatment section, students recommend treatment plans, describe potential complications, and provide prognoses. Additionally, students can access their history to view previous performances and consult feedback received.

This structure ensures that each user type has access to the appropriate functionalities necessary for their role.

Computing learning scores

The maximum score that can be scored by a student during each session is 100 points. For each section that composes the process of diagnosing and treatment recommending, the student can score a maximum of 11 points with the exception of three sections: treatment section that has a maximum of 12 points as well as home and antecedents sections that are not scored since the student’s progress in these sections will be reflected across the answers that he/she will provide during the session. Apart from spending a reasonable amount of time in each section, the scoring of each section is computed based on the following aspects:

  • anamnesis section: the student must ask the “golden-set” questions for the current case and use a logic flow when posing the questions.

  • objective exam section: the student must perform an objective exam; therefore he/she should not miss any point of auscultation and to use a logic flow when choosing which point to auscultate.

  • presumptive diagnosis: the student must provide a correct presumptive diagnosis.

  • medical investigations: the student must request all the “golden-set” medical investigations that are relevant to the current case, use a logic flow when requesting, and provide a correct interpretation for each investigation.

  • differential diagnosis: the student must provide a correct and complete differential diagnosis with correct justification.

  • final diagnosis: the student must provide a correct final diagnosis with correct justification.

  • treatment: the student must provide an accurate, clear and complete treatment plan.

  • complications: the student must describe correctly all the complication that may appears during the evolution of the illness and during the treatment with correct justification.

  • prognosis: the student must provide a correct prediction of the trajectory/evolution of the patient’s illness with correct justification.

Results

Students’ evaluation of the platform

The first evaluation of the platform took place in the phase of implementation in which the platform integrated simple scenarios associated only with the pulmonary thromboembolism disease. In this evaluation, two groups of 98 students evaluated the platform. The first group was composed of 88 students and used Romanian language, while the second group was composed of 10 students and used English language. Each student accessed the platform twice. A total of 196 evaluation sessions were performed.

The second evaluation of the platform took place in the phase of implementation in which the platform integrated simple scenarios associated with one of four acute and chronic heart diseases. In this evaluation, a group of 28 students evaluated the platform using Romanian language. Each student accessed the platform 12 times (three times for each of the four diseases). In total 336 evaluation sessions were performed (84 for each disease).

The third evaluation of the platform took place in the phase of implementation in which the platform integrated a support module. In this evaluation, a group of 14 students evaluated the platform. Each student accessed the platform 16 times (four times for each of the four diseases). In total 224 evaluation sessions were performed (56 for each disease: 28 for each language).

The fourth evaluation of the platform took place in the phase of implementation in which the platform integrated simple scenarios associated with one of six acute and chronic heart diseases or with two complex cases. In this evaluation, a group of 38 students evaluated the platform using Romanian language. Each student accessed the platform 9 times (one times for each of the five diseases and twice for each complex scenario). In total 342 evaluation sessions were performed (38 for each disease and 76 for each complex case). A part of the results is illustrated in Tables 5 and 6.

Table 5 Platform usefulness evaluation results and user recommendations
Full size table

The majority of students found the platform very useful, specifically 147 students (82.58%), while 22 students found the platform useful (12.36%). 9 students were neutral about the usefulness of the platform (5.06%). The percentages of the usefulness evaluation results provided by medical students are illustrated in Fig. 3.

Table 6 User satisfaction in relation with the scenarios that are associated to a disease/complex case. Complex case 1: acute inferior myocardial infarction + severe mitral regurgitation + heart failure; Complex case 2: heart failure + mitral stenosis
Full size table

The majority of students recommended the extension of the scenarios that are integrated in the platform to cover other diseases and the extension of the dialogue module (which they all appreciated much in both languages) to support more questions in the anamnesis section, specifically 177 students (99.44%) and 165 students (92.70%) respectively. 12 students recommended the integration of other features in the platform such as enabling animation (making the virtual patient animated in the scenarios), integration of tools that allow a better visualization of the medical investigation. The percentages of the main recommendations provided by medical students are illustrated in Fig. 4.

Regarding the satisfaction of the students with the scenarios that are associated with:

  • pulmonary thromboembolism: the majority of students are highly satisfied, specifically 139 students (78.09%), while 23 students are satisfied (12.92%), 10 students are neither satisfied nor dissatisfied (5.62%) and 6 students are dissatisfied (3.37%).

  • acute coronary syndromes: the majority of students are highly satisfied, specifically 56 students (70.00%), while 16 students are satisfied (20.00%), 6 students are neither satisfied nor dissatisfied (7.50%) and 2 students are dissatisfied (2.50%).

  • mitral regurgitation: the majority of students are highly satisfied, specifically 58 students (72.50%), while 18 students are satisfied (22.50%) and 4 students are neither satisfied nor dissatisfied (5%).

  • severe aortic stenosis: almost half of students are highly satisfied, specifically 43 students (53.75%), while 24 students are satisfied (30.00%), 9 students are neither satisfied nor dissatisfied (11.25%) and 4 students are dissatisfied (5.00%).

  • cardiac insufficiency: the majority of students are highly satisfied, specifically 24 students (63.16%), while 7 students are satisfied (18.42%), 4 students are neither satisfied nor dissatisfied (10.53%) and 3 students are dissatisfied (7.89%).

  • complex case 1: the majority of users are highly satisfied, specifically 23 students (60.53%), while 9 students are satisfied (23.68%), 2 students are neither satisfied nor dissatisfied (5.26%) and 4 students are dissatisfied (10.53%).

  • complex case 2: half of students are highly satisfied, specifically 19 students (50.00%), while 8 students are satisfied (21.05%), 4 students are neither satisfied nor dissatisfied (10.53%), 6 students are dissatisfied (15.79%) and 1 student is highly dissatisfied (2.63%).

Fig. 3
figure 3

Percentages for the usefulness evaluation results by medical students

Full size image
Fig. 4
figure 4

Percentages for the main recommendations by medical students

Full size image

In addition to this evaluation, the group of students from the National University of Science and Technology POLITEHNICA evaluated the interface and some functionalities of the platform by accessing the platform twice. Once using the Google Chrome browser from a computer that has a Windows 11 operating system and integrates a microphone; and once using the Google Chrome browser from a Samsung Galaxy Active Pro tablet (10.1-inch screen size) that has an Android 10 operating system and a built-in microphone. The participant in this evaluation appreciated much the design of the platform and its responsivity on the computer screen and on tablets. They found the possibility to collapse the menu very useful, especially on tablets. The participant found the possibilities to disable the sounds and to change the size of the fonts as well as the language of the platform useful. They were neutral about the possibility to change the voice that should be used during the speech synthesis and the colours of the buttons. The participant found the possibility to change the fonts that are used on the platform useless.

Professors’ evaluation of the platform

The professors evaluated various features of the platform. All of them greatly appreciated the ability to generate new scenarios and the platform flexibility which allows the three working modes. Additionally, they valued the capability to provide feedback and score student performance in real-time or after the session. Another highly appreciated feature was the ability to manage and annotate the data that is uploaded on the platform or to add new data, all directly from the interface of the platform.

Regarding the customizable features of the user interface, less than half of the professors expressed appreciation, specifically 3 professors (42.86%). Meanwhile, 2 professors were neutral (28.57%), and 2 professors were dissatisfied with this aspect.

Professors also evaluated the scenarios generated by the scenario generation engine. They were all very satisfied with the results for simple scenarios. However, for complex scenarios, 2 professors were satisfied (28.57%), 3 were neutral (42.86%), and 2 were dissatisfied (28.57%).

In addition, professors assessed the computer-generated feedback and automated scoring provided by the platform. The majority were very satisfied with these features, specifically 6 professors (85.71%), while one professor was satisfied but suggested that the platform should offer more explainability for the awarded scores.

As the synthetic data generation component is still in its early stages of implementation, it has not yet been evaluated by the professors.

All the different versions of the questionnaires used in our study was developed specifically for the evaluation of the Virtual Patient platform and the aggregated results reported in this paper have not previously been published elsewhere.

Discussion

The training and internships that medical students complete in hospitals are crucial for acquiring essential skills that cannot be obtained in a classroom setting. However, multiple studies have confirmed that virtual patients are widely accepted by medical students and significantly enhance the outcomes of traditional teaching methods.

Therefore, developing a platform that allows medical students to virtually replicate clinically valid hospital activities represents a significant opportunity. This platform not only provides an alternative for acquiring skills typically gained in hospitals, especially when access to hospitals is limited, but also offers a way to improve educational results and practice comfortably in their own environment before meeting real patients.

The Virtual Patient platform covers the diagnostic process, treatment planning, and patient interactions through speech and text-based natural language communication. While it does not integrate animations like some other solutions, it offers several distinct advantages compared to existing platforms.

One key advantage of the Virtual Patient is how scenarios are generated. Unlike other platforms that rely on predefined cases without the flexibility to separate different elements (such as patient history, anamnesis, ultrasound, and cardiac auscultation), the Virtual Patient platform allows professors to assign informative labels to each element of a medical case and define specific restrictions that the generation engine must follow. The scenarios are based on clinically valid associations between the labels of the elements, which come from real-life cases and generated elements based on real-life cases. This approach ensures that the scenarios are not predefined but generated dynamically from the uploaded elements, ensuring clinical validity. Moreover, the Virtual Patient platform supports natural language interactions between students and virtual patients, both in Romanian and English, through both speech and text. Furthermore, unlike other platforms, the Virtual Patient platform can generate synthetic medical data.

Another significant advantage is the combination of automated computer-based feedback and professor feedback provided to each student regarding their performance during a session. Other platforms typically offer only one form of feedback.

User evaluations showed that users are satisfied with the platform and appreciate its various features and functionalities. Some of their recommendations will be implemented in future updates, such as the development of an automatic interpretation component of medical data that includes the classification of medical images using machine learning (ML). For example, similar to the work in [13, 14], this ML component will compare the interpretation of a medical investigation provided by the student with the automatic interpretation of the medical images. In such a case, professors will have the to validate the automatic interpretation generated by the ML, similar to the work in [15].

Conclusions

The purpose of the Virtual Patient platform is to enable medical students to virtually replicate their hospital interactions with patients, including the diagnostic process and treatment planning, in clinically valid scenarios. The platform includes features such as real-time or post-session performance evaluations by professors, computer-based feedback, and the generation of clinically valid scenarios. Evaluations from 210 users have demonstrated their satisfaction with and the usefulness of the platform.

Future work includes expanding the range of medical cases to cover new diseases, enhancing the dialogue module to handle additional questions during the anamnesis section, improving the scenario generation module for complex scenarios, extending the synthetic data generation component to produce additional types of medical investigations, and including a ML component for automated interpretation of medical data.