Leveraging artificial intelligence for resident recruitment: can the dream of holistic review be realized?

INTRODUCTION

The field of natural language processing (NLP) is one of the fundamental components of Artificial Intelligence (AI) that equips computer algorithms with the capability of “understanding” human language^[1]. In the NLP framework, theories of linguistics are combined with advanced AI techniques (artificial neural networks and deep machine learning) into a computer program that analyzes and interprets text and speech. Integrating behavioral and cognitive psychology with current NLP methodologies has enabled developers of AI systems to process language data in a way that can reliably provide insight into the individuals who produced the content [Figure 1]^[2].

Figure 1. An illustration of the methodology of generating personality characteristics from personal statements based on Natural Language Processing (NLP) technology.

There may be a role for this emerging technology within the Graduate Medical Education (GME) field. For instance, the application process for medical students seeking postgraduate training has become increasingly complex and uncertain. One of the significant challenges is the demand for a more holistic screening process when selecting candidates for residency interviews, calling for broadened consideration of characteristics, experience, and motivation, in addition to their academic performance^[3]. While considering each student as a unique, whole individual rather than as a set of numerical test results, promoting diversity, and recruiting based on skills crucial to patient care and work ethic, is important, the practical application is ambiguous at best. Significant problems with the call for holistic assessment include that it is time-consuming, complicated, poorly understood, and non-standardized, making it susceptible to bias and error, which will lead many program leaders to forgo applying this to their residency recruitment^[4]. Therefore, we must identify methods to address these shortcomings to achieve the benefits that holistic recruitment is intended to bring.

These challenges can be addressed by incorporating theories of artificial intelligence, which involves the development of computer systems that can perform tasks typically requiring human perception and reasoning. In its most practical form, artificial intelligence software functions as an assistant, handling some of the more laborious aspects of a task so that experts can focus their attention on making decisions of greater importance. Accordingly, in the present study, the personal statements of general surgery applicants were analyzed using a commercially available NLP platform that incorporates psycholinguistics, statistical language modeling, and contemporary AI principles to gain insight into the characteristics of prospective surgeons. This work was intended to provide a basis for incorporating AI technology into the residency selection process to make further advancements in pursuing holistic review.

METHODS

Setting and participants

This study examined the personal statements of medical students applying for general surgery residency positions at a single academic institution during the 2021/22 application cycle. All applicants’ personal statements were included to maximize generalizability and validity. Applicant personal statements were compared to a corpus of data from our partnering NLP group that represents national averages. Personal statements submitted via the Electronic Residency Application Service (ERAS) were prospectively collected. We prepared the personal statements for analysis by de-identifying, numbering, and collating them in an Excel spreadsheet that the NLP group could use for processing. The personal statements were not altered and were submitted verbatim as the applicants had written them. Furthermore, the personal statements of the current residents of the same academic institution (University of Maryland) were analyzed for comparison to the general applicant pool. The institutional review board deemed this study exempt from review. Figure 2 represents a flow diagram illustrating the study’s design.

Figure 2. Flow diagram illustrating the study design and methods.

RESULTS

Surgical residency applicant measured characteristics

Measures of personality

Figure 2 presents a summary of the big-5 language-based personality of general surgery applicants (in blue) compared to the general population (in orange). Figure 3 shows the 30 subcategories of the big-5 personality measures for applicants compared to the general population. In their personal statements, general surgery applicants exhibited a language-based personality that was highly self-assured (mean = 71.2, SD = 9.87; P < 0.0001), trusting (mean = 67.40, SD = 9.11; P < 0.0001), cheerful (mean = 62.04, SD = 14.48; P < 0.0001), and cooperative (mean = 60.31, SD = 6.15; P < 0.0001). In addition, applicants presented language-based personalities that were considerably less stress-prone (mean = 25.24, SD = 9.54; P < 0.0001), melancholy (mean = 32.04, 11.97; P < 0.0001), emotionally aware (mean = 35.20, SD = 11.27; P < 0.0001), and impulsive (mean = 38.99, SD = 9.18; P < 0.0001).

Figure 3. Comparison of the Big-5 personality measures derived from the personal statements of general surgery applicants (blue) to the general population (orange) using Natural Language Processing technology. Statistical analysis was performed using student’s t-test(***P < 0.0001).

Measures of drive

Figure 4 presents a summary of the language-based drive (i.e., psychological motivation) of general surgery applicants (in blue) in comparison to the general population (in orange). In their personal statements, general surgery applicants demonstrated that they were highly driven by achievement (mean of 90.61, SD of 9.08; P < 0.0001) and power (mean = 69.27, SD of 24.14; P < 0.0001). Moreover, applicants were found to be far less motivated by risk aversion (mean = 43.70, SD = 22.49; P < 0.0001).

Figure 4. Comparison of the 30 components of the Big-5 personality measures derived from the personal statements of general surgery applicants (blue) to the general population (orange) using Natural Language Processing technology.

General surgery residents compared to applicants

There was a total of 64 personal statements for current general surgery residents at our institution that were analyzed in this study. Residents were significantly more self-assured (mean 67.7; P < 0.001), trusting (mean = 66;P < 0.001) and cheerful (mean = 64.1; P < 0.001) as compared to the general population. Residents were also significantly less stress-prone (mean = 27.7; P < 0.001), melancholy (mean = 35; P < 0.001), and cautious (mean = 36.6; P < 0.001).

Figures 5 and 6 include the mean difference (MD) between the residents and applicants for the personality and drive measures, respectively. Comparing current residents to the general applicant pool, residents were significantly more emotionally aware (MD 5.39; P < 0.001) and organized (MD = 5.02; P < 0.001) and were significantly less disciplined (MD = -3.48; P < 0.001) and self-assured (MD = -3.48; P < 0.001). Residents were also significantly less driven by reward (MD = -11.16; P < 0.001) and power (MD = -10.47; P < 0.001).

Figure 5. Comparison of drive measures derived from the personal statements of general surgery applicants (blue) to the general population (orange) using Natural Language Processing technology.

Figure 6. The mean difference in the personality measures derived from the personal statements of current general surgery residents at a single academic institution and applicants who did not receive employment at that institution. Results were computed by subtracting the mean of the applicant’s personality measure from the mean of the current resident’s personality measure.

DISCUSSION

This study analyzed personal statements from general surgery applicants and current residents using a commercially available computational linguistics and psychoanalytic AI interface. We discovered that applicants to general surgery include unique characteristics in their personal statements that can be quantified using NLP. Candidates showed a language-based personality that was self-assured, trusting, intellectual, and not impulsive or prone to stress. All of these traits have face validity as desirable among surgeons. Applicants also exhibited a remarkable drive for achievement and power. Variations were notable between the applicant pool and current residents of a single academic institution when characteristics were compared. The residents were more emotionally aware, organized, humble, and less assertive and cautious. Furthermore, the residents were less motivated by power and achievement and more interested in avoiding risk.

This is the first study to evaluate the use of AI in the evaluation of personal statements of applicants for general surgery. Medicine and artificial intelligence have a long and interconnected history. AI has been integral in developing many medical technologies, from diagnostic tools to treatments and surgical procedures^[7]. In turn, medicine has been vital in advancing AI, providing data and insights that have helped to improve and refine these technologies^[8]. As AI continues to evolve, it is poised to transform medicine in various ways. AI-based diagnostic tools could, for example, improve patient care by identifying diseases in coordination with expert clinicians^[9]. AI-powered treatments could be personalized to each patient, making them more effective^[10]. Furthermore, AI-assisted surgeries could help minimize complications and speed up recovery times^[11]. Ultimately, the relationship between medicine and AI is poised to result in major advances in healthcare for patients around the world.

AI plays an increasingly important role in graduate medical education^[12]. In recent years, AI-based systems have been developed to provide personalized feedback to medical students, residents, and fellows^[13]. These systems use data from various sources, including clinical encounters, social media, and electronic health records. By analyzing this data, AI-based systems can identify patterns and recommend interventions to improve a trainee’s performance. AI-based techniques can also be used to create virtual patients that can be used for educational purposes^[14]. In the future, AI-based systems will likely play an even more significant role in graduate medical education, providing individualized feedback and creating custom learning experiences for each trainee^[15].

The selection of residents is among the many areas in which AI can be effectively incorporated into graduate medical education. In a landmark decision aiming to promote more holistic evaluations, the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME), sponsors of USMLE Step 1, announced that Step 1 score reporting will transition to pass/fail beginning in January 2022^[16]. This has left both candidates and residency programs uncertain about the future of resident selection. In addition, critics have argued that the new guidelines will prove ineffective, with USMLE Step 2 becoming the de facto Step 1^[17]. As a response to these concerns, the Association of Program Directors (APDS) has continued to recommend enhancing transparency (encouraging applicants to comprehend their chances of admission) in addition to integrating holistic review (considering both individual characteristics and academic performance) as a central component of the application process going forward^[18].

Currently, it remains unclear what part of the residency application package can be utilized to evaluate personal characteristics and how these qualities can be measured accurately. The personal statement is an integral component of the application materials that may provide valuable insights into the qualities that are part of a holistic review. Through the control of their narrative, applicants can demonstrate experiences, motivations, and personal characteristics. However, these attributes can be challenging to assess and quantify individually, which renders the use of a personal statement primarily subjective and unreliable. For instance, there is poor inter-rating reliability, programs report uncertainty regarding the role of the personal statement, and this document is given variable importance in residency selection^[19-21]. The mission should be to find methods of making personal statements more useful, standardized, and informative so that they may be incorporated into the residency selection process with enhanced consistency and reproducibility. We can look to modern technology and artificial intelligence for assistance in achieving this goal.

It is necessary to consider the ethical implications of AI involved in procedures that affect vulnerable populations, such as those seeking training positions to become independent practicing doctors. Ideally, machine learning models and the use of AI and NLP are objective and neutral. However, our perspectives, opinions, stereotypes, and other personal assumptions may impact the development and implementation of these technologies^[22]. The article published in 2021 by Dirk Hovy et al. on ethics in artificial intelligence provides a detailed summary of the five primary sources of bias in natural language processing technology^[23]. In machine learning, the language data used to develop NLP systems can lead to bias if produced predominantly by a small, homogeneous sample. It is necessary to incorporate diverse sources of information and challenge the stereotypes inherent in historical documents resulting from traditional worldviews and prejudices. Having perfected their API over the past 30 years with tens of thousands of validating studies, we partnered with a leading NLP group that aimed to reduce bias with the help of linguistics, social scientists, and psychologists. However, considerable research and development are needed to practically apply NLP to the residency application process. It will be necessary to standardize and validate the NLP platform to documents consistent with the values of medical recruitment so that it can reliably and effectively measure the important characteristics for selecting future physicians and surgeons that can meet the needs of a diverse patient population.

An additional concern associated with applying NLP to resident recruitment is the possibility of manipulating the evaluation to obtain unfair advantages. The Residency MATCH is intended to be an ethical process, with minimal manipulation by applicants and programs^[24]. However, it is possible that the applicants will misrepresent their statements to gain an unfair competitive edge in the match. The NLP group we work with has conducted extensive research to develop their evaluation of written text based on how it is written instead of the specific content words. The filler words, propositions, articles, and pronouns determine the correlation with the various emotional and drive scores generated, thus making it extremely difficult to manipulate this aspect of writing. The personal statement would likely become unreadable if this were attempted. It is also possible to assess them for multiple writing styles in such long documents to determine if another is writing their statements or if their style changes at different points in the document. However, further investigation will be required to ensure that no stakeholder is gaming the system to gain an unfair advantage.

In the admissions process, “fit” plays a vital role as it identifies candidates who can be better supported and more likely to succeed at a given program^[25,26]. In most cases, the fit is evaluated at the interview stage^[27]. Despite this, it may be challenging to determine compatibility during interviews, making this practice not universally successful, a problem amplified by the advent of virtual interviews^[28]. The advancements in NLP, a primary component of AI, have enabled the ability to “understand” language through artificial neural networks and deep learning techniques^[29]. The latest innovation has resulted in AI’s ability to derive psychoanalytic measures from language data^[30,31]. The findings of our study provide a foundation for incorporating AI, specifically NLP, into the residency selection process, aiming to gain a deeper and more complete understanding of the candidates and identify those that will be a good match for the program. Moreover, our results may serve as a basis for assisting programs in meeting their admissions objectives and increasing transparency while decreasing the reliance on quantitative measures such as USMLE Step exams.

This study has several limitations that must be acknowledged, including its limited generalizability as a study of one institution. However, we incorporated a robust sample size and included personal statements from applicants outside the institution. Further, despite the limitations in our capacity to establish causal inferences, we believe that the prospective nature of this study enhances our ability to draw meaningful conclusions. Personal statements are highly edited documents that may include contributions from many individuals, skewing the writer’s psychological composition and impairing the validity of personality and drive measures. Our partnering NLP platform generates a Genuine variable that assesses the entire document for consistency and clarity as a control measure. The “genuineness” of the personal statements was no different from that of the general population. The variables were developed using general documents, which may mask meaningful variation as they may be more similar to each other than the documents used to generate the scores. Addressing these validity concerns is an important future direction of our research.

Future research aims to improve the NLP characteristics extracted from residency application personal statements. In collaboration with our partnering NLP group, we are working to standardize future measures extracted from these documents to personal statements and expert opinions. Furthermore, we are expanding the evaluation of applicants’ personal statements to include more established and objective characteristic measures (such as leadership, organizational skills, professionalism, and intelligence).

CONCLUSION

Artificial intelligence is rapidly advancing and increasingly integrated into everyday life. Modern innovations have enabled a deeper and more trusted integration of AI in the medical field and medical education. Presently, the residency admissions process is in transition, with policymakers urging a holistic review, USMLE Step 1 becoming pass/fail, and programs uncertain of their future candidate selection practices. Considering this, we used NLP, an integral aspect of AI, to evaluate the personal statements submitted by candidates seeking general surgery training. We discovered that NLP could identify unique personalities and drive measures of the residency applicants. Additionally, we identified characteristic features of current residents relevant to an academic program that can be used to facilitate a more thorough match. The proposal is to integrate this emerging technology into the selection process to facilitate a more holistic review based upon the principle of fit, which will better assist applicants in finding the programs that can better promote their training and career development.