Research

Check out results from the first year of implementation of the SurgSJT. In this paper, we provide an overview of the general trends and relationships with other applicant factors – such as medical school, test scores, and demographics.

This is a fun one looking at the impact of implementing a structured interview program and InterviewWise training within a residency program. This program found that after implementing structured interviews, they were able to interview 23% fewer candidates, over doubled interviewer baseline knowledge about structured interviews, and had significant increases in agreement among interviewers.

We all know that surgery is one of the specialties with the highest rates of attrition. Our partners at a Texas program studied if our assessment was able to predict which trainees in their program would leave over a four-year time period. They found that the situational judgment test (SJT) we created for them was impactful, finding that those who performed low on the SJT were 4 times more likely to attrite.

Passion has been lauded as a key differentiator among those who thrive in surgery careers versus those who do not. However, the lack of empirical work in this area leaves this assertion at pure speculation. This project serves as a first stop in exploring how passion for surgery might be measured, and applicant reactions to that process. Take a look!

This multi-institutional study highlights how residency screening tools, developed with the rigor and high standards expected for employee selection, can help create more equitable opportunities for applicants from all backgrounds compared to traditional standardized tests.

This article serves as a practical evidence-backed resource for organizations seeking to increase the diversity of their workforce, from both a recruitment and selection perspective.

This study explores how perceptions of organizational culture and trainee competency expectations compare and contrast between training programs within a single department of surgery. While perceptions of organizational culture and values may be similar, residency and fellowship programs may differ in the competencies they expect from entering trainees.

Any good selection system should be able to proactively identify individuals who may need additional help if selected in. This paper shows just that – that incorporation of situational judgment tests (SJTs) can predict those at risk for formal remediation interventions up to a two years later.

When talking with Program Directors about what we do, they often express concerns over what perceptions and opinions applicants may think about them “doing something different” than other programs. They fear that inserting an additional hurdle or doing something outside-the-norm might create an unfavorable opinion for those applicants. In industry, the candidate experience is similarly a big thing. In fact, organizations compete each year to show that they offer the best “candidate experience” through their recruitment and screening practices. This manuscript begins to answer the question: “What will the applicants think?” Read on to see how they respond to new selection processes.

As mentioned above, the #1 concern among organizations we talk with is “scaring away the good applicants” through implementation of new processes or screening tools. This paper should put those concerns at rest. Across organizations of different status, geographical location, and tenure, 97% of all applicants will complete a pre interview screening assessment. In fact, this manuscript highlights that that 3% of folks who choose not to complete the assessment have significantly lower licensing examination scores.

We didn’t write this one, but were certainly involved! This is the work of one of our clients, who in our first year working together compared the candidate pool of those recommended to interview by our system versus those they would have interviewed using traditional applicant screening methods that involved manual application review and making a yes/no/maybe recommendation. They found that there was only about 20% overlap in the two methods. Importantly, our process was a lot more standardized, efficient, and resulted in a substantially higher match rate than the program had ever experienced previously.

In light of recent ACGME expectations that programs implement recruitment and selection methods that promote a diverse and inclusive training program, many Program Directors may be tempted to create an arbitrary quota for selecting underrepresented applicants as a quick solution to diversity needs. This brief commentary pulls in years of organizational and social science research to highlight how that “quick fix” can have deleterious consequences for those selected in under the quota system, their peers, and the organization more broadly.

Performance expectations among surgery residents is a moving target. Through rigorous multi-method job analyses conducted at numerous general surgery residency programs across the country, we show that the expectations of competencies expected among programs not only differs across programs, but is anticipated to dramatically change over the next 3-5 years, with many of those “non-technical” skills becoming increasingly important for surgeons in training to succeed. As a result, our selection systems need to catch up to make sure we are effectively and efficiently assessing candidate competencies that will ensure candidate success in our programs.

We sometimes refer to this as our “how to” guide of implementing an evidence-based selection system. Using a surgical fellowship as our case study, we describe the step-by-step process of conducting a multi-method job analysis, creating and validating selection assessment tools, implementing an online screening platform, and data from participating faculty about the value, efficiency, and effectiveness of the new program. We also note the difference in underrepresented minority candidates recommended through the new selection system compared to who would have made it through using the “traditional” method.

Ever wonder how much all of that time spent manually reviewing applications, interviewing hundreds of candidates, and sitting in hours-long ranking meetings is costing your program? Well the quick answer – quite a lot! In industry, measuring the efficiency of the employee screening process is one of the metrics used to measure a “healthy” selection system, and thus leader and organizational time is quantified and reported. Somehow in graduate medical education, we tend to forget how valuable our time is and neglect to calculate those opportunity costs. Read on to see how the average surgery residency program spends around $100,000 EACH YEAR on the resident screening and selection process, and how those costs are substantially higher for university-based and larger training programs.

This pro/con paper puts it all out there when considering how simulation might be used in the selection process. Of note, the paper weights the value of a number of forms of simulation, including situational judgment tests, work samples, and assessment centers.

Using a return on investment framework, this paper describes how we’re doing in regard to selection in surgical education. The short answer – not very well. The paper summarizes current evidence for our current selection system as measured by efficiency, remediation, and attrition rates.

This study examines the comparative effectiveness of a customized situational judgment test (SJT) and USMLE scores at predicting multi-dimensional performance criterion across five years of residents in a single institution. Spoiler alert – SJTs are more effective!

This study reveals the power – and necessity of – structured interviews. Using a pre/post design, this paper shows that faculty members have lots of variability in how they rate candidates and approach inappropriate questions. After an in-depth InterviewWise course, however, faculty improve their knowledge of structured interview basics and are able to achieve substantial consensus in their ratings.

Diversity is critical for our workforce to succeed. This brief commentary describes how implementing selection science can create a level playing field for applicants from all walks of life.