Eliciting Information about Teacher Applicants Using Open-Ended Questions

Authors

• Joseph Aguilar-Bohorquez, University of MinnesotaFollow
• Dan Goldhaber, University of WashingtonFollow
• Cyrus Grout, University of WashingtonFollow
• Elton Mykerezi, University of MinnesotaFollow
• Prayash Pathak, University of MinnesotaFollow
• Aaron Sojourner, W.E. Upjohn Institute for Employment ResearchFollow

Upjohn Author ORCID Identifier

https://orcid.org/0000-0001-6839-2512

Series

Upjohn Institute working paper ; 26-440

DOI

10.17848/wp26-440

Issue Date

Aug-26

Abstract

Employers commonly use open-ended questions in hiring, but usually responses aren’t recorded, and knowledge about the usefulness of such responses is limited. This study investigates properties of written responses to open-ended questions collected over many years via an application tracking platform that school districts use to hire teachers. We develop a scalable process to discover topics within the corpus of responses to each question and then to classify each response with respect to the topics while preserving interpretability and transparency. It adapts a traditional qualitative topic modeling approach, Inductive Thematic Analysis, to best combine the strengths of researcher judgment and large language models (LLMs), introducing Hybrid LLM–Topic Model Inductive Thematic Analysis (HLTM). Automated scoring achieves human-level reliability in document classification. Questions elicit distinct, nonredundant information regarding a candidate’s pedagogical skills, classroom management, professional philosophy and growth mindset, relational and cultural competence as well as other professional attributes and institutional fit, but questions vary along important dimensions. Questions regarding classroom activities and student progress generate more dispersion in responses across applicants and fewer extreme demographic selection ratios. In contrast, philosophy-based questions produce more uniform, potentially scripted responses. This scalable framework empowers researchers and practitioners to assess language based hiring instruments transparently and to monitor potential demographic disparities.

Sponsorship

U.S. Institute for Educational Sciences

Subject Areas

EDUCATION; K-12 Education; Teachers and compensation; Job search