Guardian, Catcher, Architect: how AI reshapes the roles of the psychometrician (slide text and speaker notes)

Guardian, Catcher, Architect: how AI reshapes the roles of the psychometrician — slide text and speaker notes _Frank Rijmen, Cambium Assessment. AIME-Con 2026, 7 October 2026. Extracted verbatim from `Rijmen_AIME2026_provocation_Cambium.pptx` (slide text, then the presenter's speaker notes). Citable in public._ Slide 1 - GUARDIAN, CATCHER, ARCHITECT:HOW AI RESHAPES THE ROLES OF THE PSYCHOMETRICIAN - Presenter: Frank Rijmen - October 7th, 2026 **Speaker notes.** Most of the conversation about AI in measurement is about processes and products: item generation, automated scoring, adaptive testing. I want to talk about the people. Specifically, three archetypes you will recognize from any large-scale assessment organization. Slide 2 - The Architect - Bridges psychometrics with data science and emerging technology; high on openness; restless to redesign systems rather than run them. - With AI - Design to prototype, fast; must ground designs in measurement principles - The Catcher - Operational vigilance: detail-oriented, conscientious, risk-averse; works best when tasks and success criteria are well defined. - With AI - From box-ticking to forensics - AI changes not only what psychometricians need to know, but what behavioral skills are required - The Guardian - Safeguards the reliability, validity, and fairness of tests; treats the Standards as first principles. - With AI - Must reckon with evolving standards - Three archetypical roles in a testing company - Not exhaustive (for example, the Communicator). - Most psychometricians combine several, and a healthy organization needs them in balance. **Speaker notes.** These are archetypes, not job titles, and the list is not exhaustive; there is also, for example, the Communicator, who translates technical results for clients, policymakers, and the public. Today I focus on these three. Most of us are a mix, and a healthy organization needs them in balance. Many of these shifts were already underway before AI; AI is the accelerant. My claim is that AI shifts each of them differently. The Guardian has to reckon with standards that are evolving faster than the documents. The Catcher moves from box-ticking to forensics. The Architect can go from design to working prototype in days, which moves the bottleneck to grounding what gets built in measurement principles.

The Guardian has to reckon with standards that are evolving faster than the documents. The Catcher moves from box-ticking to forensics. The Architect can go from design to working prototype in days, which moves the bottleneck to grounding what gets built in measurement principles. Slide 3 - Behavioral skill profiles shift differently for each archetype - Before AI - With AI, gains weight - Guardian - Adherence; judging fit to a stable framework - Receptiveness to new methods; explaining why a principle matters; tolerance for ambiguity - Catcher - Diligence; thoroughness in checking outputs - Anticipating failure modes; comfort with open-ended problems; calibrated trust in automation - Architect - Creativity; persistence in building - Self-skepticism; verification discipline; owning results one did not produce by hand **Speaker notes.** Put together, the premium moves. For the Guardian, from adherence to openness and the ability to explain why a principle matters. For the Catcher, from diligence to anticipating failure modes, with calibrated trust in automation rather than complacency. For the Architect, from building to self-skepticism and the willingness to own results they did not produce by hand. Slide 4 - Behavior observed across time and contexts - Many observations, each with limited precision - Calls for dynamic measurement models - What test theory assumed - We cannot observe a person all the time, across all situations - So we sample behavior under standardized conditions - And infer a trait that predicts behavior across situations - AI questions an assumption on which the framework itself was built. - A deeper challenge for the Guardian - What AI makes possible **Speaker notes.** There is a deeper challenge for the Guardian, beyond standards that evolve. Much of test theory rests on a practical constraint: we cannot observe a person all the time, across all the situations we care about. So we sample behavior under standardized conditions and infer an underlying trait that predicts behavior and success across situations. AI weakens that constraint. When behavior can, at least in principle, be observed across time and contexts, the need to infer a trait from a small standardized sample is no longer self-evident. That questions an assumption the framework itself was built on. It does not make measurement theory obsolete: many observations, each with limited precision, call for dynamic measurement models.

That questions an assumption the framework itself was built on. It does not make measurement theory obsolete: many observations, each with limited precision, call for dynamic measurement models. Slide 5 - The unresolved problem - Developing behavioral skills may require a different approach than training cognitive skills. - What should professional development look like when the goal is new ways of working, not new knowledge? **Speaker notes.** So here is what I cannot resolve. The shifts I have described are not about replacing expertise; the deep subject-matter expertise psychometricians already have is the foundation. But developing behavioral skills, such as a Catcher's comfort with open-ended problems or an Architect's verification discipline, may require a different approach than training cognitive skills. We know how to teach a new method in a workshop. These skills tend to grow through experience, modeling, feedback, and learning from colleagues with a different profile. And forensic skepticism has traditionally been learned by doing the work by hand, which is exactly the work AI is taking over. So what should professional development look like when the goal is new ways of working, not new knowledge?