AIME-Con 2026 program coding

AIME-Con 2026 program coding Data behind the opening chart of the moderator deck: how the 335 items in the printed program relate to AI in assessment products, AI in measurement professionals' own work, and the profession itself.

AIME-Con 2026 program coding Data behind the opening chart of the moderator deck: how the 335 items in the printed program relate to AI in assessment products, AI in measurement professionals' own work, and the profession itself. Current coding (v2, 6 October 2026): three independent yes/no questions **File:** `program-items-coded.csv`, one row per program item, with columns `product`, `profession` and `norms` (each 1 or 0), a rationale, and the v1 code for comparison. | Question | Items coded yes | Share of 335 | |---|---|---| | **product**: AI inside an assessment, learning or scoring product or service | 221 | 66% | | **profession**: AI in how measurement professionals do their own work | 131 | 39% | | **norms**: the profession itself (norms, roles, competencies, training, accountability, governance) | 22 | 7% | Rows overlap: an item can be yes on more than one question. 48 items are both product and profession; 15 of the 22 norms items are also profession items. **How the coding was done** - **Codebooks:** `methods/codebook-2d.md` and `methods/codebook-norms.md`, verbatim. These were the coders' only instructions. - **Primary coder:** an AI coder (Claude) given only the codebook and the item text (`methods/items-text.jsonl`), blind to the v1 coding. Outputs: `methods/coder1-items.csv`, `methods/coder1-norms.csv`, `methods/coder1-sessions.csv`. - **Second coder:** an independent AI coder (Claude, a separate instance) given only the codebook and a blind stratified sample of 98 items (`methods/second-coder-sample.json`, stratified on the v1 code and ambiguity, seed 20261006). Outputs: `methods/coder2-*.csv`. - **Agreement** (`methods/agreement.json`, n = 98): | Question | Agreement | Cohen's κ | |---|---|---| | product | 91% | 0.80 | | profession | 97% | 0.94 | | norms | 94% | 0.72 | Both coders are the same model family, so agreement shows the codebook can be applied consistently. It is not independent human judgment. - **Audit:** the chair reviewed the v1 profession list on 6 October with no changes. Disagreements and the items whose classification moved are listed in `methods/audit-2d.md` for the chair's audit. The primary coder's codes stand unless the chair changes them. - **Reproduce:** `python3 data/methods/merge.py data/methods/coder2-norms.csv` rebuilds `program-items-coded.csv` and the norms agreement.

Disagreements and the items whose classification moved are listed in `methods/audit-2d.md` for the chair's audit. The primary coder's codes stand unless the chair changes them. - **Reproduce:** `python3 data/methods/merge.py data/methods/coder2-norms.csv` rebuilds `program-items-coded.csv` and the norms agreement. **Why v2:** an adversarial review (6 October) pointed out that v1's A/B categories were not mutually exclusive: 81 of 335 items were flagged ambiguous, 48 of them A↔B. It also pointed out that v1's tie rule pushed items toward "product". v2 drops the tie rule. The large shift is that AI used as a psychometric method (difficulty prediction, calibration, item review, synthetic respondents) now counts as professional work. Under v1 it counted as product. That is why v1's 9:1 became about 1.7:1. The **norms** question was added to separate *using AI in our work* from *asking what AI means for the profession*. --- v1 coding (superseded; kept for comparison) Source *2026 NCME Artificial Intelligence in Measurement and Education Conference* printed program (`Assets/Damian_Betebenner-moderator/AIME-Con-2026-Program_sm-1.pdf`, 92 pp., created 2026-09-28). The text was extracted with `pdftotext -layout`, and only the FULL SCHEDULE pages were parsed. Method These files were coded by Claude (Opus 5.5) from the printed program on 2026-10-06, and audited by the chair. Each unit is judged from its title and its printed abstract or session description. Many coordinated-session papers print a title only. - **`program-sessions.csv`**: 83 sessions, one row per scheduled session. - Excluded: the break, the keynote overflow room, and two Sponsor Showcase slots with no title or content. - **`program-presentations.csv`**: 335 presentations, one row per bulleted presentation. - Panels are one bullet each. - Training sessions, keynotes and the two titled sponsor sessions have no sub-presentations, so each counts as one session-level item. Their bulleted learning objectives are not presentations. - Includes the 24 posters, plus three presentations printed without a bullet (two in "AI for Mathematics: Scoring & Item Development", one in "Formative Assessment & Feedback"). `ambiguous = TRUE` marks rows where a reasonable coder could choose `alt_code`.

Their bulleted learning objectives are not presentations. - Includes the 24 posters, plus three presentations printed without a bullet (two in "AI for Mathematics: Scoring & Item Development", one in "Formative Assessment & Feedback"). `ambiguous = TRUE` marks rows where a reasonable coder could choose `alt_code`. Codebook (verbatim) - **A = AI in assessment/education PRODUCTS & SERVICES:** automated scoring, AI item generation, AI tutors/chatbots, AI feedback, adaptive testing, item-parameter/difficulty prediction, AI-assisted test design/alignment, synthetic respondents used to build or evaluate assessment products, LLM judges for scoring, validity/fairness evidence FOR those AI products. - **B = AI in the PROFESSIONAL WORK of measurement professionals themselves:** AI in their own workflows (coding, analysis, modeling assistance e.g. "LLMs to assist in IRT modeling", writing, documentation, simulation, research), peer review/publishing, professional roles/competencies/careers/hiring, graduate training & curricula for the profession, dissertations, norms/governance of professional practice. - **C = neither:** no AI, or psychometric methods without AI, or general keynote/logistics where the content can't be judged, or AI literacy for students/teachers (not measurement professionals). **Tie rule:** if an item builds or evaluates a tool that end-users (students, teachers, test-takers) experience, code A. If it changes how the measurement professional does their job, code B. **Applied convention:** the enumerated A list takes precedence. AI used inside test-development operations (item QA, DIF screening, enemy-item identification, item writing) is coded A and flagged as ambiguous, with B as the alternative. v1 counts (the chair reviewed the B list on 6 October; no changes) | Unit | n | A | B | C | B share of A+B | |---|---:|---:|---:|---:|---:| | Sessions | 83 | 70 (84.3%) | 10 (12.0%) | 3 (3.6%) | 12.5% | | Presentations | 335 | 274 (81.8%) | 30 (9.0%) | 31 (9.3%) | 9.9% | **Sensitivity.** "Firm B" counts only B rows that are not flagged as ambiguous. "B if every ambiguous-to-B flips" also adds every row whose alternative code is B. | Unit | Firm B | Coded B | B if every ambiguous-to-B flips | |---|---:|---:|---:| | Sessions | 6 (7.2%) | 10 (12.0%) | 19 (22.9%) | | Presentations | 12 (3.6%) | 30 (9.0%) | 68 (20.3%) |