This is post 102 of 122 in the series “Artificial Intelligence”

CIDDL Office Hours: Smarter Data Analytics with AI

Authors: Philip Garza and Kimberly Martinez; info@ciddl.org

During this CIDDL Office Hours session, Cheryl Lemke, President and CEO of The Metiri Group, walked participants through a practical seven-step workflow for using generative AI to analyze open-ended student and educator responses. The discussion demonstrated how AI can streamline thematic analysis, accelerate reporting, and support evidence-based decision-making — while maintaining the essential role of human judgment to ensure accuracy and fairness.

View the full presentation slides

Why This Matters

Open-ended responses fuel instruction, assessment, program evaluation, and accreditation processes, yet they’re notoriously time-consuming to code and summarize at scale. Recent studies suggest that, with guardrails in place, large language models (LLMs) such as GPT can achieve human-level reliability for many qualitative analysis tasks (Parker, Anderson, Stone, & Oh, 2025; Ding et al., 2024; Condor, 2020).

That means educators and researchers can ask more complex questions more often, and act on the results sooner, without sacrificing rigor or validity.

Key Takeaways

What AI Can Do Well

AI can effectively identify recurring themes across large sets of responses, produce succinct summaries with representative quotes, and align emergent codes to existing frameworks such as the ISTE Standards or Portrait of a Graduate. It can also generate ready-to-report tables and narrative sections for stakeholder reports. While these capabilities streamline analysis, they still benefit from expert review and spot-checking to ensure accuracy and transparency.

Real-World Examples

Higher Education (Small Dataset)

In a study of first-year STEM scholars at Fresno State University, AI analyzed students’ reflections on the non-financial benefits of an NSF S-STEM scholarship. With a single structured prompt, GPT-4 identified clear themes such as mentorship, belonging, tutoring, and peer networks, producing both percentage distributions and a concise, 100-word summary administrators could integrate directly into reports.

K–12 Statewide (Large Dataset)

The Utah State Board of Education’s Digital Teaching and Learning Program collected over 7,400 teacher responses. After filtering 1,099 AI-related records, Ms. Lemke used generative AI to extract and map patterns of classroom AI use to frameworks like ISTE Standards and Portrait of a Graduate.

Themes included creation and design, writing, summarizing/explaining, and personalization. Grade-band differences emerged (e.g., more writing in elementary grades, more design and collaboration in high school), directly informing professional development priorities.

The Seven-Step Workflow for AI-Assisted Thematic Analysis

Establish Purpose, Role, Context, and Outputs – Define your dataset, goals, and ethical parameters (e.g., “avoid speculation; rely only on provided text”).
Derive Preliminary Themes – Generate draft themes, short labels, and representative quotes from batches of ≤300 responses.
Apply or Create a Codebook or Framework – Align with frameworks like ISTE Standards, Portrait of a Graduate, or your district taxonomy for consistency.
Cluster and Summarize – Produce 2–3 sentence summaries for each category with example quotations.
Validate and Flag Anomalies – Identify low-confidence items for human review.
Check Reliability – Compare re-runs using the final codebook to compute agreement (Cohen’s κ or percent agreement).
Generate Narrative and Visuals – Create final tables, executive summaries, and representative examples for reporting.

This process aligns with CIDDL’s ongoing Resources Library and promotes transparency and replicability in AI-assisted evaluation.

Copy-and-paste Prompt Starter Pack

These modular prompts can be pasted directly into your AI tool:

Purpose, role, context, outputs: “Your task is to analyze open-ended survey responses from teachers. You are acting as a research assistant in a statewide evaluation. The dataset has three columns: ID, Level (elementary/middle/high), Scenario (free text). Analyze only the provided text. Avoid speculation. Allow multiple codes per response. Output a professional, report-ready summary with: draft themes, short labels, representative quotes, and estimated percentages.”
Preliminary themes: “Analyze the following responses.
Identify recurring themes of AI use.
Label each theme (short name).
Provide sample quotes.
Estimate % of responses per theme.”
Framework mapping: “Map each response to one or more of these categories: Consume; Create/Produce; Experience; Analyze Data; Communicate; Collaborate; Manage/Organize. Then provide a refined codebook with definitions and examples.”
Cluster and summarize: “For each category, summarize use patterns in 2–3 sentences in K–12 language. Note differences by grade band. Include representative quotes.”
Validate and flag: “List responses you coded with low confidence or that do not clearly fit assigned themes. Explain why and flag for human review.”
Reliability iteration: “Re-analyze the same responses using only the final codebook. Provide a table of response → category and compute percent agreement vs. your prior pass.”
Narrative and visuals: “Produce: a table of categories with descriptors and percentages; an executive summary of the top three uses by level; 2–3 verbatim sample scenarios per top use; and a one-paragraph overall summary suitable for a stakeholder report.”

Guardrails and Ethics

Treat AI as a co-analyst rather than a replacement, beginning with a clear codebook and refining it iteratively with exemplars. Always validate AI-generated outputs against a human-coded subset, documenting model versions, prompts, and any error cases that arise. It is also important to monitor for equity and bias considerations when interpreting patterns and making decisions. Together, these practices strengthen reliability, transparency, and trust with stakeholders.

Framework for Responsible AI Integration in PreK-20 Education

Insights from the Q&A:

First step for beginners: Try AI on a dataset you’ve already analyzed; compare outputs to build intuition about strengths and failure modes.
Teacher mindset shift: Make formative check-ins routine (exit tickets, one-question polls), then use AI to rapidly synthesize insights across sections. Pair analytics with an instructional chat assistant to explore evidence-based next steps for specific learners.
Collaboration insight: Many scenarios described students working solo with AI. That’s a PD opportunity: design tasks where peers iterate together and with AI as a partner.

To get started with your team, begin with one course, subject, or grade band, and pilot the seven-step workflow using last term’s reflections or exit tickets. Share the resulting codebooks and summaries at PLCs to spark strategic discussion and guide next steps.

Research Supporting Responsible AI Analytics

Recent research supports the careful integration of AI for both qualitative and quantitative educational data analysis:

Each of these studies affirms that AI can produce valid, reproducible results when prompts, frameworks, and human oversight are explicitly defined.

Join the conversation in our community!

CIDDL is committed to providing high-quality resources to support the increasing knowledge, adoption, and use of a range of educational technologies that can be used for educators, related services, or leadership preparation programs. For more resources, including videos and blogs, subscribe to our newsletter and follow us on YouTube, Facebook, and LinkedIn. The most important part of our CIDDL community is YOU. Join our community and share the innovative ways you are using technology, ask a question about technology integration, or participate in our bi-weekly live AI Community Chats. We look forward to seeing you in our community!