Your AI Tutoring Vendor Is Building a Psych Profile of Every Student. Now What?

When a student sits down with an AI tutoring system, the interaction looks routine. A question, an attempt, some feedback. Underneath that session, the system is building something no teacher ever could: a continuous, statistical, tireless psychological portrait drawn from every hesitation, every abandoned problem, every mistake, and every spike in engagement. Add the socioeconomic data, personality inventories, and social-graph information that many platforms now ingest, and the result is a behavioral dossier that will persist in a vendor’s cloud long after the student has graduated.

In their 2026 analysis Cognitive Security Architecture for Intelligent Tutoring Systems, Deniaga and Fadel of the Center for Curriculum Redesign argue that this is more than a privacy concern. It is a structural failure in how AI tutoring platforms are architected, and it leaves most school districts exposed across two distinct dimensions of risk.

This breakdown summarizes the paper for K-12 technology leaders and ends with the question every CTO needs to be ready to answer. What does this mean for the next procurement cycle?

The Two Dimensions of Risk

District technology leaders already understand the external risk. Criminal actors, nation-state groups, and ransomware operators target student information systems, vendor platforms, and the third-party tools that increasingly hold pieces of a student’s record. CCR adds a sharper edge to that familiar threat. The timeline for quantum-decryption capability has compressed, with credible estimates now pointing to the late 2020s through mid-2030s. Student records encrypted under today’s classical cryptography may be exfiltrated this year and decrypted well within their mandatory retention windows. “Harvest now, decrypt later” is no longer a hypothetical attack model for K-12 data.

The second dimension is harder for districts to see because it does not require any wrongdoing. A vendor can be fully compliant with FERPA, COPPA, and SOPPA. A platform can be encrypted, access-controlled, and contractually limited. The dossier the system constructs about a student can still be the problem. The paper describes this as the “prescribed child” risk: a behavioral or psychological label generated by an algorithm at age nine can shape teacher expectations, admissions decisions, and the student’s own self-concept for years. The harm is not the breach. The harm is the existence of the profile.

Key message for CTOs: The risk surface is no longer just unauthorized access. Authorized use can do harm of its own. District governance needs to address both.

Five Layers in Plain Language

The paper proposes a unified technical architecture for protecting students against both risk dimensions at once. CTOs do not need to implement these systems themselves, but they do need to be able to ask vendors which of the five layers are present and how each is deployed.

The five layers, translated from research vocabulary into procurement vocabulary:

1. Post-quantum cryptography. Encryption standards designed to remain secure after quantum decryption becomes feasible. Vendors should be able to describe a migration timeline, not a vague intention.
2. Federated learning with differential privacy. A design pattern that lets a system improve from many users’ interactions without centralizing the raw behavioral data in a vendor cloud. The model travels to the data, not the other way around.
3. Zero-knowledge proofs. A cryptographic method that lets a vendor demonstrate that a personalization decision is sound without exposing the underlying student data to auditors, regulators, or third parties.
4. Homomorphic encryption. A method that allows computation to run on encrypted data without the data ever being decrypted on the vendor’s servers. The system can act on the data without reading it.
5. Architecturally enforced inference boundaries. Hard constraints, built into the system’s design, on what the AI is allowed to infer about a student in the first place. Not a policy promise. Not a contract clause. A technical floor.

Key message for CTOs: This is an architecture, not a checklist. A vendor that addresses only one or two of these layers is asking the district to absorb the risk on the others.

The Analog-Only Floor

For the highest-risk categories of student information, CCR makes a recommendation that will be unfamiliar to many technology leaders accustomed to digitizing everything. Some observations should never enter a digital system at all.

Emotional state. Personality traits. Anxiety patterns. Cognitive vulnerabilities. These are the categories the paper flags as carrying the heaviest long-term risk, both because they can be inferred from ordinary behavioral data and because they can shape how a student is seen for the rest of their academic career.

The safest digital record for these categories is no digital record. A teacher’s observation that a student seemed discouraged after a difficult math test belongs in the professional judgment of that teacher, in a conversation with a counselor if it matters, and within the existing student support processes the district already operates. It does not need to become a structured field in a tutoring platform’s behavioral schema.

Data that was never digitized cannot be decrypted in 2030 by a quantum adversary. It cannot be sold in a vendor’s asset sale. It cannot be queried by an automated system that does not yet exist.

Key message for CTOs: Not everything teachers know about students should be entered into a vendor’s database. Some forms of professional judgment are too sensitive to convert into machine-readable, persistent, queryable data.

From Data Governance to Inference Governance

Most district AI conversations still center on data governance questions. What is collected, where it lives, who can access it, whether the vendor can use it for model training. Those questions remain necessary. They are not sufficient.

AI-powered tutoring platforms force a second question into the procurement conversation. What should this system be prohibited from inferring about students in the first place?

A platform may not ask a student whether they are anxious. It may infer anxiety from hesitation, repeated answer changes, topic avoidance, writing tone, response latency, or engagement drop-off. A platform may not request a personality profile. It may construct one across hundreds of interactions over a school year. Limiting what data the vendor collects does not, by itself, limit what the system computes from that data.

This is the move from data governance to inference governance. Data governance asks what is collected, stored, shared, and deleted. Inference governance asks what conclusions the system is allowed to derive, retain, and act on. The two work together. Neither one alone is enough.

Key message for CTOs: A vendor’s data policy is a starting point. The district also needs to know what the system is allowed to infer, what those inferences are used for, and how long they persist. Inference is the part of the AI conversation most current policies do not yet cover.

What This Means for Vendor Conversations

Most district AI procurement reviews are not currently structured to surface answers to the questions raised in this paper. The questions below, drawn from the paper’s recommendations, are the ones CTOs should be ready to ask in vendor meetings and renewal conversations. They are not gotcha questions. They are the operating conditions for the next generation of K-12 instructional AI.

• What categories of behavioral inference does your system derive from student interaction data? Can you provide a complete taxonomy?
• Does your architecture centralize raw student behavioral telemetry, or do you use federated learning to keep that data local?
• What is your post-quantum cryptography migration timeline? Have you inventoried which student-data stores are still protected by classical public-key encryption?
• What is your data retention policy for derived inference data, not just PII, but the behavioral and psychological profiles your system constructs?
• Can you demonstrate, through a zero-knowledge or equivalent mechanism, that your personalization decisions are sound without requiring us to expose student data to auditors?
• Does your system derive emotional-state, personality-trait, or anxiety-profile inferences? If so, under what authority and with what retention policy?
• What happens to student inference data if your company is acquired, merged, or ceases operations?

The first time a CTO asks these questions, the vendor on the other side of the table may not have ready answers. That itself is useful information. A vendor that cannot answer these questions today is unlikely to be ready for the procurement environment forming around AI in K-12 over the next 18 to 24 months.

Key message for CTOs: These questions are not designed to make anyone’s life harder. They are the new baseline. Districts that begin asking them now are building the procurement muscle they will need when these questions become standard.

The CoSN Connection

The Cognitive Security Architecture paper is complex, written for an audience of cybersecurity specialists and policy analysts. The translation work into district practice has barely begun.

This is where CoSN can lead. The paper’s findings point to several practical needs that fit directly into work the CoSN community is positioned to do:

• AI procurement guidance that incorporates inference governance, not just data governance.
• Updates to student data privacy and retention frameworks that account for derived behavioral and psychological profiles.
• Vendor transparency expectations around AI inference and profiling.
• Governance tools districts can actually operationalize, not just sign.

Beyond the operational work, there is a policy conversation forming around student cognitive privacy that K-12 voices need to shape before it is shaped for them. The 13-to-17 age range, the developmental window most exposed to identity classification by AI systems, currently receives diminished protection under federal law. That is a gap CoSN’s policy advocacy is positioned to address.

The paper ends with a call for an International Child Cognitive Privacy Standard. That is a long horizon. The shorter horizon is the next district AI procurement cycle. The vendor conversations happening in school districts over the next twelve months will set the patterns that shape what AI tutoring looks like in K-12 for the rest of the decade. The work of translating cognitive security from research vocabulary into district practice starts now.

• This blog is based on the recently released paper Cognitive Security Architecture for Student Learning Data  by Angelina Deniaga, Masters’ student at Harvard Graduate School of Education in collaboration with Charles Fadel, founder of the Center for Curriculum Redesign. It describes in detail the hidden risks and heightened need to protect student information in an age of AI and Quantum Computing, also detailing the technical architecture to do so. It concludes with recommendations about what student profiles and capabilities need to remain in analog format.   We recommend you read this work for a deeper dive.

By the CoSN AI Committee, in collaboration with the Center for Curriculum Redesign (CCR)

Published on August 5, 2026

CoSN is vendor neutral and does not endorse products or services. Any mention of a specific solution is for contextual purposes.