What should schools ask AI vendors?

Ask about instructional outcomes, evidence, measurement validity, student data, model training, human oversight, integration, workload, and exit procedures.

How long should an EdTech pilot run?

The correct duration depends on the outcome being measured. It should be long enough to observe meaningful use and collect evidence against predefined success criteria.

Are teacher satisfaction surveys enough?

No. Satisfaction can be useful, but it should generally be combined with operational and instructional outcome measures.

How to Evaluate AI Learning Tools in 2026

How Should Schools Evaluate an AI Learning Tool?

Schools should evaluate an AI learning tool across at least seven dimensions:

instructional purpose,
evidence quality,
measurement validity,
student outcomes,
privacy and safety,
educator control,
operational workload.

The U.S. Department of Education's August 2026 guidance emphasizes instructional value over recreational engagement and recommends pairing innovation with evidence, educator judgment, parent transparency, and attention to student outcomes.

That provides a useful principle:

Do not evaluate an education tool by how impressive the demonstration looks. Evaluate what changes for teaching and learning.

1. What Instructional Problem Does It Solve?

Do not begin with features.

Begin with the problem.

Examples:

inconsistent teaching quality,
delayed learning-gap detection,
assessment workload,
weak curriculum visibility,
lack of differentiation,
parent communication.

If the problem cannot be stated clearly, measuring success becomes nearly impossible.

2. What Evidence Supports the Claims?

Ask whether claims are based on:

peer-reviewed research,
controlled pilots,
observational evidence,
customer testimonials,
vendor-generated case studies,
theoretical assumptions.

These sources do not carry equal evidentiary weight.

3. What Does the System Actually Measure?

A dashboard may look sophisticated while measuring something weak.

For example:

time in app is not mastery,
number of questions answered is not retention,
course completion is not conceptual understanding.

Schools should examine whether the metrics correspond to meaningful instructional constructs.

4. Can Teachers Verify the Recommendations?

AI recommendations should be inspectable.

If a platform identifies a misconception, teachers should be able to understand the evidence behind that conclusion.

Black-box recommendations are particularly problematic when they influence important instructional decisions.

5. What Happens to Student Data?

Apply the same vendor-governance questions discussed earlier:

collection,
training,
retention,
ownership,
deletion,
subprocessors,
access.

6. Does the Tool Save or Create Work?

EdTech frequently promises efficiency while introducing new workflows.

Measure:

setup time,
teacher verification time,
data-entry requirements,
training requirements,
duplicate systems,
reporting effort.

A tool that saves five minutes but requires twenty minutes of maintenance has not improved efficiency.

7. Can You Run a Measurable Pilot?

A strong pilot has:

Baseline → Intervention → Success Metric → Observation → Decision

Do not run a "pilot" that simply gives users access and asks whether they liked the software.

Measure outcomes connected to the original problem.

The Omnix360 Instructional Value Scorecard

Evaluate every product across:

Problem Fit → Evidence → Measurement → Teacher Control → Privacy → Workload → Outcome

A visually impressive AI feature that performs poorly across these dimensions should not receive a pass simply because it uses advanced technology.