AI systems are increasingly trained on real human workflows — capturing clicks, keystrokes, and actions rather than written instructions. Image Credit: KorishTech (AI-generated).
Meta does not just want to know what employees produce.
It wants to know how they produce it.
In April 2026, reports revealed that Meta Platforms had begun collecting detailed interaction data from employee work devices — including mouse movements, clicks, keystrokes, and occasional screen snapshots. The stated goal is to improve AI training data for systems that can perform real office tasks.
AI training data is no longer just text — it now includes real human behaviour captured from everyday work.
The important part is not the tracking itself.
It is what the tracking is for: teaching AI how work unfolds step by step.
AI Cannot Learn Work From Instructions Alone
Most AI systems are trained on text.
Documents, manuals, emails, and code repositories all describe what work should look like when finished. They present tasks as clean, complete units. A report is written. A form is submitted. A request is approved.
But this is a compressed version of reality.
Written instructions describe the goal. Behaviour data shows the path.
A manual tells a person what should happen. A workflow trace shows what actually happens when that person meets a messy system.
Instructions remove hesitation, backtracking, and correction. They assume the user already knows the process.
In practice, this is not how work happens.
AI trained only on text learns what a completed task looks like. It does not learn how that task is actually carried out.
Traditional AI training data relies on text and static datasets, which miss how real work unfolds.
Real Work Happens Between the Steps
A task like “submit the report” appears simple when written down.
In reality, it unfolds through a sequence of actions.
A user opens a template, copies data from another system, checks figures, switches between tabs, corrects formatting, reviews the output, fixes an error, attaches a file, previews it, and only then submits.
The value is not in one click. It is in the sequence.
Each step changes the environment. The next decision depends on that new state.
Work is not a single action. It is a chain of state-dependent decisions — often interrupted, retried, or adjusted based on feedback.
This is the part of work that instructions do not capture.
This is also why AI orchestration systems have become important. Modern AI systems must coordinate actions across multiple steps, tools, and changing environments rather than generating a single response.
Why Clicks, Keystrokes, and Screenshots Matter for AI Training Data
This is where behaviour data becomes useful.
Mouse movements, clicks, keyboard shortcuts, and navigation patterns are not random. They are signals of how a task unfolds in real time.
A click only becomes meaningful when it is connected to the screen state before and after it.
A quick keyboard shortcut suggests familiarity. Repeated hovering over a button may indicate uncertainty. Switching between tools reveals dependencies between systems.
These micro-actions form a trace — a record of how the user moved through the task, what they tried, and how they corrected themselves.
Unlike instructions, which describe what should happen, behaviour data shows what actually happened.
It captures the invisible middle of work.
Meta’s Approach: Capture the Workflow, Not the Output
To close this gap, Meta Platforms has deployed tracking software on company-issued devices that records:
- mouse movements and clicks
- keystrokes and keyboard shortcuts
- navigation across internal tools
- occasional screen snapshots for context
This data is tied to real work sessions and collected continuously while employees use internal systems. Reports indicate there is no meaningful opt-out when using company devices.
The issue is that this is work behaviour captured as training material.
Employees are not just using tools. Their use of tools becomes part of the model-building process.
The objective is to build AI systems that can operate software the way humans do. To achieve that, Meta needs examples of real workflows — not idealised instructions, but actual sequences of actions taken under real conditions.
This is a shift in AI training data — from written instructions to real workflow behaviour.
Behaviour Data Teaches AI What Happens Next
This type of data changes what the AI is learning.
Traditional models are trained to predict outputs. Given an input, they generate a response.
Behavioural data introduces a different problem.
The system must learn what action typically follows a given state.
In simple terms, the AI is learning what action usually follows a situation.
This type of AI training data allows systems to learn what action comes next.
This is one reason modern AI agents require more than language generation. To complete tasks reliably, they must learn how actions unfold across changing environments.
From behavioural traces, AI systems can learn:
- how tasks progress step by step
- how users respond to incomplete information
- how errors are identified and corrected
- how decisions change depending on context
This is not just pattern recognition.
It is learning how to act over time.
The Shift From Output Training to Action Training
This represents a structural shift in AI development.
Traditional training relies on static datasets. Each example is treated as separate and complete. The model learns patterns across independent instances.
Behavioural training relies on sequences.
Each action depends on the previous one. The system is exposed to how tasks evolve over time, not just how they end.
This changes the structure of AI training data from static outputs to continuous behavioural sequences.
The focus moves from producing the right answer to taking the right step.
This reflects a broader shift already visible in physical AI systems, where AI must continuously interpret signals and decide what action happens next rather than simply generate outputs.
That is why behaviour data matters more for agents than for chatbots.
AI is no longer just generating outputs.
It is being trained to operate within environments.
The Problem: Behaviour Is Messy Data
Real-world behaviour is not clean.
The same task can be completed in different ways by different people. Even the same person may perform it differently depending on time, pressure, or context.
Behaviour does not always reflect best practice.
It reflects what people actually do.
If a team has a poor workflow, the data may teach the AI to repeat that poor workflow at scale.
The system may struggle to distinguish between intentional actions and accidental ones. Without context, a sequence of actions can be ambiguous.
More data does not remove this problem.
It makes it more complex.
The Trade-Off Is Capability Versus Privacy
Capturing behaviour at this level introduces a clear trade-off.
The same data that allows AI systems to learn real workflows also turns everyday work into training material.
The more useful the data becomes for AI training, the more intrusive it becomes for the worker.
When interaction data is collected continuously and tied to individual sessions, the boundary between improving systems and monitoring people becomes unclear.
There is also a technical trade-off.
Real data is more realistic, but less controlled. It introduces noise, variability, and edge cases that are difficult to standardise.
Improving capability requires accepting this uncertainty.
When AI Learns the Wrong Workflow
Learning from behaviour does not guarantee correct behaviour.
AI systems may replicate patterns that are common but inefficient. They may adopt shortcuts that work in specific contexts but fail elsewhere.
A bad habit is one thing when one employee does it.
It becomes something else when a model learns it and applies it repeatedly.
The system can become consistent without becoming optimal.
It learns what people do, not necessarily what they should do.
My Take
This is not just a change in how AI is trained.
It is a change in what AI needs to see.
AI systems are moving from learning what tasks look like to learning how tasks are performed. That requires observing real behaviour — not descriptions, but actions.
AI training data is becoming more behavioural, not just informational.
But observation is not understanding.
The system still interprets what it sees. It still predicts what should happen next. And those predictions are shaped by imperfect, inconsistent human behaviour.
The more AI moves from language into action, the more valuable human behaviour becomes as training data.
That is why this story matters.
Sources
- Reuters — Meta to start capturing employee mouse movements, keystrokes for AI training
https://www.reuters.com/sustainability/boards-policy-regulation/meta-start-capturing-employee-mouse-movements-keystrokes-ai-trai - BBC — Meta to track workers’ clicks and keystrokes to train AI
https://www.bbc.com/news/articles/cvglyklz49jo - The Irish Times — Meta to capture employee mouse movements and keystrokes for AI training
https://www.irishtimes.com/business/2026/04/21/meta-to-start-capturing-employee-mouse-movements-keystrokes-for-ai-training-data/ - Carnegie Mellon University — Behavior-Driven AI Development (CMU-HCII-24-101)
http://reports-archive.adm.cs.cmu.edu/anon/hcii/CMU-HCII-24-101.pdf - PLOS ONE — Effects of human training data on AI behaviour
https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0339482&type=printable - arXiv — AI Agents: Evolution, Architecture, and Real-World Applications
https://arxiv.org/html/2503.12687v1