Where AI Actually Helps at Work

Microsoft had analyzed 200,000 real Bing Copilot conversations related to work. The data connected those conversations to the kinds of work people were trying to do and the occupations that typically perform that work.

I combined that with O*NET, a U.S. Department of Labor database that describes hundreds of occupations using the same set of characteristics—from communication and analytical thinking to physical demands, machinery, working conditions, and much more.

Where the two datasets overlapped, I had 738 U.S. occupations that I could compare on the same dimensions.

I used that data to explore two questions:

01Can the characteristics of a job help predict where generative AI is more likely to fit?

02And when AI appears to complete the task, where are people still unhappy with the result?

O*NET describes occupations on the same underlying scales. A software developer, nurse, mechanic, paralegal, roofer, and hundreds of other jobs can all be compared on things like communication, computer use, physical demands, fine motor control, mechanical knowledge, and analytical thinking.

I wanted to know whether those characteristics alone could tell us something useful about where generative AI was already being applied successfully.

They could.

Jobs centered more heavily on communication, computers, creativity, and analytical work tended to be better aligned with the kinds of tasks people were already using Copilot for.

Jobs involving more physical, manual, and mechanical work tended to be less aligned.

That does not mean every desk job is a good AI use case or every physical job is a bad one.

But it does suggest that organizations choosing where to experiment with AI can look at the nature of the work itself instead of selecting pilots almost at random.

Some kinds of work are simply a more natural fit for today’s conversational AI than others.

Using job characteristics, my strongest model could predict much of the difference in AI applicability across occupations.

Microsoft’s data also let me compare two different signals.

Did Copilot appear to finish what the person asked it to do?

And:

Did the person actually respond positively to the result?

Those sound like they should be almost the same thing.

They weren’t.

Every line begins on the left with how often Copilot appeared to complete the requested task and ends on the right with how often users responded positively.

A nearly flat line means the two measures were similar.

A falling line means Copilot appeared to finish the request much more often than people actually liked the result.

Almost every line falls.

Across occupations, Copilot appeared to complete requests very often. Positive reactions were much less consistent.

The important point is not the exact percentages.

It is that:

“The AI completed the task” and “the AI was useful to the person” were not the same thing.

The largest gaps tended to appear in occupations involving more physical and mechanical work.

So I wanted to understand what was actually driving that pattern.

Was it because people had their hands occupied?

Because typing into a chatbot was inconvenient?

Because it required precise hand movements?

Or was something else going on?

I separated physical work into several different characteristics and compared them.

One stood out far more than the others.

The biggest difference was whether the work kept changing while you were doing it.

Some jobs let you stop, consider information, and then act.

Others require you to constantly watch what is happening, do something, see how the situation changes, and adjust again.

Think about the difference between reviewing a document and operating a piece of heavy machinery.

The second kind of work never really holds still.

Now color those same occupations by how much the work requires continuous monitoring and adjustment.

That need for continuous monitoring and adjustment was the strongest signal associated with the gap between apparent AI success and user satisfaction. Characteristics like having your hands occupied or needing precise manual control mattered much less.

A chatbot generates an answer based on the situation described at one point in time.

That works reasonably well when the problem stays put long enough for someone to read the answer, think about it, and act.

But some work is a continuous loop.

A worker observes something, acts, sees what happened, and adjusts again. The environment may change. Equipment may respond. New information may become visible.

In that kind of work, an AI answer can be perfectly reasonable when it is generated and still become less useful moments later.

The answer is a snapshot.
The work keeps moving.

You can see that pattern in the occupations themselves.

A paralegal can usually pause, read, compare information, and decide what to do next.

A roofer, truck driver, firefighter, or heavy-equipment operator may be working inside a situation that continues changing while the guidance is being used.

That does not mean AI cannot help with those jobs.

It means the system may need to work very differently.

An obvious response is to make AI easier to use away from a desk.

• Voice interfaces.
• Wearable displays.
• Hands-free assistants.

Those may absolutely help people access AI while they work.

But they do not automatically solve the deeper problem.

A voice assistant can deliver outdated guidance more conveniently.

If the situation keeps changing, the AI needs some way to keep its understanding up to date too.

That could mean:

• receiving new information as conditions change;
• using images, audio, sensors, or equipment data;
• giving shorter, conditional guidance instead of one long answer;
• checking what happened after each action;
• making clear what it does and does not know.

The challenge is not simply getting the answer to the worker.

It is keeping the AI aligned with the situation the worker is actually in.

A system saying:

✓Task completed

does not necessarily mean:

That was useful.

For work that changes over time, judging one AI response in isolation may miss the thing that actually matters.

The more useful unit may be the whole interaction:

An answer might be completely reasonable when it is generated and become inappropriate after the person acts.

So evaluating these systems means asking different questions:

• Did the AI notice when important information was missing?
• Did it update when the situation changed?
• Did it make assumptions it could not actually support?
• Did it know when to ask for more information?
• Did it remain appropriately uncertain?

For systems used in changing real-world environments, success may depend less on producing one impressive answer and more on staying useful as the world changes around it.