Testing AI Agents on Real Work: What the Ironclad Note Points To

What happened

OpenAI posted a note called "Advancing computer use with Ironclad." It says OpenAI and Ironclad are training and evaluating AI agents on complex contracting workflows.

The goal they give is to advance computer use for professional work. That means an agent that works through real software steps, not just one that answers a question in a chat box.

The summary I read is short on details, so I won't guess at results. What caught my eye is the word "evaluating" right next to "training."

What it means

When I test a model, the first question I ask is simple: tested on what? A score on a general test tells you little about your own work.

Contract work is a good example. It has many steps, and a small slip in one step can matter a lot. An agent that looks fine in a demo can still miss a step in the middle.

So the idea here is worth copying, even if you never touch contracts. Test the agent on the exact kind of work you want it to do. Then judge it step by step, not only on the final answer.

Check it yourself

Here is what I'd do before trusting any agent with real work.

First, pick a small set of real tasks from your own week. Use tasks you already know the right answer to.

Next, write down the steps a careful person would take for each one.

Then run the agent on every task. Save what it did at each step, not just the end result.

After that, mark each step right or wrong. Count where it fails. Look for a pattern, like the same step failing again and again.

Last, keep a few new tasks aside and test on them later. If it does worse on new work, the first test was too easy.

If the vendor can't show you results on work like yours, ask them to run this with you.

Where this fits

That last step is the same idea I use in my case study on testing an alert model on later data. A model can look good on the data it was built on and slip on what comes next. Checking it on later, unseen cases is how I find out.