Building an AI agent was quick. Testing it took longer.
Jinfo Blog

By Anja Chemnitz Thygesen

Abstract

• How I built an agent for subscription management
• Looking beyond a convincing report
• Each change needs another check
• A prototype is just the start.

Item

Could an AI agent help an information manager review subscriptions, analyse usage and identify issues requiring attention?

As part of Jinfo's research into AI implementation for the new report "Building AI solutions for information work", I decided to build one.

Using natural-language instructions, I configured an assistant to:

• calculate cost per use
• identify renewal dates
• highlight unusual usage patterns
• recommend follow-up actions.

It produced reports for management, subscription managers and departments from the same analysis.

Getting the initial prototype working was relatively quick. Establishing whether its reports could be relied upon took longer.

Looking beyond a convincing report

I used fictional data covering 30 subscriptions and 12 months of usage. To test the agent, I left gaps in the records, added figures that didn't match, and included sudden jumps in usage and months with no activity.

The first reports looked clear and well organised. But I still needed to check whether the agent had spotted the important issues and understood them correctly.

It sometimes missed context, drew conclusions too quickly, or struggled with conflicting figures. I then refined the instructions so it would point out problems in the data and make clear when it could not reach a firm conclusion.

Each change needed another check

In the following steps, I adjusted the instructions to help the agent explain uncertainty, take business needs into account and suggest clear next steps. After each change, I checked whether it had solved the problem and whether anything else had gone wrong.

Creating reports for different readers meant more checks. Important details could disappear from a short summary, or a possible concern could become a firm recommendation.

To catch these mistakes, I had to review each report and compare it with the full analysis to make sure the findings still told the same story.

A prototype is just the start

I found it relatively easy to build a prototype. The harder part was checking its work.

I still needed to review the results and question whether they made sense, question its conclusions and decide what needed another look.

The agent used fictional data. Before relying on it in daily work, I would need to test it with real records and work out how to keep the information up to date.

In Jinfo's report, "Building AI solutions for information work", I draw on this experiment and other practitioners' experiences to explore how to build, test and use AI tools.

Read the report for practical lessons on putting AI to work in information management.

« Blog