We Let an AI Agent Run Our Editorial Ops for a Week. Here's the Time Log.
Draft-chasing, tagging, the newsletter, the calendar: we handed a magazine's invisible second job to an agent, with four permission levels and one office whiteboard. Here's what it saved and where it broke.
Every magazine has a hidden second job. Behind the essays and interviews sits a mountain of small, unglamorous work: chasing drafts, moving deadlines, tagging stories, assembling the newsletter, reminding a contributor for the third time that their invoice needs a tax form. At Ideabro that work eats roughly a day a week of somebody's attention, and that somebody is usually me.
So for one week, we handed it to an agent.
Not a demo. Not a sandbox. A general-purpose agent built on a frontier model, connected through a handful of tool integrations to our shared calendar, our drafts folder, our content system and a dedicated email inbox. It could read everything in those places. What it could do was a different question, and answering it turned out to be the most valuable part of the experiment.
The rules we set before Monday
We gave every task one of four permission levels and wrote them on the whiteboard next to the coffee machine, which is the most sophisticated governance framework our office has ever produced:
- Read — look, summarize, report back. No changes.
- Draft — prepare the change and leave it for a human to send or publish.
- Act with approval — make the change only after a human clicks yes.
- Act — just do it, and log what you did.
Everything started at Read or Draft. A task could only move up a level after the agent had done it correctly three times in a row at the level below. Nothing customer-facing ever reached level four.
The week, day by day
Monday: the inventory
The first job was simple: read our calendar, drafts folder and editorial tracker, then tell us what was late, what was at risk and what was missing.
The report arrived in about four minutes and was uncomfortably good. It found two stories marked "in edit" that hadn't been touched in nine days, a newsletter slot with no assigned piece, and an interview scheduled during a contributor's stated vacation. It also flagged, correctly, that our tracker and our calendar disagreed about the publication date of a feature, and asked which one was the source of truth.
That question was the first sign of something we'd see all week. The agent was at its best when our own systems disagreed, because it noticed the disagreement instead of picking a side.
Tuesday: chasing drafts
Draft-chasing went to the Draft level. The agent wrote polite reminder emails for four overdue pieces, tailored to each writer's history with us. One got a warm nudge. One got a gentle note about a hard deadline. All four sat in a review folder until I sent them.
I edited one sentence in one email. The other three went out untouched, and all four writers replied by Wednesday.
Wednesday: where it broke
Wednesday was humbling, in a useful way.
We promoted tagging to Act with Approval. The agent read each new story and suggested tags from our shared vocabulary. It tagged three pieces "Regulation" because they used the word "rules" — one of them was about the rules of a board game. Approval caught all three, but it was a reminder that the agent was matching words, not understanding our editorial meaning of a tag.
Then the scheduling incident. The agent, still at Draft level for calendar changes, prepared an invite moving an interview to Thursday at 10 a.m. It was 10 a.m. for us. It was 2 a.m. for our interviewee. The invite never went out because a human reviewed it, but the draft was confident, formatted and completely wrong.
The agent never once said "I'm not sure." The mistakes arrived in exactly the same calm, competent tone as the successes.
That sentence went up on the whiteboard, too. Confidence is not a signal. If you're building any agent workflow, the review step has to be designed for errors that look exactly like correct work.
Thursday: the newsletter
Assembling The Ideabro Brief is a fiddly task: pick the stories, write short teasers, order them, check every link, fit it to our template. We let the agent prepare a full draft from the week's published pieces and our editor's shortlist.
The structure was right. The links all worked — it checked them, which is more than I always do. The teasers read like a press release that had been to a good school: tidy, correct and nobody's favorite sentence. Our editor rewrote five of eight. Still, starting from a correctly assembled draft instead of a blank template saved real time.
The tally
We logged every task, at every level, against how long the same work took us the week before. These are our own numbers from one small team over one week, so treat them as a field note, not a benchmark:
| Task | Usual time (prior week) | Agent week | Permission level reached | Verdict |
|---|---|---|---|---|
| Status inventory and risk report | 90 min | 15 min (review) | Read | Keep |
| Draft-chasing emails | 45 min | 10 min | Draft | Keep |
| Tagging new stories | 30 min | 20 min | Act with approval | Keep, with a better tag guide |
| Calendar changes | 40 min | 35 min | Draft | Not yet |
| Newsletter assembly | 2 hrs | 1 hr 10 min | Draft | Keep |
| Invoice and paperwork reminders | 30 min | 5 min | Act | Keep |
Total: about five and a half hours of work became roughly two and a quarter. The saving was real, but it wasn't the headline. The headline was where the time went: almost all of the remaining two hours was review. We swapped doing for checking, and checking is a different skill.
What we kept, and what we didn't
We kept the Monday inventory, permanently. It's the single best use of an agent we've found, because it reads boring systems tirelessly and reports contradictions instead of resolving them.
We kept draft-chasing and paperwork reminders. Low stakes, high repetition, easy to review.
We did not keep calendar changes. Time zones, vacations and the unwritten rules about which contributors hate early calls live in people's heads, not in our calendar. Until they live somewhere the agent can read, it will keep confidently proposing 2 a.m.
If you want to run this yourself
A checklist, earned the hard way:
- Write the permission levels down before you connect anything. Decide what "act" means for each task.
- Promote slowly. Three correct runs before moving up a level is a good default.
- Design review for confident errors. Assume the wrong answer will look exactly like the right one.
- Log everything, including the time you spend reviewing. Otherwise you'll overestimate the savings.
- Start with Read. An agent that only notices things is already worth having.
- Fix the source, not the symptom. Our tagging got better when we rewrote our tag guide, not when we rewrote the prompt.
The agent is still running our Monday inventory. This week it noticed that I'd left this very story marked "in edit" for three days. Fair.
Filed under
How this was made AI-assisted
AI-assisted. Written by a person with AI help for research, outlining or editing.
Dev used an AI assistant to compile the week's raw time log into the summary table; all reporting, analysis and writing are his.
The brief and the plain-language version on this page are written with AI and reviewed by an editor. The story itself is the source of truth.
Ask about this story
Conversation0
No comments yet.
Start the conversation — the first reply sets the tone.