Never make the model do the counting

The failure that taught me this was not dramatic, which is why it took a while to see. I had built a small tool for pausing work mid-task, one that writes a handoff document and keeps a list of the projects currently parked, and it worked, in the sense that it produced a handoff every time I asked for one. What it did not do was produce the same shape of handoff twice. The filename convention drifted over a fortnight, the timestamp format wandered, the list of parked projects came back numbered differently depending on how the model felt about ordering that day, and once, memorably, the count at the top of the list disagreed with the number of items underneath it.

None of that is the model being bad at its job, it is the model being asked to do the wrong job entirely. Every one of those drifting details was a mechanical fact with exactly one right answer, and I had put all of them in prose, in a skill file, which is to say I had written a specification and asked something non-deterministic to re-implement it from scratch on every single run. It gets the specification right most times, and most is a rather worse guarantee than it sounds when the failures are all silent.

So the rule I ended up with is blunt, and it has survived every tool I have built since. The model must never do arithmetic, dates, or counting. If a run's output contains a number, a script produced that number, not the model. Anything with one correct answer belongs in code, and the model is left with the only thing it is actually better at than a script, which is judgement.

In practice that means every skill of mine that holds state is now two things instead of one. There is a command line tool, a single file of plain Node with no dependencies at all, which owns the timestamps, the filenames, the slugs, reading and writing the store, deduplicating it, the date arithmetic, the git commits, and the exact format of anything a human reads. Then there is a much smaller instruction file that owns the judgement, which in this case means writing the actual prose of the handoff, deciding where the genuine stopping point was, and working out what the single next action ought to be. The instruction file is a router, mostly a table mapping what I might say to the command it should run, with a note against the mechanical rows telling it plainly to run the command and relay the output, not reason about the result. Without that note the model helpfully recomputes what the script just worked out, and then you have two answers again.

The rules underneath that split are all rather small and all learned by getting them wrong first. The tool takes zero dependencies, node builtins only, and does its own terminal colour with raw escape codes instead of pulling in a library, because a skill that needs an install step is a skill that breaks on a fresh machine. It resolves where its state lives explicitly and fails loudly when it cannot, never quietly defaulting to the current directory, which would scatter state into whichever folder a session happened to be sitting in. Mutations take a stable key rather than the number printed in the list, because those numbers re-sort as items move and a stale one silently targets the wrong project, which is the exact class of bug that never announces itself.

The rule I would put above the others, though, is that a failure must never look like a success. My tool swallows one specific git error, the benign case where there was nothing to commit because the content was already identical, and on any other git failure it prints that the changes are on disk but uncommitted rather than exiting zero. That distinction matters rather more than it appears to, because a script that reports success on a failed write is teaching the model to trust a lie, and the model has no way of catching it. Everything the skill needs to react to gets emitted as a single uppercase marker word rather than a sentence, so I can reword a message freely without breaking the thing that reads it.

Then there is the honest cost argument, which is what actually made me do the work instead of merely agreeing with it. Skill instructions load into the context every time the skill fires, so a wordy skill is a bill you pay per invocation forever, and mine had grown to around two and a half thousand words. Slimming it to the common path, moving the rarely-used procedures into separate files that only load when their trigger fires, and pushing every mechanical step into the script took a routine invocation to roughly a fifth of what it had been costing, with no behaviour lost at all. One idea got rejected along the way and I think the reasoning generalises, which is that I considered handing the handoff writing to a cheap subagent and did not, because the writing was only about six hundred tokens to begin with and a subagent cannot see the conversation, so feeding it enough context to do the job would have cost more than doing the job. The expensive part was never the writing, it was the loading.

The tests follow from the same logic and there are not many of them. One file, plain assertions, no framework, and the only things it covers are the ones that can be wrong without throwing an error: the store writes and reads back identically, a rendered file parses back to what produced it, the date arithmetic survives a month boundary, the deduplication turns two copies of the same record into one, and every threshold gets checked at one below, exactly on, and one above. A one-line passthrough gets no test, because a test for that is scaffolding rather than safety.

It would be rather a mistake to read all of this as an argument for putting everything in code, and I want to be clear about where the line falls, because I have watched myself get it wrong in the other direction too. This pattern only pays off when a skill holds state across sessions, writes files, commits, or counts things. A skill that is purely judgement, a review checklist or a writing-style guide or a discipline I want a session to hold to, gets nothing from a command line tool at all, and building one for it is scaffolding around an empty space. The gate is state and determinism, not how sophisticated the thing feels.

Two traps are worth passing on, both of which cost me an evening. The tool reads an environment variable to find its state directory, and that variable is exported globally on my machine, so any casual invocation writes to the live store no matter which directory I ran it from. Changing directory does not protect you, and I now override that variable per invocation whenever I am testing anything. The other is rather subtler: a change to the script takes effect on the very next run, whereas a change to the instruction file needs a fresh session to be picked up, so when a fix appears to do nothing the first question is not whether the fix was wrong, it is which half of the thing I edited.

The part I did not expect is that the tool turned out rather nicer to use as a tool. Because all the mechanics live in a script rather than in a conversation, I can query the state of my parked work straight from a terminal with no session open and no tokens spent at all, which was not the goal and is now something I do several times a day. That turns out to be a decent test of whether the split is in the right place. If a skill cannot be driven by hand from a shell, the mechanics are still sitting in prose somewhere, and the model is quietly re-deriving them for you on every run, slightly differently each time.

Nathan Wong is in Singapore, building AI automation and low-code tooling, most of it self-hosted on an Unraid server called Obelisk. Portfolio