A scheduled job can fail in two ways that both read as success. The same task can run twice for one day, when it fires from two machines, and the message goes out twice. Or it can fail silently: a parse error returns an empty result that looks exactly like “nothing to report.” The reflex is to trust the run’s exit status, and that reflex is the trap.
I run one automation that shows the first risk. The dog-walk scheduler runs from two places at once, a Mac woken on an hourly launchd catch-up and a server firing at 06:00 on cron; both could cover the same day, and the walk window’s room would get the message twice.
The ground came first: one pipeline per job, with a single daily entry point. My monitor digest runs once at 07:00 CEST every day and gates its weekly and monthly work inside itself, on Monday and on the first of the month, rather than one cron entry per check. The trio below keeps it from reporting a success it did not earn, the operating layer of Reliable agent systems. Moving a WordPress site’s own scheduler onto a system crontab is its own decision.
TL;DR
Three disciplines keep a daily pipeline honest, over one pipeline per job with a single daily entry point: a per-machine idempotency sentinel so a day’s run cannot happen twice, fail-loud erroring instead of silent empty output, and a mandatory review pass on every reader-facing stage.
The three disciplines, as decisions
Each is a decision with a live alternative, and together they cover the two ways a daily run lies about itself.
The first is a per-machine idempotency sentinel. A run checks for a file that marks today and exits if it is there. The pipeline writes that file itself, one per machine, so two machines never coordinate. I rejected a lock file plus a database state machine because the pipeline is otherwise stateless. The calendar read and the scheduling slots behind the dog-walk automation are a different register.
The second is fail loud. A failed JSON parse on the model response must raise a descriptive error rather than return an empty result. Silently returning empty output, the rejected alternative, is what the discipline rules out: a result a caller cannot tell apart from a run that found nothing. The failure is non-fatal here, and that is why raising it matters.
The third is a mandatory review pass on every reader-facing stage. It runs on every stage a reader sees, always, never conditional on a heuristic. Its statement, its one exemption, and the detect-then-fix alternative it rejected are not mine to restate; the review pass that makes AI copy publishable carries them. Here it is the third leg, a gate no heuristic can skip.
How the three disciplines work
The order of operations is what makes them work.
- The idempotency sentinel.
already_ran_today()sits at the head of the run: it looks for today’sdog-walk-{YYYYMMDD}-*.mdmarker in the output directory, and the run exits before any work if it finds one. The marker is written after delivery returns and before the pipeline’s own success-or-failure branch. A hard failure upstream of delivery writes none, so the hourly catch-up retries the whole pipeline; a delivery-failed run still writes the sentinel, so no same-day retry fires. - The fail-loud rule. It is a raise, not a return: on a bad parse the code raises a descriptive error and the caller decides what to do. It surfaces the failure; it does not stop the pipeline.
- The review pass. Every reader-facing stage is gated by a post-generation review that always runs, a stage gate and not a spot check.
| A failure that reads as success | The discipline that catches it |
|---|---|
| The same job runs twice for one day | The per-machine idempotency sentinel |
| A failed parse returns empty output | Fail-loud erroring |
| Poor generated text reaches the reader | The mandatory review pass |
One entry point at a fixed hour is the ground under all three: the weekly and monthly work runs inside the same pipeline, so it inherits the same guards.
Nothing broke
Every record behind this piece reports that nothing broke. I will not invent an incident to fill it: the disciplines went in as prevention, not repair for a failure. What I can report is the pathway each guards, the two failure classes above.
One honest piece of friction lives in the fail-loud rule: the raise is non-fatal at the pipeline level, so the report still builds while the failure stays on record.
What it costs to run
I have no running-cost figure and will not invent one; what the record carries is maintenance cost.
The sentinel file is the concrete one: the pipeline writes one small file a day, and it doubles as post-hoc verification of what the run saw, the formatted message and the raw forecast JSON beside it. The review pass is the other cost, and the record calls it load-bearing, not a nicety: the result is reliable through the pass, not through the generation prompt.
What this does not solve
Three boundaries, drawn plainly, because a discipline that is oversold stops being useful.
First, the disciplines cover a pipeline’s liveness and honesty, not the content of its output. A sentinel cannot tell whether a line is grounded; a review pass reviews the text, it does not validate the fact. Grounding is a deterministic guard of its own, on the reader-facing lines rather than the run: Stop hallucinations with deterministic guards. The shape of the output is a third mechanism, deciding what the model produces and what code renders.
Second, fail loud surfaces a failure; it does not halt the pipeline. When the pipeline judges a failure non-fatal, the report still builds and the failed part is left out, not stopped. The discipline is honesty about the failure, not a guarantee the failed work never ships.
Third, the sentinel makes a same-day re-run safe by the day, not by the payload: it stops the run repeating, it does not deduplicate what it sends. This is application-level operation, not autonomy: the human stays in the chain, and a stage whose output no reader sees is outside the review pass’s reach. Keeping a person accountable for what ships is its own pattern: Separating spec, build and QA, and so is the boundary confining what an automation may reach: The security layer is the OS boundary, not agent guardrails. Gating a service reload behind a configuration check is a gate of a different kind, and a backup whose notification fails while the run reports green is a silent success of this kind, told from the backup side.
The pattern I would start from is the trio above: one entry point for each day, a run that marks its own day, a failure that raises instead of hiding, and a review pass in front of every reader-facing stage. Every record behind this piece is partial, so I describe the design and claim no measured rate. The operator behind this method is on the about page. If this is the shape of your problem, let’s talk.