An automated digest has one promise its subscriber takes for granted: it will not serve the same thing twice. Mine keeps that promise for the article and breaks it for the story. When a second source covers the same subject, or an earlier digest already treated it, the reader still receives it as new. The check I shipped keys on the article’s identity, never on the story it carries. This sits in Reliable agent systems.
TL;DR
- Article-level dedup ships. Two passes catch a repeat: a merge on the resolved canonical URL, and a title fallback that merges inside a batch and skips a title served in the previous seven days.
- Subject-level dedup is decided, not built. Merging two different sources that cover one story, and skipping a story an earlier digest treated, are decisions with no shipped code behind them.
- Nothing broke. All five records report no failure, so the friction is a gap between what the digest is assumed to catch and what it catches.
The repeat the digest catches, and the one it cannot
The digest must not serve the same item twice, and it must catch a repeat without resurfacing something the reader already has. The obvious fix is to deduplicate on the article’s URL. Two constraints rule that out. One article can reach the digest under different marketing redirects across two newsletters, and resolving the address to its canonical form can be blocked by an anti-bot challenge, so the canonical URL is sometimes unavailable. A check keyed on an exact address either misses the copy or fails outright when it will not resolve.
That distinction also separates a repeat from a re-run. A sentinel that stops a scheduled job happening twice says nothing about what that job sends.
Two passes, both keyed on the article
The design that ships is a two-pass deduplication. Pass A merges items that share the resolved canonical URL, so cross-newsletter copies of one article collapse into a single item. The resolver runs before that merge, so it keys on the real address rather than the redirect that delivered it.
Pass B is the title fallback, and it exists because of that blocked-resolution constraint. It carries two sub-cases: an in-session merge when two solo items in the current batch share a normalized title but different URLs, and a cross-day check that skips insertion when that headline was served inside the previous seven days.
Two alternatives were live and neither survived. Exact-URL dedup alone failed on a blocked resolution. A fallback keyed on the title alone failed to merge the cross-source canonical copies. The wire is the same in both passes: what ships keys on the article, not on the subject it reports.
How the check runs
Normalization is deliberately plain. The title is lowercased and its whitespace is collapsed before comparison, so two spellings of the same headline meet. The cross-day check reads the recent days before each insert, and a headline already served inside the window is skipped at that moment rather than filtered later. A run the passes empty out is logged as empty; it is never delivered as new. The skips are reported as a count, so a quiet stretch can be read as a working check and not as a failed fetch.
The newspaper side runs a different mechanism on top. Its shipped check is a preference, not an exclusion: the scoring rules give the original Le Monde byline a boost over a wire version, so a partnership article does not outrank the original reporting it repeats. Which item a digest keeps when several are relevant is a selection posture of its own.
What broke: nothing, and why that is the finding
All five records behind this piece report that nothing broke. No run failed, no duplicate shipped by mistake, and no incident is reconstructed here to fill the space. The friction is a gap in expectation, not a failure in the check. A digest read as no repeats guarantees only that no article repeats. Its subscriber hears a promise the architecture never made, and the check that would keep the larger promise is decided but not built.
The guard that decides whether a line is grounded at all is a separate mechanism; this one decides only whether the line was already sent.
What it costs to run
No cost figure is recorded for either dedup pass, so I state none. Neither is priced in the records, and a number invented here would be exactly the over-claim the rest of this piece argues against. Both run inside the digest’s existing pipeline, at the point where it decides what to insert, rather than as a separate job with its own budget.
What this does not solve
The shipped check stops at the article. A subject that arrives from a second source still produces two blocks, and a subject an earlier digest already treated still returns as new. Both are decided and neither is built. The newspaper digest carries two further ideas in the same state: excluding any article tagged as produced by a press agency, so a story served through an AFP or Reuters partnership does not sit beside the original, and a memory of past digests, so framing already used is dropped and only the new elements show.
What the architecture separates, then, is a repeat from a re-run, an article from a story, and a decision from a deployment. The rest of the pillar works the same boundary from other sides. The always-run review pass sits over the reader-facing stages; separating spec, build, and quality assurance holds the roles apart; and the operating-system boundary governs the machine underneath. None of them closes the subject-level gap either.
Two more decisions touch how the digest is run rather than what it deduplicates: the round’s own cost, and where the model actually runs. Neither is the article-level check restated, and neither reaches the story.
A digest like this needs no new tooling; it needs the article caught and the story named as a different problem. I describe how I run this stack on the about page. If you run a digest and cannot tell a repeat from a new story, tell me how yours handles it.