Visibility is invisible until it is lost, and a book of clients makes that loss quieter than one business. One owner notices late; an agency across the book notices later still, and across more accounts at once. The instrument I keep against that keeps each account’s chatbot visibility observable: an account slipping out of a chatbot’s answer becomes a fact the monitor surfaces, not one the owner reports. That shift is the conviction behind this pillar; this is the machine that watches it, inside Found by AI.
TL;DR
One brand-agnostic engine, a folder per client, and a probe pass paid once and cached. The per-client report, its recommendations, and the delivery summary rebuild from that cache without re-paying the probe. A composite score is then corrected by two overrides the system applies, and a recommendation-engine failure degrades the run to a warning instead of losing the report.
Why re-running the pipeline is the wrong fix
The obvious answer when a report comes back wrong is to run the pipeline again. The cost of the probe pass rules that out: AI probing alone takes ~75-90 minutes for every (query, model) pair in QUERY_MATRIX x LLM_TARGETS (6 models x N queries), so re-running to fix a report, a recommendation, or a delivery bug is prohibitively expensive.
The stack’s other automations add the second constraint: single-tenant, built for one account, so none can carry a book. The monitor had to be multi-client from the start.
One engine, a folder per client, and the module that breaks the promise
The decision was to split the routine rather than extend it. Everything brand-agnostic went into visibility/engine/ (gsc_collector.py, ai_probe.py, report_builder.py, recommendation_engine.py), each runnable standalone; everything client-specific went into per-client folders (visibility/monitor_localSEO/: a gitignored config.py, a version-controlled config.example.py, pipeline.py, db.py). A second client means a new visibility/CLIENT_FOLDER/ reusing the same engine/ modules unchanged.
That promise does not hold everywhere, and I state the break plainly. The reuse promise fails for report_builder.py: its layer labels and its city, department, and region names are hardcoded in the file rather than derived from config.py, so onboarding a second client requires editing that module’s strings. The gap is tracked in BACKLOG.md.
The probe calls the provider directly, outside the router
Probing is the one routine outside the shared router, and the reason is availability: the cluster of US commercial chat models the probe needs answers only against OpenRouter, because the router’s primary account does not carry these models. So engine/ai_probe.py calls OpenRouter directly for the 6-target pass, building its own OpenAI(base_url=...) client plus a raw requests.post() path to override a target provider, and addressing each by its raw slug rather than its logical name. Those providers are authorized on the OpenRouter account only for this routine.
Scoring and recommendations were once direct too; as of the 2026-08-11 migration they go through the shared primary-and-fallback router (libs/llm_router.py, libs/llm_registry.py, and chat_completion()), and both roles now have a real fallback model. Routing it through that chain was the live alternative, rejected because the primary account does not carry these models, so probing has no chain to join. The probe pays for direct access; everything downstream takes the router. That choice is the opposite of a general routing heuristic.
Pay for the probe once, rebuild the report for free
The probe result and the Google Search Console data are cached to .cache/last_report_data.json immediately after the probing step, and a --report-only mode rebuilds the report, the recommendations, and the delivery summary from that cache, with no Google Search Console cost and no LLM cost.
The precondition is deliberate. --report-only requires a prior successful probing run and raises on a missing cache file. It is a rebuild path, not a way to report without ever probing: a fix to report_builder.py or recommendation_engine.py can be re-verified in seconds rather than re-paying the 75-90 minute probe pass.
The composite score, and the overrides the system applies
A composite score decides how a client reads in a chatbot’s answer; two overrides sit after it. The case it cannot handle alone is a particular kind of wrong answer: a model that denies the entity exists should not score normally, and neither should a model that names the entity correctly but places it in the wrong city.
| Score or override | What it sets |
|---|---|
presence_score | 0-2 |
sentiment_score | 0-3 |
specificity_score | 0-3 |
| composite (the three summed) | 0-8 |
| denial detected | forces the composite to 0/8, with no further scoring |
| wrong location detected | caps the composite at 3/8 |
Both overrides are applied by the system and not by the model, and they are mutually exclusive: denial is checked first and short-circuits, and the wrong-location cap applies only when the entity is correctly named but geographically contradicted. A specific kind of answer gets a specific fixed score, whatever the model-derived sub-scores say. A deterministic guard against an ungrounded line is a different layer, and the panel median a per-chatbot rule is switched on against stays with its own register.
When the recommendation engine fails, the report still ships
Recommendations are the optional part of the run, and the pipeline is built so their failure cannot take the report with it. If recommendation_engine.py raises, report_builder.run() collects the failure into a warnings list and the delivered alert is prefixed with a warning (recs_failed=True); the report is still built and delivered, and the pipeline never aborts solely because recommendations failed. Aborting the run outright was the alternative, rejected so the output and the rest of the audit would not be lost.
A recommendation engine that fails degrades the run; it does not end it. Recommendations are an optional enrichment; their failure surfaces as a visible warning in the delivered alert, not a pipeline failure. That posture toward a failed daily automation is its own discipline.
What this does not solve
The design is complete; the evidence behind it is not.
- The five records are partial. Every record behind this piece is narrated from the wiki page rather than read off a run, so I state the design and its recorded status, never a measured visibility outcome or a success rate. The 75-90 minute figure is the record’s own estimate, not a measured run.
- The cache path needs a prior run.
--report-onlyraises on a missing cache file, so it rebuilds from a successful probe; it does not recover a run whose probe failed. - Degradation is a warning, not a fix. A recommendation-engine failure leaves a visible warning on the delivered alert; it repairs nothing.
- The overrides are scoring, not assurance. They correct how a denial or a wrong-city answer is scored; they do not make a client appear.
What the monitor gives the agency across the book is narrower than an outcome claim and more useful than a dashboard: each account stays observable, and each report stays inside its owner’s reach. Deliverability is the rule that keeps that output usable: a recommendation the non-technical owner cannot carry out fails quietly through the agency. The operator behind this stack is on the about page. If you run local visibility across a book of clients and want to judge a monitor by how each account actually gets seen, let’s talk.