#955 · task premise 1 / 6
1 · Review guide · #955

fix(signals): fetch a shared news page once per pass until it is tiered

What is about to happen. The news collector worker on stage and prod pays Scrape.do or Firecrawl for every news page fetch. Today a page with no tier yet can be fetched again by each actor that shares it, minutes apart. This PR makes such a page wait 6 hours after a fetch of it starts, for every actor. The code change is one function, plus two tests. A revert undoes it, and no stored data changes.

Review artifact: task premise. Reviewed 2026-11-17 by Claude (review-artifact skill), a different agent session from the author. PR head e8cdd15a9 is one commit on top of the stage tip 1b77564e1. I read the code at the PR head, ran both test files, and ran the new integration test against the old isPageDue. I read PR 954 for the origin. I did not query any database and did not run the worker. A second agent did an independent pass without the review skills; its notes are in 955.independent-pass.md.

Use → / space / swipe to move.

2 · Words used

Words this guide uses

actor
A company, person or body the collector gathers news for. Each actor has its own collection run.
pass
The source texts use "pass" two ways. NEWS_PAGES.md means one actor's collection run. PR 954 and 955 mean one round of the worker: it takes up to 10 actors and runs 4 at once. This review says "actor run" and "worker round".
news page
A listing page on a website, such as a blog or a news section. The collector fetches it and follows its article links. docs/signals/NEWS_PAGES.md
tier
How often a page is fetched: A every day, B every 3 days, C every 7 days, none never. A page gets its first tier at the end of its first fetch that returns the page, from the number of articles it lists. apps/api-server/src/services/signals/actors/newsPages.ts:167
untiered page
A page with no tier yet (bucket is null). Either nobody fetched it, its first fetch is still running, or its fetches failed.
claim
How an actor run takes a page: it writes a new lastFetchedAt only if the value is still the one it read. If another run changed it first, the write matches nothing and the page is skipped. apps/api-server/src/services/signals/NewsCollector.ts:795
cooldown
NEWS_COLLECTOR_COOLDOWN_HOURS, default 6. Until this PR it only meant the time before a finished actor is taken again. With this PR, an untiered page also waits this many hours, counted from the start of its last fetch. apps/api-server/src/config/news-collector.config.ts:15
3 · Why this exists

The premises, general to specific

  1. Each page fetch costs a paid scraper call, so the collector must not fetch the same page twice when once is enough.
    “Every extra fetch is a wasted Scrape.do/Firecrawl call.” — PR 955 description
  2. A page shared by several actors is fetched once per interval, in whichever of their runs comes first.
    “A page shared by ten actors is fetched once, in whichever of their passes comes first” — docs/signals/NEWS_PAGES.md §4 item 2
  3. That rule must also hold for an untiered page, while its first fetch runs and after it fails.
    “a page that has no tier yet stays due, so every actor sharing it can fetch it again in the same pass. finance.yahoo.com and cnbc were fetched 3 times, autonews twice; in the failed first pass, pages were fetched up to 5 times.” — PR 954, "Bug found (not fixed here)", from the pages-only run on preview newspages-2026-11-17

Level 1 is the most general. A “no” at one level makes everything under it moot.

4 · What I checked

4 questions against the code

Basis: ran = a command, test or query I ran; read = code I read; quoted = words from a PR, issue or report.

holdsDid shared untiered pages really get fetched more than once? ran + read + quoted · review-premise/origin-trace

Yes. After the first run claims a page, the page has a new lastFetchedAt and still no tier. The next actor reads it as due, and its claim matches the new value, so it fetches again. On the old code the test sees 2 fetches.

How: Read PR 954 and the claim code. Ran the new integration test against the old isPageDue. Did not look at the preview data. Skill pass.

  • [read] stage newsPages.ts: if (!page.bucket || !page.lastFetchedAt) return page.bucket?.tier !== 'none' makes every untiered page due.
  • [read] NewsCollector.ts:795-797: the claim matches on the lastFetchedAt just read.
  • [ran] new integration test on the old code: 2 fetches of the listing, expected 1.
  • [quoted] PR 954: "finance.yahoo.com and cnbc were fetched 3 times, autonews twice".
unknownDid a person ask for this fix? quoted · review-simulator/H1

The written origin is an agent's bug note in PR 954. The PR 955 text says the user asked for no docs change, so a person did steer it. The fix saves paid calls and changes no data, so the risk of an unwanted fix is low.

How: Read PR 954 and PR 955. I could not see the Slack thread of the run. Skill pass.

  • [quoted] PR 954: "Bug found (not fixed here)".
  • [quoted] PR 955: "No docs change, as asked."
holdsDoes the fix also cover a first fetch that works but is still running? ran + read · free-form

Yes. While the fetch runs, the page has lastFetchedAt set and no tier, so another actor sees it as not due. When the fetch ends, the page gets tier A, B or C (1, 3 or 7 days), or none. This race caused the 2 extra fetches of the pages that PR 954 fetched with success 3 times.

How: Read the code path. The unit test covers the branch. No integration test runs two actors at the same time. Skill pass and independent pass.

  • [read] NewsCollector.ts:859-861: the tier is written only at the end of fetchPage.
  • [ran] pageLinks.test.ts: isPageDue({ bucket: null, lastFetchedAt: now }, now) is false.
holdsDoes the fix lower the paid calls, and is anything left that costs more than it should? read · review-async/llm-calls-and-cost

It lowers them: a shared untiered page is fetched at most once per 6 hours instead of once per actor. A page that always fails has no failure count and is still retried every 6 hours, about 4 times a day. That is more often than tier A. It was worse before, so it is a follow-up idea, not a fault of this PR.

How: Code read. Did not measure on a preview. Both passes.

  • [read] NewsCollector.ts:825-829: a failed fetch writes only lastError; there is no failure count on the page.
  • [read] news-collector.config.ts:62: TIER_A_DAYS is 1.
5 · Not checked, and what to do next

What this review did not do

  • I did not run the real worker on a preview with this fix. The PR has no rerun of the PR 954 worker round, so the "fetched once" result on real data is not shown.
  • I did not see the human words that asked for this PR (the Slack thread of the PR 954 run). The only written origin is the bug note in PR 954.
  • No test runs two actors at the same time. Both tests run them one after the other, so the "fetch still running" case is covered only by the unit test of isPageDue.
  • I did not check the prod or stage value of NEWS_COLLECTOR_COOLDOWN_HOURS. With 0 the fix does nothing.
6 · How to respond

React on the walkthrough, or reply in the PR

  • The walkthrough behind this guide: the YAML.
  • Or reply on #955, naming the decision number.