What is about to happen. The news collector worker on stage and prod pays Scrape.do or Firecrawl for every news page fetch. Today a page with no tier yet can be fetched again by each actor that shares it, minutes apart. This PR makes such a page wait 6 hours after a fetch of it starts, for every actor. The code change is one function, plus two tests. A revert undoes it, and no stored data changes.
Review artifact: task premise. Reviewed 2026-11-17 by Claude (review-artifact skill), a different agent session from the author. PR head e8cdd15a9 is one commit on top of the stage tip 1b77564e1. I read the code at the PR head, ran both test files, and ran the new integration test against the old isPageDue. I read PR 954 for the origin. I did not query any database and did not run the worker. A second agent did an independent pass without the review skills; its notes are in 955.independent-pass.md.
Use → / space / swipe to move.
docs/signals/NEWS_PAGES.mdnone never. A page gets its first tier at the end of its first fetch that returns the page, from the number of articles it lists. apps/api-server/src/services/signals/actors/newsPages.ts:167bucket is null). Either nobody fetched it, its first fetch is still running, or its fetches failed.lastFetchedAt only if the value is still the one it read. If another run changed it first, the write matches nothing and the page is skipped. apps/api-server/src/services/signals/NewsCollector.ts:795NEWS_COLLECTOR_COOLDOWN_HOURS, default 6. Until this PR it only meant the time before a finished actor is taken again. With this PR, an untiered page also waits this many hours, counted from the start of its last fetch. apps/api-server/src/config/news-collector.config.ts:15“Every extra fetch is a wasted Scrape.do/Firecrawl call.” — PR 955 description
“A page shared by ten actors is fetched once, in whichever of their passes comes first” — docs/signals/NEWS_PAGES.md §4 item 2
“a page that has no tier yet stays due, so every actor sharing it can fetch it again in the same pass. finance.yahoo.com and cnbc were fetched 3 times, autonews twice; in the failed first pass, pages were fetched up to 5 times.” — PR 954, "Bug found (not fixed here)", from the pages-only run on preview newspages-2026-11-17
Level 1 is the most general. A “no” at one level makes everything under it moot.
Basis: ran = a command, test or query I ran; read = code I read; quoted = words from a PR, issue or report.
Yes. After the first run claims a page, the page has a new lastFetchedAt and still no tier. The next actor reads it as due, and its claim matches the new value, so it fetches again. On the old code the test sees 2 fetches.
How: Read PR 954 and the claim code. Ran the new integration test against the old isPageDue. Did not look at the preview data. Skill pass.
if (!page.bucket || !page.lastFetchedAt) return page.bucket?.tier !== 'none' makes every untiered page due.lastFetchedAt just read.The written origin is an agent's bug note in PR 954. The PR 955 text says the user asked for no docs change, so a person did steer it. The fix saves paid calls and changes no data, so the risk of an unwanted fix is low.
How: Read PR 954 and PR 955. I could not see the Slack thread of the run. Skill pass.
Yes. While the fetch runs, the page has lastFetchedAt set and no tier, so another actor sees it as not due. When the fetch ends, the page gets tier A, B or C (1, 3 or 7 days), or none. This race caused the 2 extra fetches of the pages that PR 954 fetched with success 3 times.
How: Read the code path. The unit test covers the branch. No integration test runs two actors at the same time. Skill pass and independent pass.
fetchPage.isPageDue({ bucket: null, lastFetchedAt: now }, now) is false.It lowers them: a shared untiered page is fetched at most once per 6 hours instead of once per actor. A page that always fails has no failure count and is still retried every 6 hours, about 4 times a day. That is more often than tier A. It was worse before, so it is a follow-up idea, not a fault of this PR.
How: Code read. Did not measure on a preview. Both passes.
lastError; there is no failure count on the page.isPageDue.NEWS_COLLECTOR_COOLDOWN_HOURS. With 0 the fix does nothing.