What is about to happen. The news collector worker on stage and prod pays Scrape.do or Firecrawl for every news page fetch. Today a page with no tier yet can be fetched again by each actor that shares it, minutes apart. This PR makes such a page wait 6 hours after a fetch of it starts, for every actor. The code change is one function, plus two tests. A revert undoes it, and no stored data changes.
Review artifact: PR review. Reviewed 2026-11-17 by Claude (review-artifact skill), a different agent session from the author. PR head e8cdd15a9 is one commit on top of the stage tip 1b77564e1. I read the code at the PR head, ran both test files, and ran the new integration test against the old isPageDue. I read PR 954 for the origin. I did not query any database and did not run the worker. A second agent did an independent pass without the review skills; its notes are in 955.independent-pass.md.
Use → / space / swipe to move.
1. Update NEWS_PAGES.md §4.1 in a follow-up docs PR?
Recommended: Yes, a one-line follow-up PR.
If no: A reader of §4.1 may think a failed page is retried by the next actor. They may change the code back to match the doc.
Options: Yes, a one-line follow-up PR · Yes, add it to this PR · No change
2. Add one sentence to the COOLDOWN_HOURS comment before merge?
Recommended: Yes, one sentence in the config comment.
If no: Someone tunes the actor cooldown and the double fetches come back with no warning.
Options: Yes, one sentence in the config comment · Yes, a separate page retry setting · No change
apps/api-server/src/services/signals/NewsCollector.ts:369docs/signals/NEWS_PAGES.mdnone never. A page gets its first tier at the end of its first fetch that returns the page, from the number of articles it lists. apps/api-server/src/services/signals/actors/newsPages.ts:167bucket is null). Either nobody fetched it, its first fetch is still running, or its fetches failed.lastFetchedAt only if the value is still the one it read. If another run changed it first, the write matches nothing and the page is skipped. apps/api-server/src/services/signals/NewsCollector.ts:795NEWS_COLLECTOR_COOLDOWN_HOURS, default 6. Until this PR it only meant the time before a finished actor is taken again. With this PR, an untiered page also waits this many hours, counted from the start of its last fetch. apps/api-server/src/config/news-collector.config.ts:15none.apps/api-server/src/config/news-collector.config.ts:20“Every extra fetch is a wasted Scrape.do/Firecrawl call.” — PR 955 description
“A page shared by ten actors is fetched once, in whichever of their passes comes first” — docs/signals/NEWS_PAGES.md §4 item 2
“a page that has no tier yet stays due, so every actor sharing it can fetch it again in the same pass. finance.yahoo.com and cnbc were fetched 3 times, autonews twice; in the failed first pass, pages were fetched up to 5 times.” — PR 954, "Bug found (not fixed here)", from the pages-only run on preview newspages-2026-11-17
Level 1 is the most general. A “no” at one level makes everything under it moot.
Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.
The claim already blocks two runs that read the page at the same moment. It fails when a second run reads the page after the first one claimed it, because the page still looks due.
isPageDue, so an untiered page that was fetched recently is not due.collectPages, so it also requires the page to be tiered or its lastFetchedAt to be old.Taken: A. The predicate is where "due" is decided, and its only caller is collectPages. One line changes and the claim keeps working as before.
bucket null and lastFetchedAt set.Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.
NEWS_PAGES.md §4.1 says a page that could not be fetched "is due again next pass", meaning the next actor run. An actor runs again at least 6 hours after its last run finished.
NEWS_COLLECTOR_COOLDOWN_HOURS (6 hours).NEWS_COLLECTOR_PAGE_RETRY_HOURS.Taken: A. 6 hours is also the least time between two runs of the same actor, so the §4.1 line "due again next pass" still holds for that actor. Other actors that share the page wait until the 6 hours pass.
Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.
Taken: A. The user asked for no docs change. The PR description says so.
Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.
isPageDue with no tier, and an integration test: two actors share a page that fails, run one after the other.Taken: A. The "still running" and "failed" cases go through the same branch of isPageDue. The unit test covers that branch.
Basis: ran = a command, test or query I ran; read = code I read; quoted = words from a PR, issue or report.
Yes. After the first run claims a page, the page has a new lastFetchedAt and still no tier. The next actor reads it as due, and its claim matches the new value, so it fetches again. On the old code the test sees 2 fetches.
How: Read PR 954 and the claim code. Ran the new integration test against the old isPageDue. Did not look at the preview data. Skill pass.
if (!page.bucket || !page.lastFetchedAt) return page.bucket?.tier !== 'none' makes every untiered page due.lastFetchedAt just read.The written origin is an agent's bug note in PR 954. The PR 955 text says the user asked for no docs change, so a person did steer it. The fix saves paid calls and changes no data, so the risk of an unwanted fix is low.
How: Read PR 954 and PR 955. I could not see the Slack thread of the run. Skill pass.
Yes to both. It fails on the old code and passes on the PR head. It uses the integration tier, not mocks of models. The fake fetcher now returns null (a failed fetch) for an empty body; no other test passes an empty body, so nothing else changes.
How: Ran the integration test with stage's isPageDue copied into a scratch worktree, then removed the worktree. Skill pass and independent pass.
For the same actor, yes. The page wait starts when its run claims the page. The actor cooldown starts when that run finishes, which is later. So when the actor's next run starts, more than 6 hours have passed since the claim, and the page is due again. Other actors that share the page now skip it until 6 hours pass. That is the point of the fix.
How: Compared the page wait with the actor cooldown in claimBatch. Code read only. Skill pass and independent pass.
now when page collection starts; :797 writes it to lastFetchedAt.newsCollectionFinished, written at the end of the run.Yes. While the fetch runs, the page has lastFetchedAt set and no tier, so another actor sees it as not due. When the fetch ends, the page gets tier A, B or C (1, 3 or 7 days), or none. This race caused the 2 extra fetches of the pages that PR 954 fetched with success 3 times.
How: Read the code path. The unit test covers the branch. No integration test runs two actors at the same time. Skill pass and independent pass.
fetchPage.isPageDue({ bucket: null, lastFetchedAt: now }, now) is false.none? read · free-formTier none is unchanged: the Mongo filter still drops it and the predicate still returns false. An own blog has one actor, and that actor runs at most every 6 hours, so it sees no change in normal runs. With this PR, an operator pull or a lease takeover inside the 6 hours no longer retries a failed own blog.
How: Code read. Skill pass.
bucket.tier: { $ne: 'none' } also matches pages with no tier.none still returns false for a fetched page.Each now waits up to 6 hours before it retries an untiered page. A worker that dies mid-fetch is taken over after 90 seconds, and the new run skips that page. An operator pull of the actor skips it too. Before, both retried it at once. This is a small cost for a rare case. Tiered pages already worked this way.
How: Read fetchPage, probePage, claimBatch and the lease setting. Did not run them. Both passes found this.
fetchPage records "The page could not be fetched".collectPages does not get claim.forced, so a pull does not force pages.none for good? read · free-formYes, today, but rarely. If the run aborts during the article links, the code still sets the first tier from the links it got through. If fewer than 2 articles got through, the page gets tier none, and a none page is never fetched again. It is not in this PR's scope. It needs its own ticket.
How: Read fetchPage. This is not changed by the PR. Found by the independent pass; I checked it.
§4.1 says a page that could not be fetched "is due again next pass". That is still true for the same actor, but the doc does not say that other actors wait 6 hours. §4 item 2 ("fetched once per interval") now also holds for untiered pages. The gap was left on purpose, as the user asked.
How: Read §4 and §4.1 at the PR head. Both passes found this.
Today it works, but by luck of the default. The config comment says the setting is for finished actors only. Someone who sets it to 0 to run actors back to back also switches this fix off, and the new unit test then fails.
How: Read the config and the new unit test. Did not check the deployed env values. Found by the independent pass; I checked it.
lastFetchedAt: now gives true, and the test expects false.It lowers them: a shared untiered page is fetched at most once per 6 hours instead of once per actor. A page that always fails has no failure count and is still retried every 6 hours, about 4 times a day. That is more often than tier A. It was worse before, so it is a follow-up idea, not a fault of this PR.
How: Code read. Did not measure on a preview. Both passes.
lastError; there is no failure count on the page.Yes. "This test fails without the fix" is true. The unit test checks the page at 0 hours, at 5 hours and at exactly 6 hours after the fetch. The PR does not claim a rerun on real data, and there is none.
How: Ran both test files and the old-code check myself.
Both passes are Claude, as is the author, so their errors may be alike. The change is one function, and its test fails on the old code. A short human read of isPageDue is enough.
How: Two passes by Claude agents: one with the review skills, one without. Same model family as the author.
No. It touches one service file and two tests. "page mirror: not affected" is right.
How: Read the diff file list.
isPageDue.NEWS_COLLECTOR_COOLDOWN_HOURS. With 0 the fix does nothing.COOLDOWN_HOURS comment in news-collector.config.ts that the page retry wait uses it too.none. Such a page is never fetched again.