#955 · PR review 1 / 11
1 · Review guide · #955

fix(signals): fetch a shared news page once per pass until it is tiered

What is about to happen. The news collector worker on stage and prod pays Scrape.do or Firecrawl for every news page fetch. Today a page with no tier yet can be fetched again by each actor that shares it, minutes apart. This PR makes such a page wait 6 hours after a fetch of it starts, for every actor. The code change is one function, plus two tests. A revert undoes it, and no stored data changes.

Review artifact: PR review. Reviewed 2026-11-17 by Claude (review-artifact skill), a different agent session from the author. PR head e8cdd15a9 is one commit on top of the stage tip 1b77564e1. I read the code at the PR head, ran both test files, and ran the new integration test against the old isPageDue. I read PR 954 for the origin. I did not query any database and did not run the worker. A second agent did an independent pass without the review skills; its notes are in 955.independent-pass.md.

Use → / space / swipe to move.

2 · Your decisions

2 calls. Blocking ones first.

Judgement

1. Update NEWS_PAGES.md §4.1 in a follow-up docs PR?
Recommended: Yes, a one-line follow-up PR.

If no: A reader of §4.1 may think a failed page is retried by the next actor. They may change the code back to match the doc.

Options: Yes, a one-line follow-up PR · Yes, add it to this PR · No change

Judgement

2. Add one sentence to the COOLDOWN_HOURS comment before merge?
Recommended: Yes, one sentence in the config comment.

If no: Someone tunes the actor cooldown and the double fetches come back with no warning.

Options: Yes, one sentence in the config comment · Yes, a separate page retry setting · No change

3 · Words used

Words this guide uses

actor
A company, person or body the collector gathers news for. Each actor has its own collection run.
pass
The source texts use "pass" two ways. NEWS_PAGES.md means one actor's collection run. PR 954 and 955 mean one round of the worker: it takes up to 10 actors and runs 4 at once. This review says "actor run" and "worker round".
actor run
One collection run for one actor. An actor is not taken again until 6 hours after its last run finished. apps/api-server/src/services/signals/NewsCollector.ts:369
news page
A listing page on a website, such as a blog or a news section. The collector fetches it and follows its article links. docs/signals/NEWS_PAGES.md
shared page
A news page linked to more than one actor. Its articles are found for all of them from one fetch.
tier
How often a page is fetched: A every day, B every 3 days, C every 7 days, none never. A page gets its first tier at the end of its first fetch that returns the page, from the number of articles it lists. apps/api-server/src/services/signals/actors/newsPages.ts:167
untiered page
A page with no tier yet (bucket is null). Either nobody fetched it, its first fetch is still running, or its fetches failed.
claim
How an actor run takes a page: it writes a new lastFetchedAt only if the value is still the one it read. If another run changed it first, the write matches nothing and the page is skipped. apps/api-server/src/services/signals/NewsCollector.ts:795
cooldown
NEWS_COLLECTOR_COOLDOWN_HOURS, default 6. Until this PR it only meant the time before a finished actor is taken again. With this PR, an untiered page also waits this many hours, counted from the start of its last fetch. apps/api-server/src/config/news-collector.config.ts:15
pull now
An operator button that runs one actor at once, even inside its cooldown. It treats every search query as due. News pages keep their normal due rule.
own blog
A news page that is an actor's own blog. It has exactly one actor, its owner, and never falls to tier none.
page mirror
Healthboard code that rebuilds what some dashboard pages show. A PR that changes those pages must update it. This PR does not touch them.
lease takeover
When a worker dies, another worker takes over its actor after 90 seconds without a heartbeat. apps/api-server/src/config/news-collector.config.ts:20
4 · Why this exists

The premises, general to specific

  1. Each page fetch costs a paid scraper call, so the collector must not fetch the same page twice when once is enough.
    “Every extra fetch is a wasted Scrape.do/Firecrawl call.” — PR 955 description
  2. A page shared by several actors is fetched once per interval, in whichever of their runs comes first.
    “A page shared by ten actors is fetched once, in whichever of their passes comes first” — docs/signals/NEWS_PAGES.md §4 item 2
  3. That rule must also hold for an untiered page, while its first fetch runs and after it fails.
    “a page that has no tier yet stays due, so every actor sharing it can fetch it again in the same pass. finance.yahoo.com and cnbc were fetched 3 times, autonews twice; in the failed first pass, pages were fetched up to 5 times.” — PR 954, "Bug found (not fixed here)", from the pages-only run on preview newspages-2026-11-17

Level 1 is the most general. A “no” at one level makes everything under it moot.

5 · The path · decision 1 of 4

Where should the fix go?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

The claim already blocks two runs that read the page at the same moment. It fails when a second run reads the page after the first one claimed it, because the page still looks due.

AChange isPageDue, so an untiered page that was fetched recently is not due.
One function, pure and unit-testable.
The claim stays as it is.
The function now reads config, so it is less pure.
BChange the claim in collectPages, so it also requires the page to be tiered or its lastFetchedAt to be old.
The rule lives at the one place that writes.
A longer Mongo filter. The "due" rule is then split over two places.
CKeep a list of pages fetched in this worker process and skip them.
No change to stored data or rules.
Two worker processes still fetch the page twice.
It does not help a failed page in the next batch.

Taken: A. The predicate is where "due" is decided, and its only caller is collectPages. One line changes and the claim keeps working as before.

  • [read] PR 955 newsPages.ts:176-182: the new branch for bucket null and lastFetchedAt set.
  • [ran] grep isPageDue in apps and packages: only NewsCollector.ts and the two tests use it.
6 · The path · decision 2 of 4

How long should an untiered page wait after a fetch starts?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

NEWS_PAGES.md §4.1 says a page that could not be fetched "is due again next pass", meaning the next actor run. An actor runs again at least 6 hours after its last run finished.

AReuse NEWS_COLLECTOR_COOLDOWN_HOURS (6 hours).
No new setting.
The same actor still retries the page on its next run, so the doc line stays true for it.
One setting now means two things. Set to 0, it switches the fix off with no warning.
BAdd a new setting, for example NEWS_COLLECTOR_PAGE_RETRY_HOURS.
Each wait can be tuned on its own.
The config comment stays accurate.
One more setting to document and to keep in step with the actor cooldown.
CWait a whole tier C interval (7 days) after a failure.
A dead page costs far less.
A page that failed once in a short outage is not fetched for a week.
It contradicts the doc line.
DWait only while a fetch runs (a few minutes), then retry at once.
A failed page is retried by the next actor that shares it.
A failed shared page is still fetched once per actor.
PR 954 saw one page fetched up to 5 times this way.

Taken: A. 6 hours is also the least time between two runs of the same actor, so the §4.1 line "due again next pass" still holds for that actor. Other actors that share the page wait until the 6 hours pass.

  • [read] news-collector.config.ts:11-15: COOLDOWN_HOURS, default 6, documented for finished actors only.
  • [read] NewsCollector.ts:374: an actor is taken again only when it finished more than COOLDOWN_HOURS ago.
7 · The path · decision 3 of 4

Should this PR also update NEWS_PAGES.md?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

ALeave the docs as they are.
The user asked for no docs change.
A smaller diff.
§4.1 no longer says when a failed page is retried.
BChange one line of §4.1 to name the cooldown.
The doc and the code agree.
It goes against what the user asked.

Taken: A. The user asked for no docs change. The PR description says so.

  • [quoted] PR 955: "No docs change, as asked."
8 · The path · decision 4 of 4

How is the fix tested?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

AA unit test of isPageDue with no tier, and an integration test: two actors share a page that fails, run one after the other.
Real Mongo and the real claim.
The integration test fails on the old code.
No test with two actors running at the same time.
BAlso an integration test with two actors at the same time and a slow fetch.
It covers the "fetch still running" case end to end.
Timing tests can be flaky.

Taken: A. The "still running" and "failed" cases go through the same branch of isPageDue. The unit test covers that branch.

  • [ran] old isPageDue with the new integration test: fails, "expected [ …(2) ] to have a length of 1 but got 2".
  • [ran] PR head: pageLinks.test.ts 6/6 pass; collectorNewsPages.test.ts 4/4 pass.
9 · What I checked

14 questions against the code

Basis: ran = a command, test or query I ran; read = code I read; quoted = words from a PR, issue or report.

holdsDid shared untiered pages really get fetched more than once? ran + read + quoted · review-premise/origin-trace

Yes. After the first run claims a page, the page has a new lastFetchedAt and still no tier. The next actor reads it as due, and its claim matches the new value, so it fetches again. On the old code the test sees 2 fetches.

How: Read PR 954 and the claim code. Ran the new integration test against the old isPageDue. Did not look at the preview data. Skill pass.

  • [read] stage newsPages.ts: if (!page.bucket || !page.lastFetchedAt) return page.bucket?.tier !== 'none' makes every untiered page due.
  • [read] NewsCollector.ts:795-797: the claim matches on the lastFetchedAt just read.
  • [ran] new integration test on the old code: 2 fetches of the listing, expected 1.
  • [quoted] PR 954: "finance.yahoo.com and cnbc were fetched 3 times, autonews twice".
unknownDid a person ask for this fix? quoted · review-simulator/H1

The written origin is an agent's bug note in PR 954. The PR 955 text says the user asked for no docs change, so a person did steer it. The fix saves paid calls and changes no data, so the risk of an unwanted fix is low.

How: Read PR 954 and PR 955. I could not see the Slack thread of the run. Skill pass.

  • [quoted] PR 954: "Bug found (not fixed here)".
  • [quoted] PR 955: "No docs change, as asked."
holdsDoes the new test fail on the old code, and does it use real Mongo? ran + read · review-backend/tests

Yes to both. It fails on the old code and passes on the PR head. It uses the integration tier, not mocks of models. The fake fetcher now returns null (a failed fetch) for an empty body; no other test passes an empty body, so nothing else changes.

How: Ran the integration test with stage's isPageDue copied into a scratch worktree, then removed the worktree. Skill pass and independent pass.

  • [ran] old code: 1 failed, 3 passed. PR head: 4 passed.
  • [read] PR 955 collectorNewsPages.test.ts:30 and :150: the only other body is non-empty HTML.
holdsIs the 6-hour wait right, given the doc says a failed page "is due again next pass"? read · free-form

For the same actor, yes. The page wait starts when its run claims the page. The actor cooldown starts when that run finishes, which is later. So when the actor's next run starts, more than 6 hours have passed since the claim, and the page is due again. Other actors that share the page now skip it until 6 hours pass. That is the point of the fix.

How: Compared the page wait with the actor cooldown in claimBatch. Code read only. Skill pass and independent pass.

  • [read] NewsCollector.ts:783 sets now when page collection starts; :797 writes it to lastFetchedAt.
  • [read] NewsCollector.ts:374 and :763: the actor cooldown counts from newsCollectionFinished, written at the end of the run.
holdsDoes the fix also cover a first fetch that works but is still running? ran + read · free-form

Yes. While the fetch runs, the page has lastFetchedAt set and no tier, so another actor sees it as not due. When the fetch ends, the page gets tier A, B or C (1, 3 or 7 days), or none. This race caused the 2 extra fetches of the pages that PR 954 fetched with success 3 times.

How: Read the code path. The unit test covers the branch. No integration test runs two actors at the same time. Skill pass and independent pass.

  • [read] NewsCollector.ts:859-861: the tier is written only at the end of fetchPage.
  • [ran] pageLinks.test.ts: isPageDue({ bucket: null, lastFetchedAt: now }, now) is false.
holdsDoes anything change for own blogs or for pages at tier none? read · free-form

Tier none is unchanged: the Mongo filter still drops it and the predicate still returns false. An own blog has one actor, and that actor runs at most every 6 hours, so it sees no change in normal runs. With this PR, an operator pull or a lease takeover inside the 6 hours no longer retries a failed own blog.

How: Code read. Skill pass.

  • [read] NewsCollector.ts:785: filter bucket.tier: { $ne: 'none' } also matches pages with no tier.
  • [read] newsPages.ts:180: none still returns false for a fetched page.
holdsWhat happens on the abort, crash and "pull now" paths? read · review-async/safety

Each now waits up to 6 hours before it retries an untiered page. A worker that dies mid-fetch is taken over after 90 seconds, and the new run skips that page. An operator pull of the actor skips it too. Before, both retried it at once. This is a small cost for a rare case. Tiered pages already worked this way.

How: Read fetchPage, probePage, claimBatch and the lease setting. Did not run them. Both passes found this.

  • [read] pageProbe.ts:34-35: a fetch error or abort becomes null, and fetchPage records "The page could not be fetched".
  • [read] NewsCollector.ts:575: collectPages does not get claim.forced, so a pull does not force pages.
failsCan an aborted first fetch put a page at tier none for good? read · free-form

Yes, today, but rarely. If the run aborts during the article links, the code still sets the first tier from the links it got through. If fewer than 2 articles got through, the page gets tier none, and a none page is never fetched again. It is not in this PR's scope. It needs its own ticket.

How: Read fetchPage. This is not changed by the PR. Found by the independent pass; I checked it.

  • [read] NewsCollector.ts:839: the link loop returns early when aborted.
  • [read] NewsCollector.ts:859-861: the first tier is set after the loop with no abort check.
failsDoes NEWS_PAGES.md now disagree with the code? read · review-scope-defensibility/omissions · decision 1

§4.1 says a page that could not be fetched "is due again next pass". That is still true for the same actor, but the doc does not say that other actors wait 6 hours. §4 item 2 ("fetched once per interval") now also holds for untiered pages. The gap was left on purpose, as the user asked.

How: Read §4 and §4.1 at the PR head. Both passes found this.

  • [read] docs/signals/NEWS_PAGES.md:52: "A page that could not be fetched is not tiered and is due again next pass."
failsDoes reusing the actor cooldown setting make the fix fragile? read · free-form · decision 2

Today it works, but by luck of the default. The config comment says the setting is for finished actors only. Someone who sets it to 0 to run actors back to back also switches this fix off, and the new unit test then fails.

How: Read the config and the new unit test. Did not check the deployed env values. Found by the independent pass; I checked it.

  • [read] news-collector.config.ts:11-15: the comment names only actors.
  • [read] PR 955 pageLinks.test.ts: with a cooldown of 0, lastFetchedAt: now gives true, and the test expects false.
holdsDoes the fix lower the paid calls, and is anything left that costs more than it should? read · review-async/llm-calls-and-cost

It lowers them: a shared untiered page is fetched at most once per 6 hours instead of once per actor. A page that always fails has no failure count and is still retried every 6 hours, about 4 times a day. That is more often than tier A. It was worse before, so it is a follow-up idea, not a fault of this PR.

How: Code read. Did not measure on a preview. Both passes.

  • [read] NewsCollector.ts:825-829: a failed fetch writes only lastError; there is no failure count on the page.
  • [read] news-collector.config.ts:62: TIER_A_DAYS is 1.
holdsAre the PR's testing claims true? ran · review-simulator/H8

Yes. "This test fails without the fix" is true. The unit test checks the page at 0 hours, at 5 hours and at exactly 6 hours after the fetch. The PR does not claim a rerun on real data, and there is none.

How: Ran both test files and the old-code check myself.

  • [ran] old code: integration test fails with 2 fetches. PR head: 6/6 unit and 4/4 integration tests pass.
unknownIs this review independent of the agent that wrote the PR? read · review-simulator/H10

Both passes are Claude, as is the author, so their errors may be alike. The change is one function, and its test fails on the old code. A short human read of isPageDue is enough.

How: Two passes by Claude agents: one with the review skills, one without. Same model family as the author.

  • [read] independent pass notes in 955.independent-pass.md: same verdict, no blocker.
holdsDoes the PR touch the healthboard page mirror or any dashboard view? ran · free-form

No. It touches one service file and two tests. "page mirror: not affected" is right.

How: Read the diff file list.

  • [ran] gh pr diff 955: newsPages.ts, collectorNewsPages.test.ts and pageLinks.test.ts only.
10 · Not checked, and what to do next

What this review did not do

  • I did not run the real worker on a preview with this fix. The PR has no rerun of the PR 954 worker round, so the "fetched once" result on real data is not shown.
  • I did not see the human words that asked for this PR (the Slack thread of the PR 954 run). The only written origin is the bug note in PR 954.
  • No test runs two actors at the same time. Both tests run them one after the other, so the "fetch still running" case is covered only by the unit test of isPageDue.
  • I did not check the prod or stage value of NEWS_COLLECTOR_COOLDOWN_HOURS. With 0 the fix does nothing.

What to do next

  1. Merge it. It is correct and small, and the integration test fails on the old code.
  2. Optional, before merge: add one sentence to the COOLDOWN_HOURS comment in news-collector.config.ts that the page retry wait uses it too.
  3. Follow-up docs PR: change NEWS_PAGES.md §4.1 from "due again next pass" to "due again after NEWS_COLLECTOR_COOLDOWN_HOURS".
  4. Separate ticket, not caused by this PR: an aborted first fetch can give a page tier none. Such a page is never fetched again.
11 · How to respond

React on the walkthrough, or reply in the PR

  • The walkthrough behind this guide: the YAML.
  • Or reply on #955, naming the decision number.