#955 · execution path 1 / 10
1 · Review guide · #955

fix(signals): fetch a shared news page once per pass until it is tiered

What is about to happen. The news collector worker on stage and prod pays Scrape.do or Firecrawl for every news page fetch. Today a page with no tier yet can be fetched again by each actor that shares it, minutes apart. This PR makes such a page wait 6 hours after a fetch of it starts, for every actor. The code change is one function, plus two tests. A revert undoes it, and no stored data changes.

Review artifact: execution path. Reviewed 2026-11-17 by Claude (review-artifact skill), a different agent session from the author. PR head e8cdd15a9 is one commit on top of the stage tip 1b77564e1. I read the code at the PR head, ran both test files, and ran the new integration test against the old isPageDue. I read PR 954 for the origin. I did not query any database and did not run the worker. A second agent did an independent pass without the review skills; its notes are in 955.independent-pass.md.

Use → / space / swipe to move.

2 · Your decisions

2 calls. Blocking ones first.

Judgement

1. Update NEWS_PAGES.md §4.1 in a follow-up docs PR?
Recommended: Yes, a one-line follow-up PR.

If no: A reader of §4.1 may think a failed page is retried by the next actor. They may change the code back to match the doc.

Options: Yes, a one-line follow-up PR · Yes, add it to this PR · No change

Judgement

2. Add one sentence to the COOLDOWN_HOURS comment before merge?
Recommended: Yes, one sentence in the config comment.

If no: Someone tunes the actor cooldown and the double fetches come back with no warning.

Options: Yes, one sentence in the config comment · Yes, a separate page retry setting · No change

3 · Words used

Words this guide uses

actor
A company, person or body the collector gathers news for. Each actor has its own collection run.
pass
The source texts use "pass" two ways. NEWS_PAGES.md means one actor's collection run. PR 954 and 955 mean one round of the worker: it takes up to 10 actors and runs 4 at once. This review says "actor run" and "worker round".
actor run
One collection run for one actor. An actor is not taken again until 6 hours after its last run finished. apps/api-server/src/services/signals/NewsCollector.ts:369
news page
A listing page on a website, such as a blog or a news section. The collector fetches it and follows its article links. docs/signals/NEWS_PAGES.md
shared page
A news page linked to more than one actor. Its articles are found for all of them from one fetch.
tier
How often a page is fetched: A every day, B every 3 days, C every 7 days, none never. A page gets its first tier at the end of its first fetch that returns the page, from the number of articles it lists. apps/api-server/src/services/signals/actors/newsPages.ts:167
untiered page
A page with no tier yet (bucket is null). Either nobody fetched it, its first fetch is still running, or its fetches failed.
claim
How an actor run takes a page: it writes a new lastFetchedAt only if the value is still the one it read. If another run changed it first, the write matches nothing and the page is skipped. apps/api-server/src/services/signals/NewsCollector.ts:795
cooldown
NEWS_COLLECTOR_COOLDOWN_HOURS, default 6. Until this PR it only meant the time before a finished actor is taken again. With this PR, an untiered page also waits this many hours, counted from the start of its last fetch. apps/api-server/src/config/news-collector.config.ts:15
pull now
An operator button that runs one actor at once, even inside its cooldown. It treats every search query as due. News pages keep their normal due rule.
own blog
A news page that is an actor's own blog. It has exactly one actor, its owner, and never falls to tier none.
lease takeover
When a worker dies, another worker takes over its actor after 90 seconds without a heartbeat. apps/api-server/src/config/news-collector.config.ts:20
4 · The path · decision 1 of 4

Where should the fix go?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

The claim already blocks two runs that read the page at the same moment. It fails when a second run reads the page after the first one claimed it, because the page still looks due.

AChange isPageDue, so an untiered page that was fetched recently is not due.
One function, pure and unit-testable.
The claim stays as it is.
The function now reads config, so it is less pure.
BChange the claim in collectPages, so it also requires the page to be tiered or its lastFetchedAt to be old.
The rule lives at the one place that writes.
A longer Mongo filter. The "due" rule is then split over two places.
CKeep a list of pages fetched in this worker process and skip them.
No change to stored data or rules.
Two worker processes still fetch the page twice.
It does not help a failed page in the next batch.

Taken: A. The predicate is where "due" is decided, and its only caller is collectPages. One line changes and the claim keeps working as before.

  • [read] PR 955 newsPages.ts:176-182: the new branch for bucket null and lastFetchedAt set.
  • [ran] grep isPageDue in apps and packages: only NewsCollector.ts and the two tests use it.
5 · The path · decision 2 of 4

How long should an untiered page wait after a fetch starts?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

NEWS_PAGES.md §4.1 says a page that could not be fetched "is due again next pass", meaning the next actor run. An actor runs again at least 6 hours after its last run finished.

AReuse NEWS_COLLECTOR_COOLDOWN_HOURS (6 hours).
No new setting.
The same actor still retries the page on its next run, so the doc line stays true for it.
One setting now means two things. Set to 0, it switches the fix off with no warning.
BAdd a new setting, for example NEWS_COLLECTOR_PAGE_RETRY_HOURS.
Each wait can be tuned on its own.
The config comment stays accurate.
One more setting to document and to keep in step with the actor cooldown.
CWait a whole tier C interval (7 days) after a failure.
A dead page costs far less.
A page that failed once in a short outage is not fetched for a week.
It contradicts the doc line.
DWait only while a fetch runs (a few minutes), then retry at once.
A failed page is retried by the next actor that shares it.
A failed shared page is still fetched once per actor.
PR 954 saw one page fetched up to 5 times this way.

Taken: A. 6 hours is also the least time between two runs of the same actor, so the §4.1 line "due again next pass" still holds for that actor. Other actors that share the page wait until the 6 hours pass.

  • [read] news-collector.config.ts:11-15: COOLDOWN_HOURS, default 6, documented for finished actors only.
  • [read] NewsCollector.ts:374: an actor is taken again only when it finished more than COOLDOWN_HOURS ago.
6 · The path · decision 3 of 4

Should this PR also update NEWS_PAGES.md?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

ALeave the docs as they are.
The user asked for no docs change.
A smaller diff.
§4.1 no longer says when a failed page is retried.
BChange one line of §4.1 to name the cooldown.
The doc and the code agree.
It goes against what the user asked.

Taken: A. The user asked for no docs change. The PR description says so.

  • [quoted] PR 955: "No docs change, as asked."
7 · The path · decision 4 of 4

How is the fix tested?

Serves: That rule must also hold for an untiered page, while its first fetch runs and after it fails.

AA unit test of isPageDue with no tier, and an integration test: two actors share a page that fails, run one after the other.
Real Mongo and the real claim.
The integration test fails on the old code.
No test with two actors running at the same time.
BAlso an integration test with two actors at the same time and a slow fetch.
It covers the "fetch still running" case end to end.
Timing tests can be flaky.

Taken: A. The "still running" and "failed" cases go through the same branch of isPageDue. The unit test covers that branch.

  • [ran] old isPageDue with the new integration test: fails, "expected [ …(2) ] to have a length of 1 but got 2".
  • [ran] PR head: pageLinks.test.ts 6/6 pass; collectorNewsPages.test.ts 4/4 pass.
8 · What I checked

7 questions against the code

Basis: ran = a command, test or query I ran; read = code I read; quoted = words from a PR, issue or report.

holdsDoes the new test fail on the old code, and does it use real Mongo? ran + read · review-backend/tests

Yes to both. It fails on the old code and passes on the PR head. It uses the integration tier, not mocks of models. The fake fetcher now returns null (a failed fetch) for an empty body; no other test passes an empty body, so nothing else changes.

How: Ran the integration test with stage's isPageDue copied into a scratch worktree, then removed the worktree. Skill pass and independent pass.

  • [ran] old code: 1 failed, 3 passed. PR head: 4 passed.
  • [read] PR 955 collectorNewsPages.test.ts:30 and :150: the only other body is non-empty HTML.
holdsIs the 6-hour wait right, given the doc says a failed page "is due again next pass"? read · free-form

For the same actor, yes. The page wait starts when its run claims the page. The actor cooldown starts when that run finishes, which is later. So when the actor's next run starts, more than 6 hours have passed since the claim, and the page is due again. Other actors that share the page now skip it until 6 hours pass. That is the point of the fix.

How: Compared the page wait with the actor cooldown in claimBatch. Code read only. Skill pass and independent pass.

  • [read] NewsCollector.ts:783 sets now when page collection starts; :797 writes it to lastFetchedAt.
  • [read] NewsCollector.ts:374 and :763: the actor cooldown counts from newsCollectionFinished, written at the end of the run.
holdsDoes anything change for own blogs or for pages at tier none? read · free-form

Tier none is unchanged: the Mongo filter still drops it and the predicate still returns false. An own blog has one actor, and that actor runs at most every 6 hours, so it sees no change in normal runs. With this PR, an operator pull or a lease takeover inside the 6 hours no longer retries a failed own blog.

How: Code read. Skill pass.

  • [read] NewsCollector.ts:785: filter bucket.tier: { $ne: 'none' } also matches pages with no tier.
  • [read] newsPages.ts:180: none still returns false for a fetched page.
holdsWhat happens on the abort, crash and "pull now" paths? read · review-async/safety

Each now waits up to 6 hours before it retries an untiered page. A worker that dies mid-fetch is taken over after 90 seconds, and the new run skips that page. An operator pull of the actor skips it too. Before, both retried it at once. This is a small cost for a rare case. Tiered pages already worked this way.

How: Read fetchPage, probePage, claimBatch and the lease setting. Did not run them. Both passes found this.

  • [read] pageProbe.ts:34-35: a fetch error or abort becomes null, and fetchPage records "The page could not be fetched".
  • [read] NewsCollector.ts:575: collectPages does not get claim.forced, so a pull does not force pages.
failsDoes NEWS_PAGES.md now disagree with the code? read · review-scope-defensibility/omissions · decision 1

§4.1 says a page that could not be fetched "is due again next pass". That is still true for the same actor, but the doc does not say that other actors wait 6 hours. §4 item 2 ("fetched once per interval") now also holds for untiered pages. The gap was left on purpose, as the user asked.

How: Read §4 and §4.1 at the PR head. Both passes found this.

  • [read] docs/signals/NEWS_PAGES.md:52: "A page that could not be fetched is not tiered and is due again next pass."
failsDoes reusing the actor cooldown setting make the fix fragile? read · free-form · decision 2

Today it works, but by luck of the default. The config comment says the setting is for finished actors only. Someone who sets it to 0 to run actors back to back also switches this fix off, and the new unit test then fails.

How: Read the config and the new unit test. Did not check the deployed env values. Found by the independent pass; I checked it.

  • [read] news-collector.config.ts:11-15: the comment names only actors.
  • [read] PR 955 pageLinks.test.ts: with a cooldown of 0, lastFetchedAt: now gives true, and the test expects false.
holdsAre the PR's testing claims true? ran · review-simulator/H8

Yes. "This test fails without the fix" is true. The unit test checks the page at 0 hours, at 5 hours and at exactly 6 hours after the fetch. The PR does not claim a rerun on real data, and there is none.

How: Ran both test files and the old-code check myself.

  • [ran] old code: integration test fails with 2 fetches. PR head: 6/6 unit and 4/4 integration tests pass.
9 · Not checked, and what to do next

What this review did not do

  • I did not run the real worker on a preview with this fix. The PR has no rerun of the PR 954 worker round, so the "fetched once" result on real data is not shown.
  • I did not see the human words that asked for this PR (the Slack thread of the PR 954 run). The only written origin is the bug note in PR 954.
  • No test runs two actors at the same time. Both tests run them one after the other, so the "fetch still running" case is covered only by the unit test of isPageDue.
  • I did not check the prod or stage value of NEWS_COLLECTOR_COOLDOWN_HOURS. With 0 the fix does nothing.
10 · How to respond

React on the walkthrough, or reply in the PR

  • The walkthrough behind this guide: the YAML.
  • Or reply on #955, naming the decision number.