Skip to main content
Scout can store a local article corpus separately from discovered entries. Use this when you need dated news records first, then want scout run to mine those articles for people, organizations, and initiatives. Article records are not Atlas profiles. They are saved source records with a URL, title, publication date, source domain, provider id, provider metadata, and text-derived mention candidates. Guardian imports use the Guardian Content API. They do not open a browser or crawl article pages. Scout requests article text fields from the API, stores reviewable metadata, and extracts mention candidates from headline, trail text, and body text fields.

​
Import Guardian Articles

Guardian imports require a Guardian Content API key:
Scout imports yearly windows across the requested range so a large target does not collapse into the first dense archive period. Use a lower delay for local backfills only when the provider limit allows it:

​
Crawl News Sites

Use the crawler when you have public news sitemaps, RSS feeds, Atom feeds, or article index pages and want Scout to discover article URLs itself:
For larger jobs, put one seed URL per line and pass --seed-file seeds.txt. Scout parses sitemap and feed resources as discovery indexes, prioritizes likely article URLs ahead of deeper index pages, canonicalizes tracking-query variants, and stores crawl provenance with each article record. Scout fetches ordinary HTML first. If browser fallback is enabled in the active Scout profile and the page looks useful but JavaScript-dependent, Scout may use bounded headless Chromium rendering through Playwright. --browser-renders sets the crawl-level render budget; use 0 for a no-browser crawl.

​
Extract From Stored Articles

After a corpus exists, run the normal discovery command for the place and issue set you care about:
Scout extracts from stored articles first. If the target has not been reached and search is connected, the same run can continue into new public-source discovery.

​
Verify Coverage

Check corpus size and coverage before using it for extraction:
JSON output includes total articles, earliest and latest publication dates, counts by year, counts by source domain, counts by provider, metadata-complete article counts, mention coverage, total mentions, unique mentions, and mention type counts:
Use articles verify before a large handoff:
--from-date and --to-date on stats and verify prove the corpus covers a date range. --published-from and --published-through on verify prove every stored article is inside the accepted publication window. Use the published window flags when stale rows would make the corpus unfit for review. guardian_tags are Guardian taxonomy labels such as keywords, contributors, series, tones, or tracking tags. They are source-provided metadata. Scout keeps them separate from mentions, which are candidates extracted from article text. Do not treat provider tags as proof that Scout found a person, organization, or place in the article body.

​
Refresh Mention Candidates

When Scout’s text mention extractor improves, refresh stored article metadata without importing new source records:
The refresh command recomputes mentions from each stored article title, trail_text, and body_text_excerpt. It preserves provider metadata such as guardian_tags, and it does not convert provider tags into mentions.

​
Export Articles

JSONL is the safest large-file handoff:
Use CSV when the next review step is a spreadsheet:
The export keeps URLs, titles, publication dates, source domains, provider ids, API URLs, metadata, and local creation timestamps together. Metadata includes Guardian tags, bylines, short URLs, thumbnails, body-text length and excerpt, and text-derived mention candidates when available. Keep those fields attached when handing the file to another review or extraction step.