> ## Documentation Index
> Fetch the complete documentation index at: https://atlas.rebuildingus.org/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Build article corpora with Scout

> Collect dated news article records that normal Scout runs can extract from.

Scout can store a local article corpus separately from discovered entries. Use
this when you need dated news records first, then want `scout run` to mine those
articles for people, organizations, and initiatives.

Article records are not Atlas profiles. They are saved source records with a
URL, title, publication date, source domain, provider id, provider metadata, and
text-derived mention candidates.

Guardian imports use the Guardian Content API. They do not open a browser or
crawl article pages. Scout requests article text fields from the API, stores
reviewable metadata, and extracts mention candidates from headline, trail text,
and body text fields.

## Import Guardian Articles

Guardian imports require a Guardian Content API key:

```bash theme={null}
scout articles import guardian \
  --api-key "$GUARDIAN_API_KEY" \
  --from-date 2006-07-06 \
  --to-date 2026-07-06 \
  --target-count 100000
```

Scout imports yearly windows across the requested range so a large target does
not collapse into the first dense archive period. Use a lower delay for local
backfills only when the provider limit allows it:

```bash theme={null}
scout articles import guardian \
  --api-key "$GUARDIAN_API_KEY" \
  --from-date 2006-07-06 \
  --to-date 2026-07-06 \
  --target-count 100000 \
  --page-size 200 \
  --delay-ms 100 \
  --json
```

## Crawl News Sites

Use the crawler when you have public news sitemaps, RSS feeds, Atom feeds, or
article index pages and want Scout to discover article URLs itself:

```bash theme={null}
scout articles crawl \
  --seed https://example-news.org/sitemap.xml \
  --target-count 1000 \
  --max-pages 10000 \
  --max-depth 3 \
  --max-concurrent 24 \
  --max-per-domain 2 \
  --from-date 2006-07-06 \
  --to-date 2026-07-06 \
  --browser-renders 8 \
  --json
```

For larger jobs, put one seed URL per line and pass `--seed-file seeds.txt`.
Scout parses sitemap and feed resources as discovery indexes, prioritizes likely
article URLs ahead of deeper index pages, canonicalizes tracking-query variants,
and stores crawl provenance with each article record.

Scout fetches ordinary HTML first. If browser fallback is enabled in the active
Scout profile and the page looks useful but JavaScript-dependent, Scout may use
bounded headless Chromium rendering through Playwright. `--browser-renders`
sets the crawl-level render budget; use `0` for a no-browser crawl.

## Extract From Stored Articles

After a corpus exists, run the normal discovery command for the place and issue
set you care about:

```bash theme={null}
scout run \
  --location "Las Vegas, NV" \
  --issues housing_affordability,public_transit \
  --target-count 250
```

Scout extracts from stored articles first. If the target has not been reached
and search is connected, the same run can continue into new public-source
discovery.

## Verify Coverage

Check corpus size and coverage before using it for extraction:

```bash theme={null}
scout articles stats \
  --min-count 100000 \
  --min-with-mentions 100000 \
  --min-metadata-complete 100000 \
  --from-date 2006-07-06 \
  --to-date 2026-07-06
```

JSON output includes total articles, earliest and latest publication dates,
counts by year, counts by source domain, counts by provider, metadata-complete
article counts, mention coverage, total mentions, unique mentions, and mention
type counts:

```bash theme={null}
scout articles stats --json
```

Use `articles verify` before a large handoff:

```bash theme={null}
scout articles verify \
  --min-count 100000 \
  --min-unique-urls 100000 \
  --min-crawl 100000 \
  --min-crawl-discovered 100000 \
  --min-with-mentions 100000 \
  --min-metadata-complete 100000 \
  --published-from 2006-07-06 \
  --published-through 2026-07-06
```

`--from-date` and `--to-date` on `stats` and `verify` prove the corpus covers a
date range. `--published-from` and `--published-through` on `verify` prove every
stored article is inside the accepted publication window. Use the published
window flags when stale rows would make the corpus unfit for review.

`guardian_tags` are Guardian taxonomy labels such as keywords, contributors,
series, tones, or tracking tags. They are source-provided metadata. Scout keeps
them separate from `mentions`, which are candidates extracted from article text.
Do not treat provider tags as proof that Scout found a person, organization, or
place in the article body.

## Refresh Mention Candidates

When Scout's text mention extractor improves, refresh stored article metadata
without importing new source records:

```bash theme={null}
scout articles refresh-mentions --json
```

The refresh command recomputes `mentions` from each stored article title,
`trail_text`, and `body_text_excerpt`. It preserves provider metadata such as
`guardian_tags`, and it does not convert provider tags into mentions.

## Export Articles

JSONL is the safest large-file handoff:

```bash theme={null}
scout articles export \
  --format jsonl \
  --output articles.jsonl
```

Use CSV when the next review step is a spreadsheet:

```bash theme={null}
scout articles export \
  --format csv \
  --output articles.csv
```

The export keeps URLs, titles, publication dates, source domains, provider ids,
API URLs, metadata, and local creation timestamps together. Metadata includes
Guardian tags, bylines, short URLs, thumbnails, body-text length and excerpt,
and text-derived mention candidates when available. Keep those fields attached
when handing the file to another review or extraction step.