Shadow Octopus Supported Sources¶

Source definitions are TOML files under sources/.

Files directly under sources/ are production source definitions. Files under sources/examples/ are reusable source-kind examples and are disabled by default.

Production Sources¶

Source	`source_kind`	Remote source	Scheduler role	Output
`cninfo_announcements`	`cninfo`	CNInfo announcements and historical search	Recent-window refresh, historical backfill, object downloads	Announcement records, PDF manifests, PDF objects
`cls_telegraph`	`cls_telegraph`	财联社电报	Latest news sync	Telegraph news records
`sina7x24`	`sina7x24`	新浪财经 7x24	Latest news sync	7x24 news records
`eastmoney_7x24`	`eastmoney_7x24`	东方财富 7x24	Latest sync and bounded backfill	Fast-news records
`stcn_kuaixun`	`stcn_kuaixun`	证券时报快讯	Latest sync and bounded backfill	Kuaixun records
`jin10_qh_flash`	`jin10_qh_flash`	金十期货快讯	Latest sync and bounded backfill	Futures flash records
`gangtise_foreign_reports`	`gangtise_foreign_reports`	Gangtise foreign research reports	Metadata sync, optional PDF object materialization	Research-report records, PDF manifests, resolved PDFs when requested

Scheduler Defaults¶

Source	Interval	Backfill	Notes
`cninfo_announcements`	latest every `300s`, history every `600s`, downloads every `60s`	From `2000-01-01` by history search	Requires `CNINFO_ACCEPT_ENC_KEY` when the company-page API header is needed
`cls_telegraph`	`60s`	Manual command path	Uses CLS roll-list API
`sina7x24`	`300s`	Manual command path	Pages backward from local minimum id when cursor is omitted
`eastmoney_7x24`	`60s`	`600s`	Backfill auto-detects oldest local `sortEnd` cursor
`stcn_kuaixun`	`60s`	`600s`	Uses STCN date-window list API by default
`jin10_qh_flash`	`60s`	`600s`	Backfill auto-detects oldest local `max_time` cursor
`gangtise_foreign_reports`	`3600s`	Metadata paging	Requires `GANGTISE_OPENAPI_TOKEN`

These intervals are intentionally conservative. Octopus favors resumability and source respect over aggressive crawling.

CNInfo Announcements¶

Configured file:

sources/cninfo_announcements.toml

Main commands:

uv run shadow-octopus --config config.example.toml refresh-cninfo-latest \
  --source cninfo_announcements \
  --window-days 14 \
  --pages-per-day 5

uv run shadow-octopus --config config.example.toml backfill-cninfo-history \
  --source cninfo_announcements \
  --stream a_share_all \
  --start-date 2000-01-01 \
  --max-pages 10

uv run shadow-octopus --config config.example.toml download-objects \
  --source cninfo_announcements \
  --limit 200

The current production path treats refresh-cninfo-latest as the recent freshness job and backfill-cninfo-history as the historical completeness path. sync-cninfo --rotate-companies remains available for diagnostics when the required CNInfo header is valid.

Live News Sources¶

CLS:

uv run shadow-octopus --config config.example.toml sync-cls-telegraph \
  --source cls_telegraph \
  --count 20 \
  --pages 1

Sina:

uv run shadow-octopus --config config.example.toml sync-sina7x24 \
  --source sina7x24 \
  --count 100 \
  --pages 1

Eastmoney:

uv run shadow-octopus --config config.example.toml sync-eastmoney-7x24 \
  --source eastmoney_7x24 \
  --count 50 \
  --pages 5

STCN:

uv run shadow-octopus --config config.example.toml sync-stcn-kuaixun \
  --source stcn_kuaixun \
  --count 50 \
  --pages 5

Jin10:

uv run shadow-octopus --config config.example.toml sync-jin10-qh-flash \
  --source jin10_qh_flash \
  --count 50 \
  --pages 2

Gangtise Foreign Reports¶

Configured file:

sources/gangtise_foreign_reports.toml

Metadata sync:

export GANGTISE_OPENAPI_TOKEN='...'

uv run shadow-octopus --config config.example.toml sync-gangtise-foreign-reports \
  --source gangtise_foreign_reports \
  --size 50 \
  --pages 1

Materialize one requested PDF:

uv run shadow-octopus --config config.example.toml materialize-gangtise-report-pdf \
  --source gangtise_foreign_reports \
  --source-record-id gangtise_report:454901220168568832

Lighthouse exposes metadata-first browsing and PDF download request/status APIs. Octopus remains the component that writes the resolved PDF object manifest.

Manual Acquisition Paths¶

Octopus also supports manual or semi-manual ingestion into the same raw contract:

Command	Purpose
`import-object`	Import a local PDF, MP3, MP4, XLSX, or similar file into a source workspace
`capture-url`	Fetch one URL directly into the source-local object store
`capture-url-list`	Fetch a slow URL list from text or JSONL input
`discover-links`	Discover supported object links from an HTML page into pending manifests
`import-shadow-pdf`	Import legacy `shadow-pdf` CNInfo records and cached PDFs
`import-gangtise-legacy-folder`	Import locally downloaded Gangtise PDF files

Reusable Source Kinds¶

Example source kinds live under sources/examples/:

Example source	`source_kind`	Use case
`example_news_feed`	`feed`	RSS/Atom feeds
`example_json_news`	`json_api`	Simple JSON list APIs with configured paths and pagination
`example_html_news`	`html_page`	Simple HTML listing pages parsed by configured patterns

Use generic source kinds for simple sources. Prefer a dedicated adapter when a website needs signing, anti-abuse handling, unusual pagination, or source-specific normalization.