Shadow Octopus Operations¶

Octopus is operated through CLI commands and the source registry scheduler.

It does not run as a public API service. Its durable output is the source-local raw contract under the Octopus lakehouse root.

Configuration¶

Local example:

data_root = "data/octopus"
sources_dir = "sources"
request_timeout = 30
verify_tls = true

Production convention:

data_root = "/dev/data1/shadow-lakehouse/octopus"
sources_dir = "sources"
request_timeout = 30
verify_tls = true

On the server, use:

export SHADOW_OCTOPUS_CONFIG=/dev/data1/shadow-octopus/config.prod.toml

Scheduler¶

Run the scheduler locally:

uv run shadow-octopus --config config.example.toml run-scheduler \
  --max-workers 4

Run one due cycle for selected sources:

uv run shadow-octopus --config config.example.toml run-scheduler \
  --source cls_telegraph \
  --source eastmoney_7x24 \
  --max-workers 2 \
  --once

Production wrapper:

scripts/run_scheduler.sh --max-workers 4

The wrapper keeps a scheduler-level lock and writes scheduler/task logs under the ops directory. Each worker writes to only one source workspace.

Status And Verification¶

Check one source:

uv run shadow-octopus --config config.example.toml source-status \
  --source cninfo_announcements

Check initialized sources:

uv run shadow-octopus --config config.example.toml root-status

Verify a source before Lighthouse ingest:

uv run shadow-octopus --config config.example.toml verify-source \
  --source cninfo_announcements \
  --require-data

Status output includes raw record counts, manifest counts, pending/resolved/failed object counts, object bytes, latest file mtimes, latest checkpoint, and recent run records.

CNInfo Operating Path¶

Refresh the recent history window:

uv run shadow-octopus --config config.example.toml refresh-cninfo-latest \
  --source cninfo_announcements \
  --stream a_share_all \
  --window-days 14 \
  --pages-per-day 5

Backfill historical announcements:

uv run shadow-octopus --config config.example.toml backfill-cninfo-history \
  --source cninfo_announcements \
  --stream a_share_all \
  --start-date 2000-01-01 \
  --max-pages 10

Rebuild the source-local object queue when needed:

uv run shadow-octopus --config config.example.toml rebuild-object-download-queue \
  --source cninfo_announcements

Download a bounded object batch:

uv run shadow-octopus --config config.example.toml download-objects \
  --source cninfo_announcements \
  --limit 200

The downloader selects ready rows from state.db first. It can fall back to manifest scanning when the queue has not been built.

Live News Operating Path¶

Latest sync commands:

uv run shadow-octopus --config config.example.toml sync-cls-telegraph --source cls_telegraph --count 20 --pages 1
uv run shadow-octopus --config config.example.toml sync-sina7x24 --source sina7x24 --count 100 --pages 1
uv run shadow-octopus --config config.example.toml sync-eastmoney-7x24 --source eastmoney_7x24 --count 50 --pages 5
uv run shadow-octopus --config config.example.toml sync-stcn-kuaixun --source stcn_kuaixun --count 50 --pages 5
uv run shadow-octopus --config config.example.toml sync-jin10-qh-flash --source jin10_qh_flash --count 50 --pages 2

Backfill commands:

uv run shadow-octopus --config config.example.toml backfill-sina7x24 --source sina7x24 --pages 100
uv run shadow-octopus --config config.example.toml backfill-eastmoney-7x24 --source eastmoney_7x24 --pages 20
uv run shadow-octopus --config config.example.toml backfill-stcn-kuaixun --source stcn_kuaixun --pages 40
uv run shadow-octopus --config config.example.toml backfill-jin10-qh-flash --source jin10_qh_flash --pages 20

When a cursor is omitted, backfill commands derive the oldest local cursor from partitioned raw records where the source supports it.

Gangtise Operating Path¶

Metadata sync:

export GANGTISE_OPENAPI_TOKEN='...'

uv run shadow-octopus --config config.example.toml sync-gangtise-foreign-reports \
  --source gangtise_foreign_reports \
  --size 50 \
  --pages 1

Materialize one requested PDF:

uv run shadow-octopus --config config.example.toml materialize-gangtise-report-pdf \
  --source gangtise_foreign_reports \
  --source-record-id gangtise_report:454901220168568832

Gangtise can be operated metadata-first. Lighthouse exposes PDF download request/status APIs; Octopus performs the actual object materialization and writes objects-resolved.jsonl.

Upload And Extract Lakehouse Data¶

Upload local raw data as shard archives:

scripts/upload_lakehouse_to_remote.sh

Extract on the server after checksum verification:

ssh aliyun 'SHADOW_REMOTE_ZSTD_BIN=/dev/data1/bin/zstd \
  /dev/data1/shadow-octopus/scripts/extract_lakehouse_shards_remote.sh \
  /dev/data1/shadow-lakehouse/.staging/<batch>'

The extractor verifies shard checksums and zstd frames before moving source directories into:

/dev/data1/shadow-lakehouse/octopus

Handoff To Lighthouse¶

After Octopus writes or migrates raw data, Lighthouse ingests from:

/dev/data1/shadow-lakehouse/octopus/<source>

and writes read-side indexes under:

/dev/data1/shadow-lakehouse/lighthouse/<source>

Keep this handoff one-way: Octopus writes raw data, Lighthouse reads raw data and serves queries.