跳转至

Shadow Ingest

shadow-ingest is a dataframe-first, read-only access layer over the raw parquet datasets generated by download_rqdata.

The source of truth remains the existing download_rqdata/data tree. There is no second database.

For most Python users, the intended experience is simple:

  • import shadow_ingest as si
  • use the small stable SDK surface
  • receive either a final polars.DataFrame or a small discovery list

Stable Public APIs

The public SDK surface is intentionally small and currently split into four groups.

Core data APIs:

  • gather_daily_price(...)
  • gather_daily_snapshot(...)
  • gather_financial_snapshot(...)

Discoverability APIs:

  • list_fields(...)
  • list_market_calendar(...)
  • list_universe(...)

Industry APIs:

  • get_industry_standards()
  • get_industry_mapping(...)
  • get_industry_members(...)

Hong Kong APIs:

  • list_hk_market_calendar(...)
  • list_hk_universe(...)
  • gather_hk_daily_price(...)
  • gather_hk_daily_snapshot(...)

The dataframe-returning APIs return polars.DataFrame.

Current Design

The maintained contract is organized around three modules:

  • shadow_ingest/catalog.py is the dataset manifest for the filesystem-backed data tree. It records dataset layout, domain, grain, time/entity columns, and public column aliases.
  • shadow_ingest/operations.py is the public operation registry. It defines maintained SDK names, HTTP routes, and dataset dependencies for each operation.
  • shadow_ingest/kernel.py is the internal single-dataset read kernel. It keeps lookup helpers and metadata scans on one path without changing the public SDK surface.

FastAPI routes and SDK entrypoints resolve public names from the operation registry.

The generic POST /query route remains available only as an internal/debug primitive. It is not the maintained public API surface.

Recent Updates

Recent shadow-ingest changes that are now reflected in this documentation:

  • list_fields(...) is the stable public field-discovery helper for gather_daily_price(...)
  • total_turnover is documented as traded value, not share volume
  • list_universe(...) now requires an explicit date; the SDK accepts YYYYMMDD inputs and normalizes backing order_book_id columns to public stock_code results
  • industry lookup APIs were added for standards, stock-to-industry mapping, and industry-to-members queries
  • Hong Kong market data now has separate APIs instead of sharing the A-share market parameter path
  • serve-fastapi is now the maintained HTTP entrypoint; the legacy serve HTTP entrypoint has been removed

Who This Is For

Use shadow-ingest if you need:

  • a simple Python data pull interface
  • a maintained remote service mode
  • efficient transport for larger dataframe-shaped responses
  • a stable query interface that hides transport and batching details

Do not use shadow-ingest for data acquisition, license management, factor formulas, or strategy execution. Those belong to download_rqdata, shadow-factor, and shadow-backtest.

If you are new to the SDK, the easiest mental model is:

  1. use list_market_calendar(...) to pick valid trading dates
  2. use list_universe(...) to pick valid stock codes for a date
  3. use list_fields(...) when you are calling gather_daily_price(...)
  4. use the default standard="sws" when that taxonomy is acceptable, or call get_industry_standards() when you need to inspect available taxonomy names
  5. call one of the dataframe-returning APIs to fetch the final table

For Hong Kong stocks, use the HK-specific helpers:

import shadow_ingest as si

hk_dates = si.list_hk_market_calendar(year=2026)
hk_codes = si.list_hk_universe(date="2026-06-25")
hk_df = si.gather_hk_daily_price(
    stock_codes=["00700.XHKG"],
    start_date="2026-06-25",
    end_date="2026-06-25",
    fields=["open", "close"],
)