Skip to content

Shadow Ingest Deployment

shadow-ingest uses a maintained dual-service deployment model.

Ports

  • 30110: FastAPI / HTTP service
  • 30100: Arrow Flight service

Required Environment

export SHADOW_INGEST_DATA_ROOT=/dev/data1/iceberg/data
export SHADOW_MAX_SYMBOLS=1000
export SHADOW_MAX_DATE_RANGE_DAYS=3650
export SHADOW_MAX_RESULT_ROWS=1000000
export SHADOW_MAX_RESULT_BYTES=52428800
export SHADOW_QUERY_TIMEOUT=30.0
export SHADOW_MAX_CONCURRENT_QUERIES=4

Manual Start

HTTP service

shadow-ingest serve-fastapi \
  --data-root /dev/data1/iceberg/data \
  --host 0.0.0.0 \
  --port 30110

The maintained HTTP entrypoint is serve-fastapi. The legacy serve command has been removed.

Flight service

shadow-ingest serve-flight \
  --data-root /dev/data1/iceberg/data \
  --host 0.0.0.0 \
  --port 30100

Standard Maintained Mode

Use both systemd services:

  • shadow-ingest-http.service
  • shadow-ingest-flight.service

Typical upgrade flow on the server:

cd ~/shadow-ingest
git pull
source .venv/bin/activate
uv pip install -e ".[server]"
sudo systemctl restart shadow-ingest-http.service
sudo systemctl restart shadow-ingest-flight.service

Post-Deploy Checks

curl http://127.0.0.1:30110/health
curl -s http://127.0.0.1:30110/openapi.json | python3 -m json.tool | head -n 20
curl -s http://127.0.0.1:30110/api/industry/standards | python3 -m json.tool
./.venv/bin/python scripts/check_public_apis.py --url http://127.0.0.1:30110 --skip-financial
./.venv/bin/python scripts/check_flight.py --url grpc://127.0.0.1:30100