跳转至

Shadow Lakehouse Storage

shadow-lakehouse is the server-side storage convention behind shadow-octopus and shadow-lighthouse.

It is not a single database. It is a filesystem-first lakehouse layout that separates raw acquisition from read-side indexes:

  • shadow-octopus writes source-local raw records, object manifests, object bytes, and source state.
  • shadow-lighthouse reads Octopus raw contracts and builds query indexes, search projections, and document evidence surfaces.

Production storage root:

/dev/data1/shadow-lakehouse/

Production Layout

/dev/data1/shadow-lakehouse/
  octopus/
    <source_name>/
      state.db
      runs/
      records/
        month=YYYY-MM/
          detail.jsonl
      manifests/
        objects.jsonl
        objects-resolved.jsonl
        objects-failed.jsonl
      objects/
        sha256/
          <sha256-prefix-or-key>/...
  lighthouse/
    <source_name>/
      jobs/
      canonical/
      indexes/
        catalog.sqlite
        fts.sqlite
        tables.sqlite
      artifacts/
    global/
      indexes/
        news.sqlite
        document_evidence.sqlite
  meilisearch/

The important habit is source-local ownership. A source has its own Octopus workspace, state database, raw records, manifests, object store, Lighthouse indexes, and artifacts. Cross-source views are projections over those source-local workspaces, not the source of truth.

Storage Formats

Layer Format Why
Raw records Month-partitioned JSONL Append-friendly, auditable, easy to replay, bounded by source and month
Source state SQLite Small durable scheduler/checkpoint state without running a database service
Object manifests JSONL Appendable object queue, resolution, and failure history
Raw objects Files keyed by SHA-256 Content-addressed storage, stable references, object bytes kept outside indexes
Catalog/search indexes SQLite Low-latency local lookup, FTS, issuer lookup, table row search, rebuildable projections
Global views SQLite projections Fast cross-source lists/search without making raw storage global
Full-text engine Optional Meilisearch projection Faster product search when SQLite FTS is not the right serving shape
Large columnar artifacts Parquet when needed Efficient scan/filter/compact path for large derived table or text-unit artifacts

Space And Speed Tradeoffs

Raw storage optimizes for durability and recovery, not direct query speed. Octopus writes records and manifests as append-friendly files, partitions raw records by month, and stores objects by content hash. That keeps ingestion simple and makes failures local to one source.

Lighthouse optimizes for query speed. It duplicates selected metadata, text, issuer mappings, table cells, and evidence pointers into SQLite or search projections. That duplication is intentional: raw JSONL and object files remain the source of truth, while read indexes are disposable and rebuildable.

This gives the system two different cost profiles:

  • ingestion stays cheap because sources append raw records and do not maintain global query state
  • serving stays fast because users and agents hit read indexes instead of scanning raw source files
  • rebuilds are bounded because one source can be re-indexed without touching every other source
  • disk usage is controlled because heavy object bytes are stored once by content identity, while indexes keep only query-shaped projections

Operating Habits

Use raw paths when auditing acquisition, replaying a source, repairing object downloads, or proving provenance.

Use Lighthouse indexes and APIs when browsing documents, searching text, resolving issuers, serving original objects, or feeding Shadow Assembly and agents.

Do not patch raw files on the server by hand. Repair the writer, rerun the source task, or rebuild the read projection. If an index looks wrong but Octopus raw data is correct, rebuild Lighthouse rather than editing SQLite rows manually.

Avoid turning every raw byte into a hot searchable artifact. Large PDFs, parsed text, OCR, table extraction, and AI evidence should be promoted according to product value and freshness needs. Metadata can be searchable first; expensive parsed artifacts can arrive later.