Hierarchical Embeddings

Note

This page is the user-facing reference. For the technique explained from first principles, plus a controlled evaluation across local embedding encoders, see Raw-Span Hierarchical Retrieval: Method and Evaluation.

By default LocalVectorDB embeds and indexes documents at a single granularity: each document is chunked, and every chunk is embedded and stored in one FAISS index. Hierarchical embeddings add two coarser levels on top of that, giving a three-level retrieval hierarchy:

document   ← whole-document vector (coarsest)
  └── section   ← a run of chunks under one heading
        └── chunk   ← the usual fine-grained unit (default)

Each level has its own FAISS index, so a single database can be searched at whichever granularity fits the question:

  • chunks — precise passage retrieval (the default, unchanged).

  • sections — find the most relevant section of a document. Useful for long, structured documents (manuals, papers, books) where the right region matters more than the exact sentence.

  • documents — coarse “which document is this about?” retrieval.

  • fused — blend chunk retrieval with section retrieval into one ranking. This is the mode that improves retrieval quality on real, structured documents (see Fused retrieval).

Sections are an overlay on top of chunking, not a different chunking strategy: the document is chunked exactly as before, sections are detected from the document’s heading structure, and each chunk is assigned to its containing section.

The feature is fully opt-in. With hierarchical_embeddings=False (the default) nothing changes, no extra indices are built, and the default retrieval path is byte-for-byte identical.

How a section is represented: raw-span vs centroid

The interesting question is what vector stands in for a section. There are two strategies, selected by the section_vector_strategy constructor argument.

"rawspan" (default for new hierarchical databases)

The section’s vector is the embedding of the section’s own text — the concatenated span of every chunk under that heading, embedded directly.

"centroid"

The section’s vector is the mean of its member chunk embeddings. This is free (no extra embedding calls) but is the weaker representation; it is the legacy behaviour and the automatic choice for databases created before the strategy option existed.

Why the default is raw-span

A centroid is a blur: averaging unit vectors throws away every cross-chunk interaction — coreference, topic composition, the way an argument builds across paragraphs — and leaves only a mean direction. Embedding the section’s actual text keeps that structure.

We measured this before changing the default, on real embeddings across Qasper (real NLP papers) and synthetic section corpora built from BEIR — and again across several local encoders. Raw-span sections beat both the chunk baseline and the centroid at every meaningful target; averaging genuinely discards signal the span embedding keeps, and on some corpora the centroid section scored below chunk-only while the raw-span section scored well above it. The full tables, encoders, and methodology are in Raw-Span Hierarchical Retrieval: Method and Evaluation.

Cost and back-compatibility

  • Raw-span costs one extra embedding call per section at ingest. Sections are far fewer than chunks, so this is modest, but it is not free the way a centroid is. It also means ingest calls the embedding provider for the section text.

  • The document-level vector is always a centroid, regardless of strategy. The document level carries little retrieval signal (see Limitations), is not used by fused retrieval, and is not worth an extra embedding call.

  • The strategy is persisted with the database and resolves on reopen with a strict precedence, so existing indices are never silently reinterpreted:

    1. a persisted section_vector_strategy value always wins on reopen;

    2. a hierarchical database created before this option existed resolves to "centroid" — its stored vectors are centroids and must be scored as such;

    3. a brand-new hierarchical database (or an existing non-hierarchical one turned hierarchical) defaults to "rawspan".

    To migrate an existing centroid database to raw-span, pass section_vector_strategy="rawspan" and call rebuild_hierarchical_embeddings() (see Rebuilding indices for existing databases).

Enabling hierarchical embeddings

Pass hierarchical_embeddings=True when constructing the database:

from localvectordb import VectorDB

db = VectorDB(
    "manuals",
    "./vector_storage",
    hierarchical_embeddings=True,
    section_vector_strategy="rawspan",              # default for new hierarchical DBs
    # optional: customise how sections are detected
    section_pattern=r"^(#{1,6})\s+(.+)$",           # default: Markdown headers
    # optional: attach metadata to every detected section
    section_metadata_extractors=["heading_path", "word_count", "keywords"],
)

db.upsert(open("guide.md").read(), ids="guide.md")

Constructor parameters:

hierarchical_embeddingsbool, default False

Enable section- and document-level indices. When True, two extra FAISS sidecar files are created next to the main index: <name>_sections.faiss and <name>_documents.faiss.

section_vector_strategy"rawspan" | "centroid", optional

How section vectors are built (see How a section is represented: raw-span vs centroid). Defaults to "rawspan" for a new hierarchical database and "centroid" for a legacy one; once set it is persisted and wins on reopen.

section_patternstr, default r"^(#{1,6})\s+(.+)$"

Regex (MULTILINE) used to detect section headings. It must expose two capture groups — group 1 is the level indicator (e.g. the # characters), group 2 is the heading text. The default matches Markdown headings, which suits extracted content (the file extractors emit Markdown).

section_metadata_extractorslist, optional

Built-in extractor names and/or SectionMetadataExtractor instances to run against each section during ingestion. See Section metadata.

Section detection

Detection is implemented by SectionDetector, which scans text for headings and emits SectionBoundary objects. You can use it directly to preview how a document will be split:

from localvectordb.section_detection import SectionDetector

detector = SectionDetector()           # defaults to Markdown headers (# … ######)
sections = detector.detect_sections(document_text)

for s in sections:
    print(s.heading_level, s.heading, s.start_pos, s.end_pos)

Key behaviours:

  • Preamble handling — text before the first heading becomes a leading section with heading=None.

  • Code-fence aware — # lines inside fenced code blocks (triple backticks or ~~~) are not treated as headings, so shell prompts and comments in example snippets do not create spurious sections. The helper find_code_fence_spans() computes the fenced regions that are excluded.

  • Custom patterns — pass a section_pattern with two capture groups to detect non-Markdown structures, e.g. section_pattern=r"^(SECTION \d+): (.+)$".

Because raw-span section quality depends on sections being meaningful spans, detection quality matters more with "rawspan" than with "centroid". A document with no headings becomes a single whole-document section; a document whose headings carve it into coherent topics is where raw-span pays off most.

Section metadata

Section metadata extractors run once per detected section during ingestion and merge their output into that section’s metadata, where it is returned on section-level query results. Built-in extractors (all text-based, no external dependencies) are referenced by name:

  • "word_count" — adds word_count: number of words in the section.

  • "char_count" — adds char_count: number of characters in the section.

  • "keywords" — adds keywords: top-N keywords by frequency (stop-words removed).

  • "heading_path" — adds heading_path: the nested heading trail, e.g. "Chapter 1 > Introduction > Background".

db = VectorDB(
    "manuals", "./vector_storage",
    hierarchical_embeddings=True,
    section_metadata_extractors=["heading_path", "keywords"],
)

Write a custom extractor by subclassing SectionMetadataExtractor and passing an instance:

from localvectordb.section_metadata import SectionMetadataExtractor

class ReadingTimeExtractor(SectionMetadataExtractor):
    name = "reading_time"

    def extract(self, section_text, heading, context):
        minutes = max(1, round(len(section_text.split()) / 200))
        return {"reading_time_min": minutes}

db = VectorDB(
    "manuals", "./vector_storage",
    hierarchical_embeddings=True,
    section_metadata_extractors=[ReadingTimeExtractor()],
)

Querying across levels

The search_level option on query() (and on the query builder) selects how the query is matched:

"chunks" (default)

The normal chunk search. Unchanged by hierarchical embeddings.

"sections"

Search the section index directly — one result per matching section. Requires hierarchical_embeddings=True.

"documents"

Coarse document-index retrieval against the per-document centroid.

"fused"

Blend chunk and section retrieval into a single ranking (see Fused retrieval). Requires hierarchical_embeddings=True.

# Search the section index directly — one result per matching section
results = db.query(
    "how do I rotate the API key?",
    search_level="sections",
)

for r in results:
    print(r.score, r.metadata["section_heading"], r.metadata.get("heading_path"))

Section results carry type="section" and expose the section’s structure in their metadata (section_heading, section_level, section_index, plus anything added by section metadata extractors) along with a ChunkPosition spanning the section. Document-level results carry type="document".

Choosing the level and the unit separately

search_level decides how the query is matched; return_type decides what you get back. They are independent. Left alone, return_type follows the level — search_level="sections" answers in sections, as above — but you can ask for a different unit:

# Rank whole documents by their single best-matching section.
results = db.query(
    "how do I rotate the API key?",
    search_level="sections",
    return_type="documents",
)
for r in results:
    print(r.score, r.id)   # r.type == "document"

Each document is scored by its strongest section, so a document with one excellent section outranks one that is vaguely on-topic throughout. This is the same roll-up search_level="fused" performs with return_type="documents", without the chunk leg. search_level="sections" reports "sections" or "documents"; search_level="documents" reports only "documents". Anything else raises ValueError rather than quietly answering in the wrong unit.

Metadata filters apply at every level: for section, document, and fused searches the filter is matched against the parent document’s metadata.

Fused retrieval

search_level="fused" is the mode that improves retrieval quality rather than just changing granularity. It runs both a chunk search and a section (raw-span) search for the query, maps every hit up to its target unit, and merges the two rankings:

# Fuse chunk + section signal; return the best documents.
docs = db.query(
    "what latency does the gateway add under load?",
    search_level="fused",
    return_type="documents",   # or "sections"
    section_weight=0.65,       # weight on the section leg (0–1)
)

It supports two targets via return_type:

  • "documents" — a chunk hit and a section hit both credit their parent document; the fused score ranks documents. This is where the measured win lives.

  • "sections" — chunk hits are rolled up to their containing section (best chunk score wins) and fused with the section-index hits; the fused score ranks sections.

Why fusion beats either level alone

The two levels make different mistakes. A chunk match is precise but narrow — it can miss a document whose relevance is spread thinly across a section rather than concentrated in one passage. A section match captures that diffuse relevance but is coarse. Fusing them recovers documents that neither level ranks well alone, and on real long documents fusing raw-span sections with chunks beat chunk-only retrieval at the document target. The absolute gains on real documents are modest — a genuine quality-per-cost judgement, not a slam dunk — and much larger on section-dense corpora. See Raw-Span Hierarchical Retrieval: Method and Evaluation for the per-corpus numbers.

The fusion itself is relative-score fusion: each leg’s scores are min-max normalised within that query’s own candidate pool (putting the chunk cosine and the section cosine on a common 0–1 scale), then blended by section_weight. This is the same normalise-then-blend rule LocalVectorDB uses for hybrid vector+keyword search.

Tuning section_weight

section_weight is the weight on the section leg:

  • 0.0 — chunk-only (fusion contributes nothing).

  • 1.0 — section-only.

  • 0.5 — an equal blend.

The default is 0.65, tuned by sweeping the weight on the real long-document set: leaning slightly toward the section leg retrieved best there. Equal-weight fusion (0.5) still beats chunk-only everywhere we measured, but it is suboptimal when one leg clearly dominates — an equal blend drags a strong section score back toward a weaker chunk score. If your corpus is more like a collection of short, single-topic passages than long structured documents, a lower section_weight (closer to chunk-only) is the safer choice; for long, well-sectioned documents, the default or slightly higher works best.

Treat section_weight as corpus-dependent and tune it on your own relevance judgments if you have them. Because fused scores are relative to each query’s candidate pool (exactly like hybrid search), a score_threshold on a fused query behaves as a rank-position cutoff, not an absolute match-quality gate.

The query builder exposes the same controls:

results = (
    db.query_builder()
    .search("network timeout errors")
    .search_level("fused", section_weight=0.65)
    .filter("product", "gateway")
    .limit(5)
    .execute()
)

Choosing a level: match granularity to the answer

The single most important rule from the experiments: retrieval granularity should match relevance granularity.

  • When the answer is a single passage, the chunk level is unbeatable and adding coarser levels only dilutes it. Fusion does not help here; use the default search_level="chunks".

  • When the answer is a region of a long document — the relevant material is spread across a section rather than sitting in one sentence — a section-aware representation wins, and search_level="fused" is the mode to use.

So fused retrieval is a tool for long, structured corpora (papers, manuals, books, reports), not for short-passage collections where chunk retrieval already has the right granularity.

Limitations

Raw-span needs a long-context embedding model. A section can be much larger than a chunk. If a section fits inside the encoder’s context window it is embedded whole; if it is longer, LocalVectorDB embeds it in overlapping windows and mean-pools the results rather than truncating, so the whole section is represented. But a short-context encoder is a poor fit: on a 512-token model, long sections are dominated by pooling artefacts and raw-span can lose to the centroid. Use a long-context embedding model (e.g. OpenAI text-embedding-3-* at 8k tokens, nomic-embed-text, or another 8k+ encoder) when you enable "rawspan". Raw-Span Hierarchical Retrieval: Method and Evaluation reports results across OpenAI text-embedding-3-small and several local encoders.

The document level carries little signal. Whole-document-centroid retrieval (search_level="documents") was the weakest arm in every experiment — a single vector for a long document is too blurry to rank well. Fused retrieval deliberately ignores the document index. Prefer "sections" or "fused" for coarse retrieval, not "documents".

Hierarchical vectors are built on the synchronous ingest path only. Section and document vectors are computed during a normal synchronous upsert. The async ingest path and the “from chunks” ingest path do not build them; if you ingest through those, run rebuild_hierarchical_embeddings() afterwards to populate the sidecar indices.

Fused retrieval is local-only for now. search_level="fused" is a LocalVectorDB feature. Remote databases (RemoteVectorDB) raise NotImplementedError for a fused query, and the streaming/cursor query paths do not support it — use a materialised query() instead.

Gains are real but modest, and corpus-dependent. On real long documents the improvement is a low single-digit nDCG@10 gain (see Raw-Span Hierarchical Retrieval: Method and Evaluation for exact figures). Measure on your own data before committing to a section_weight far from the default, and keep chunk-only retrieval for short-passage corpora.

Rebuilding indices for existing databases

Section and document vectors are computed during ingestion. If you enable hierarchical embeddings on a database that already contains documents, switch a database from centroid to raw-span, or need to rebuild after corruption, call rebuild_hierarchical_embeddings():

db = VectorDB(
    "existing_db", "./vector_storage",
    hierarchical_embeddings=True,
    section_vector_strategy="rawspan",   # e.g. migrating from centroid
)
db.rebuild_hierarchical_embeddings()      # re-detects sections and rebuilds both indices

This re-runs section detection over every stored document and rebuilds both sidecar indices. With "centroid" it recomputes the means from the existing chunk embeddings without re-embedding; with "rawspan" it re-embeds each section’s text (embedding-provider calls, one batch per document). The document-level vector is a centroid in both cases. It raises ValueError if hierarchical_embeddings is not enabled.

Notes

  • Backward compatible — leaving hierarchical_embeddings=False (the default) preserves the original single-index behaviour exactly, and the default chunk-only retrieval path is unchanged even when hierarchical embeddings are enabled.

  • Persistence — the hierarchical_embeddings and section_vector_strategy settings are saved with the database; reopening re-loads the section/document sidecar indices automatically. Renaming a database via the CLI moves the sidecar files too.

  • Embedding cost — "centroid" adds no embedding-provider calls; "rawspan" adds one section embedding per section at ingest (sections are far fewer than chunks, so the overhead is modest).