localvectordb.section_detection module

Section detection for hierarchical document embeddings.

This module provides a SectionDetector that identifies section boundaries in documents based on configurable patterns (e.g., markdown headers). Sections are an overlay on top of existing chunking - they group chunks by document structure for mid-level retrieval.

localvectordb.section_detection.find_code_fence_spans(text: str) → List[tuple[int, int]]

Return (start, end) character spans covered by fenced code blocks.

Both backtick (```) and tilde (~~~) fences are recognised, including unterminated fences (which extend to end of text) following CommonMark behaviour. Spans are returned in document order and are used to suppress header matches that fall inside code, so example Markdown or shell snippets in extracted documents do not create spurious section boundaries.

class localvectordb.section_detection.SectionDetector(pattern: str = '^(#{1,6})\\s+(.+)$')

Bases: object

Detects section boundaries in documents using regex patterns.

By default, detects markdown-style headers (# through ######). Headers inside fenced code blocks are ignored, so example snippets in extracted Markdown do not create spurious sections. Custom patterns can be provided for other document formats.

Parameters:

pattern (str) – Regex pattern for detecting section headers. Must use MULTILINE mode. The pattern should have two capture groups: - Group 1: heading level indicator (e.g., ‘#’ characters) - Group 2: heading text

__init__(pattern: str = '^(#{1,6})\\s+(.+)$')
detect_sections(text: str) → List[SectionBoundary]

Detect section boundaries in the given text.

Parameters:

text (str) – The full document text to scan for sections.

Returns:

List of detected section boundaries, ordered by position. Text before the first header becomes a “preamble” section (index 0, heading=None).

Return type:

List[SectionBoundary]

static assign_chunks_to_sections(chunks: List[Chunk], sections: List[SectionBoundary]) → Dict[int, List[int]]

Map each chunk to the single section containing its midpoint.

Midpoint attribution: each chunk is credited to exactly one section, the one holding the chunk’s centre. This is the owner relation used for section centroids (where it measured ~+0.02 nDCG over crediting every overlapped section). For the complete chunk↔section overlap relation — which every section participates in, so none is unreachable — see assign_chunks_to_sections_overlap().

Uses binary search for efficiency with large numbers of chunks/sections.

Parameters:

  • chunks (List[Chunk]) – The chunks to assign to sections.

  • sections (List[SectionBoundary]) – The detected section boundaries.

Returns:

Mapping from section index to list of chunk indices (chunk.index values).

Return type:

Dict[int, List[int]]

static assign_chunks_to_sections_overlap(chunks: List[Chunk], sections: List[SectionBoundary]) → Dict[int, List[int]]

Map each section to every chunk whose span overlaps it.

Overlap attribution: a chunk larger than a section spans several of them, and midpoint attribution credits only one — at the shipped chunk_size=500 over real corpora that left ~40% of sections owning no chunk at all, making them unreturnable by any chunk-level roll-up. Because chunks tile the document, every non-empty section overlaps at least one chunk, so this relation reaches every section by construction.

Sections are non-overlapping and ordered, so each chunk’s overlapping sections form a contiguous run located by binary search.

Parameters:

  • chunks (List[Chunk]) – The chunks to assign to sections.

  • sections (List[SectionBoundary]) – The detected section boundaries.

Returns:

Mapping from section index to the list of chunk indices (chunk.index values) whose spans overlap that section.

Return type:

Dict[int, List[int]]

static compute_section_content_hash(text: str, section: SectionBoundary) → str

Compute SHA-256 hash of a section’s content.