localvectordb.section_detection module

Section detection for hierarchical document embeddings.

This module provides a SectionDetector that identifies section boundaries in documents based on configurable patterns (e.g., markdown headers). Sections are an overlay on top of existing chunking - they group chunks by document structure for mid-level retrieval.

localvectordb.section_detection.find_code_fence_spans(text: str) → List[tuple[int, int]]

Return (start, end) character spans covered by fenced code blocks.

Both backtick (```) and tilde (~~~) fences are recognised, including unterminated fences (which extend to end of text) following CommonMark behaviour. Spans are returned in document order and are used to suppress header matches that fall inside code, so example Markdown or shell snippets in extracted documents do not create spurious section boundaries.

class localvectordb.section_detection.SectionDetector(pattern: str = '^(#{1,6})\\s+(.+)$')

Bases: object

Detects section boundaries in documents using regex patterns.

By default, detects markdown-style headers (# through ######). Headers inside fenced code blocks are ignored, so example snippets in extracted Markdown do not create spurious sections. Custom patterns can be provided for other document formats.

Parameters:

pattern (str) – Regex pattern for detecting section headers. Must use MULTILINE mode. The pattern should have two capture groups: - Group 1: heading level indicator (e.g., ‘#’ characters) - Group 2: heading text

__init__(pattern: str = '^(#{1,6})\\s+(.+)$')

detect_sections(text: str) → List[SectionBoundary]

Detect section boundaries in the given text.

Parameters:

text (str) – The full document text to scan for sections.

Returns:

List of detected section boundaries, ordered by position. Text before the first header becomes a “preamble” section (index 0, heading=None).

Return type:

List[SectionBoundary]

static assign_chunks_to_sections(chunks: List[Chunk], sections: List[SectionBoundary]) → Dict[int, List[int]]

Map each chunk to its containing section by position overlap.

Uses binary search for efficiency with large numbers of chunks/sections.

Parameters:

• chunks (List[Chunk]) – The chunks to assign to sections.

• sections (List[SectionBoundary]) – The detected section boundaries.

Returns:

Mapping from section index to list of chunk indices (chunk.index values).

Return type:

Dict[int, List[int]]

static compute_section_content_hash(text: str, section: SectionBoundary) → str

Compute SHA-256 hash of a section’s content.