localvectordb.chunking module

Position-tracking chunking system for LocalVectorDB v1.0

This module provides chunkers that track exact positions in the original document, enabling perfect reconstruction and precise highlighting.

class localvectordb.chunking.PositionTrackingChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: ABC

Base class for chunkers that track exact positions

__init__(max_tokens: int = 500, overlap: int = 0, **kwargs)

abstractmethod chunk(text: str) → List[Chunk]: Split text into chunks with position tracking

count_tokens(text: str) → int

Count tokens in text.

Uses encode_ordinary rather than encode: the latter runs a special-token scan (regex) on every call to enforce disallowed_special, which is pure overhead here and would also raise on text that merely contains a special-token literal. encode_ordinary skips that check and yields identical counts for ordinary text.

class localvectordb.chunking.SentenceChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by sentences while preserving boundaries

sentence_pattern = re.compile('(?<=[.!?])\\s+|(?<=[.!?]")(?=\\s+[A-Z])|(?<=[.!?])\\n+', re.MULTILINE)

chunk(text: str) → List[Chunk]: Split text by sentences

class localvectordb.chunking.TokenChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by token boundaries with position tracking

chunk(text: str) → List[Chunk]: Split text by token boundaries

class localvectordb.chunking.WordChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by word boundaries while preserving all whitespace

chunk(text: str) → List[Chunk]: Split text by word boundaries while preserving whitespace

class localvectordb.chunking.LineChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by line boundaries

chunk(text: str) → List[Chunk]: Split text by line boundaries

class localvectordb.chunking.CharChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by character boundaries with exact position tracking

chunk(text: str) → List[Chunk]: Split text by character boundaries

class localvectordb.chunking.ParagraphChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by paragraph boundaries

paragraph_pattern = re.compile('\\n\\s*\\n', re.MULTILINE)

chunk(text: str) → List[Chunk]: Split text by paragraph boundaries

class localvectordb.chunking.DelimiterChunker(max_tokens: int = 500, overlap: int = 0, delimiter: str = '\n\n', **kwargs)

Bases: PositionTrackingChunker

Split on a literal delimiter sequence, with a character-level fallback.

The document is cut on every occurrence of delimiter (a literal string, "\n\n" by default), and the resulting segments are packed into chunks up to max_tokens. A single segment that is itself larger than max_tokens becomes its own chunk and is then split character-by-character by the shared _ensure_chunks_within_limit() safeguard – so no chunk ever exceeds the limit even when the delimiter leaves an over-long piece.

Like SentenceChunker and ParagraphChunker, each segment owns the delimiter that follows it, so the [start, end) spans are contiguous and cover the whole document and reconstruct_document() is exact.

__init__(max_tokens: int = 500, overlap: int = 0, delimiter: str = '\n\n', **kwargs)

chunk(text: str) → List[Chunk]: Split text on the configured delimiter.

class localvectordb.chunking.SectionChunker(max_tokens: int = 500, overlap: int = 0)

Bases: PositionTrackingChunker

Chunk by section headers (markdown-style)

header_pattern = re.compile('^(#{1,6})\\s+(.+)$', re.MULTILINE)

__init__(max_tokens: int = 500, overlap: int = 0)

chunk(text: str) → List[Chunk]: Split text by section headers

class localvectordb.chunking.CodeBlockChunker(max_tokens: int = 500, overlap: int = 0, language: str | None = None, **kwargs)

Bases: PositionTrackingChunker

Chunk code while preserving logical code blocks

__init__(max_tokens: int = 500, overlap: int = 0, language: str | None = None, **kwargs)

chunk(text: str) → List[Chunk]: Split code while preserving logical blocks

class localvectordb.chunking.StructureChunker(max_tokens: int = 500, overlap: int = 0, min_fill: float = 0.35, heading_finder: Callable[[str], Iterable[int]] | None = None, **kwargs)

Bases: PositionTrackingChunker

Cut at the strongest human-authored boundary inside the size envelope.

Fixed-size chunkers cut at a token count and hope the cut lands somewhere harmless. This one collects the boundaries a human actually wrote – headings, blank-line paragraph breaks, line breaks, sentence ends – ranks them by strength, and cuts at the strongest one still inside the token budget. A chunk therefore ends where the document says a thought ends, and only falls back to an arbitrary cut when the span offers no boundary at all.

Why this shape (measured, see experiments/span-length-crossover-findings.md):

Retrieval quality rises monotonically as chunks shrink, but the return per stored vector collapses after ~500 tokens (Qasper/openai: 1000 -> 500 buys +0.056 nDCG, 500 -> 250 buys +0.017 for 2.4x the vectors). max_tokens is a budget knob; the boundary choice is what is free.
Past roughly 2k tokens a single embedding of a span is beaten by an average of its parts, so oversized chunks are not merely wasteful, they are worse. Keeping chunks inside the envelope is a correctness property, not a preference.

overlap must be 0, as for SectionChunker: these chunks are logical units, and re-emitting the tail of one inside the next would break the boundary alignment that is the entire point. Reconstruction is exact – chunks tile the source contiguously, separators folded onto the preceding chunk.

Parameters:

max_tokens (int) – Upper bound on chunk size; the cut is chosen at or before this budget.
min_fill (float) – Fraction of max_tokens a chunk must reach before a boundary is eligible, so a heading two lines in cannot produce a sliver. Set to 0 to always cut at the earliest strongest boundary.
heading_finder (Callable[[str], Iterable[int]], optional) –
Supplies extra heading-strength cut positions, for documents whose structure is not markdown. The built-in _HEADING recognises markdown ATX only; a corpus of plain-text contracts, RFCs, or statutes carries no # at all, and on such text this chunker silently degrades to a paragraph splitter (measured: no better than fixed-size, and worse once its under-fill is counted). Positions from this callable are added to the markdown ones, never replace them.

Detecting such headings well is not a matter of a wider regex – numbered headings collide with cross-references (“pursuant to Section 2.05”) and with a table of contents that repeats every heading verbatim, so a usable finder needs its own precision filtering. That belongs to the caller who knows the document family, not here.

STRENGTH_HEADING = 4

STRENGTH_PARAGRAPH = 3

STRENGTH_LINE = 2

STRENGTH_SENTENCE = 1

__init__(max_tokens: int = 500, overlap: int = 0, min_fill: float = 0.35, heading_finder: Callable[[str], Iterable[int]] | None = None, **kwargs)

chunk(text: str) → List[Chunk]: Split text into chunks with position tracking

class localvectordb.chunking.ChunkerFactory

Bases: object

Factory for creating chunkers

CHUNKERS: dict[str, Type[PositionTrackingChunker]] = {'characters': <class 'localvectordb.chunking.CharChunker'>, 'code-blocks': <class 'localvectordb.chunking.CodeBlockChunker'>, 'delimiter': <class 'localvectordb.chunking.DelimiterChunker'>, 'lines': <class 'localvectordb.chunking.LineChunker'>, 'paragraphs': <class 'localvectordb.chunking.ParagraphChunker'>, 'sections': <class 'localvectordb.chunking.SectionChunker'>, 'sentences': <class 'localvectordb.chunking.SentenceChunker'>, 'structure': <class 'localvectordb.chunking.StructureChunker'>, 'tokens': <class 'localvectordb.chunking.TokenChunker'>, 'words': <class 'localvectordb.chunking.WordChunker'>}

classmethod create_chunker(method: str | Type[PositionTrackingChunker], max_tokens: int = 500, overlap: int = 0, **kwargs) → PositionTrackingChunker: Create a chunker instance

classmethod list_methods() → List[str]: List available chunking methods

localvectordb.chunking.reconstruct_document(chunks: List[Chunk], original_length: int) → str

Reconstruct a document exactly from the chunks a chunker produced.

The general-purpose chunkers (sentences – the default – paragraphs, words, lines, characters, tokens, sections) emit chunks whose [start, end) spans cover [0, original_length) with no gaps (overlap is fine – an overlapped position is simply filled twice with the same character), so the return value equals the original text character-for-character. sentences and paragraphs achieve this by folding each inter-unit separator onto the preceding chunk’s span rather than leaving it uncovered.

The code-blocks chunker is the exception: it is specialised for splitting source code and its multi-chunk path is line-oriented, so it guarantees exact reconstruction only when the whole input fits a single chunk. A chunk list assembled by hand with gaps between spans will likewise leave those positions blank.