localvectordb.chunking module

Position-tracking chunking system for LocalVectorDB v1.0

This module provides chunkers that track exact positions in the original document, enabling perfect reconstruction and precise highlighting.

class localvectordb.chunking.PositionTrackingChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: ABC

Base class for chunkers that track exact positions

__init__(max_tokens: int = 500, overlap: int = 0, **kwargs)

abstractmethod chunk(text: str) → List[Chunk]

Split text into chunks with position tracking

count_tokens(text: str) → int

Count tokens in text.

Uses encode_ordinary rather than encode: the latter runs a special-token scan (regex) on every call to enforce disallowed_special, which is pure overhead here and would also raise on text that merely contains a special-token literal. encode_ordinary skips that check and yields identical counts for ordinary text.

class localvectordb.chunking.SentenceChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by sentences while preserving boundaries

sentence_pattern = re.compile('(?<=[.!?])\\s+|(?<=[.!?]")(?=\\s+[A-Z])|(?<=[.!?])\\n+', re.MULTILINE)

chunk(text: str) → List[Chunk]

Split text by sentences

class localvectordb.chunking.TokenChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by token boundaries with position tracking

chunk(text: str) → List[Chunk]

Split text by token boundaries

class localvectordb.chunking.WordChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by word boundaries while preserving all whitespace

chunk(text: str) → List[Chunk]

Split text by word boundaries while preserving whitespace

class localvectordb.chunking.LineChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by line boundaries

chunk(text: str) → List[Chunk]

Split text by line boundaries

class localvectordb.chunking.CharChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by character boundaries with exact position tracking

chunk(text: str) → List[Chunk]

Split text by character boundaries

class localvectordb.chunking.ParagraphChunker(max_tokens: int = 500, overlap: int = 0, **kwargs)

Bases: PositionTrackingChunker

Chunk by paragraph boundaries

paragraph_pattern = re.compile('\\n\\s*\\n', re.MULTILINE)

chunk(text: str) → List[Chunk]

Split text by paragraph boundaries

class localvectordb.chunking.SectionChunker(max_tokens: int = 500, overlap: int = 0)

Bases: PositionTrackingChunker

Chunk by section headers (markdown-style)

header_pattern = re.compile('^(#{1,6})\\s+(.+)$', re.MULTILINE)

__init__(max_tokens: int = 500, overlap: int = 0)

chunk(text: str) → List[Chunk]

Split text by section headers

class localvectordb.chunking.CodeBlockChunker(max_tokens: int = 500, overlap: int = 0, language: str | None = None, **kwargs)

Bases: PositionTrackingChunker

Chunk code while preserving logical code blocks

__init__(max_tokens: int = 500, overlap: int = 0, language: str | None = None, **kwargs)

chunk(text: str) → List[Chunk]

Split code while preserving logical blocks

class localvectordb.chunking.ChunkerFactory

Bases: object

Factory for creating chunkers

CHUNKERS: dict[str, Type[PositionTrackingChunker]] = {'characters': <class 'localvectordb.chunking.CharChunker'>, 'code-blocks': <class 'localvectordb.chunking.CodeBlockChunker'>, 'lines': <class 'localvectordb.chunking.LineChunker'>, 'paragraphs': <class 'localvectordb.chunking.ParagraphChunker'>, 'sections': <class 'localvectordb.chunking.SectionChunker'>, 'sentences': <class 'localvectordb.chunking.SentenceChunker'>, 'tokens': <class 'localvectordb.chunking.TokenChunker'>, 'words': <class 'localvectordb.chunking.WordChunker'>}

classmethod create_chunker(method: str | Type[PositionTrackingChunker], max_tokens: int = 500, overlap: int = 0, **kwargs) → PositionTrackingChunker

Create a chunker instance

classmethod list_methods() → List[str]

List available chunking methods

localvectordb.chunking.reconstruct_document(chunks: List[Chunk], original_length: int) → str

Reconstruct a document exactly from the chunks a chunker produced.

The general-purpose chunkers (sentences – the default – paragraphs, words, lines, characters, tokens, sections) emit chunks whose [start, end) spans cover [0, original_length) with no gaps (overlap is fine – an overlapped position is simply filled twice with the same character), so the return value equals the original text character-for-character. sentences and paragraphs achieve this by folding each inter-unit separator onto the preceding chunk’s span rather than leaving it uncovered.

The code-blocks chunker is the exception: it is specialised for splitting source code and its multi-chunk path is line-oriented, so it guarantees exact reconstruction only when the whole input fits a single chunk. A chunk list assembled by hand with gaps between spans will likewise leave those positions blank.