wijjit.elements.input.code_editor.SyntaxHighlighter

class wijjit.elements.input.code_editor.SyntaxHighlighter(language=None, theme='monokai', filename_hint=None)[source]

Manages syntax highlighting tokenization and caching.

This class handles tokenizing source code using Pygments lexers and caching the results for efficient rendering. It supports incremental re-tokenization when edits are made, and full re-tokenization for large changes.

Parameters:

language (str, optional) – Programming language name (e.g., “python”, “javascript”). Use “auto” for automatic detection, or None to disable highlighting.
theme (str, optional) – Color theme name (default: “monokai”)
filename_hint (str, optional) – Filename hint for language auto-detection (e.g., “main.py”)

Variables:

language (str or None) – Current language name
theme (str) – Current theme name
lexer (Lexer or None) – Pygments lexer instance
line_tokens (list of list of tuple) – Cached tokens per line: [[(token_type, text), …], …]

__init__(language=None, theme='monokai', filename_hint=None)[source]

Parameters:

language (str | None)
theme (str)
filename_hint (str | None)

Return type:

None

Methods

`__init__`([language, theme, filename_hint])
`detect_language`(text)	Detect the programming language from content and/or filename.
`get_line_tokens`(line_idx)	Get cached tokens for a specific line.
`invalidate_from_line`(line_idx)	Mark lines from the given index as needing re-tokenization.
`is_highlighting_enabled`()	Check if syntax highlighting is currently enabled.
`retokenize_incremental`(lines)	Re-tokenize only from the dirty line forward.
`set_language`(language)	Set the highlighting language.
`set_theme`(theme)	Set the color theme.
`tokenize_document`(lines)	Tokenize the entire document.

Attributes

lexer

Get the Pygments lexer, creating it if needed.

__getstate__()[source]

Get state for pickling, excluding the lexer.

Returns:: Picklable state dictionary
Return type:: dict

__setstate__(state)[source]

Restore state from pickle, recreating the lexer.

Parameters:: state (dict) – Pickled state dictionary
Return type:: None

__deepcopy__(memo)[source]

Create a deep copy, excluding the lexer.

Parameters:: memo (dict) – Memoization dictionary for deepcopy
Returns:: Deep copy of this highlighter
Return type:: SyntaxHighlighter

property lexer: Lexer | None

Get the Pygments lexer, creating it if needed.

Returns:: The Pygments lexer instance
Return type:: Lexer or None

detect_language(text)[source]

Detect the programming language from content and/or filename.

Parameters:: text (str) – Source code content
Returns:: Detected language name, or None if detection failed
Return type:: str or None

Notes

Filename-based detection is more reliable for short code snippets. Content-based detection works best with substantial code (50+ lines).

set_language(language)[source]

Set the highlighting language.

Parameters:: language (str or None) – Language name, “auto”, or None to disable highlighting
Return type:: None

set_theme(theme)[source]

Set the color theme.

Parameters:: theme (str) – Theme name (e.g., “monokai”, “dracula”)
Return type:: None

tokenize_document(lines)[source]

Tokenize the entire document.

Parameters:: lines (list of str) – Document lines to tokenize
Return type:: None

Notes

This performs a full tokenization of the document. For large documents, this may take 100-200ms. Results are cached per line for fast rendering.

invalidate_from_line(line_idx)[source]

Mark lines from the given index as needing re-tokenization.

Parameters:: line_idx (int) – Starting line index for invalidation
Return type:: None

Notes

This is used for incremental updates. When a line is edited, all lines from that point forward may need re-tokenization (due to multi-line constructs like strings and comments).

retokenize_incremental(lines)[source]

Re-tokenize only from the dirty line forward.

Parameters:: lines (list of str) – Document lines
Return type:: None

Notes

This is more efficient than full re-tokenization for small edits. However, for safety and simplicity with multi-line constructs, we re-tokenize from the dirty line to the end.

get_line_tokens(line_idx)[source]

Get cached tokens for a specific line.

Parameters:: line_idx (int) – Line index (0-based)
Returns:: List of (token_type, text) tuples for the line
Return type:: list of tuple

is_highlighting_enabled()[source]

Check if syntax highlighting is currently enabled.

Returns:: True if highlighting is enabled and a lexer is available
Return type:: bool