Fact-Checking (Reverse RAG)

The localvectordb.validation module verifies LLM-generated text against the documents you have already stored. Where ordinary RAG retrieves context before generating an answer, fact-checking runs after: it breaks a piece of text into individual factual claims, searches your databases for evidence, and uses an LLM to decide whether each claim is supported, contradicted, or unrelated to the retrieved sources. This “reverse RAG” pass is a practical way to catch hallucinations before showing model output to a user.

Overview

The pipeline has three stages, all orchestrated by FactChecker:

  1. Claim extraction – the LLM splits the input text into atomic factual claims.

  2. Retrieval – each claim is used as a query against one or more LocalVectorDB instances (vector, keyword, or hybrid search).

  3. Polarity classification – the LLM judges each retrieved chunk against the claim and assigns a Polarity (SUPPORTS, CONTRADICTS, or UNRELATED) with a confidence score.

All of the public classes are re-exported from the top-level package:

from localvectordb import FactChecker, FactCheckResult, ClaimResult, Polarity

The provider adapters live one level down, in localvectordb.validation:

from localvectordb.validation import (
    AnthropicProvider,
    OpenAIProvider,
    GeminiProvider,
    LLMProvider,
)

Basic Usage

Construct a FactChecker with one or more databases and an LLM client, then call check():

import anthropic
from localvectordb import LocalVectorDB, FactChecker

db = LocalVectorDB("knowledge.db")
client = anthropic.Anthropic()  # reads ANTHROPIC_API_KEY from the environment

checker = FactChecker(databases=db, llm=client)

result = checker.check("The policy allows 10 days of PTO per year.")

print(f"Overall score: {result.overall_score:.2f}")
print(f"Contradictions: {result.has_contradictions}")

databases accepts either a single LocalVectorDB or a list of them. llm accepts an Anthropic, OpenAI, or Google GenAI client (the provider is auto-detected), or any object implementing the LLMProvider protocol.

The FactChecker constructor signature is:

FactChecker(
    databases,                     # LocalVectorDB | list[LocalVectorDB]
    llm,                           # an LLM client or LLMProvider
    model=None,                    # provider-specific default when None
    similarity_threshold=0.3,      # min retrieval score for a chunk to count
    min_grounding_score=0.7,       # min polarity confidence to call a claim grounded
    search_type="hybrid",          # "vector", "keyword", or "hybrid"
    top_k=5,                       # chunks retrieved per claim per database
    max_concurrent=5,              # claims processed concurrently
)

Async

check() is a thin synchronous wrapper around check_async(). In an async application, call the coroutine directly to avoid nesting event loops:

result = await checker.check_async(text)

Both methods share the same signature – check(text, sources=None) and check_async(text, sources=None) – and return a FactCheckResult.

Interpreting the Result

check() returns a FactCheckResult with the following fields:

  • claims – a list of ClaimResult objects, one per extracted claim.

  • overall_score – a float in [0.0, 1.0]. It is the mean claim confidence, but is forced to 0.0 if any claim is contradicted, and is 1.0 when no factual claims are detected.

  • has_contradictions – True if any claim was contradicted by a source.

  • citation_text – a human-readable summary of sources consulted, contradictions, and ungrounded claims.

  • annotated_text – the original text with inline [N] citation markers and appended footnotes for grounded claims, or None if no annotations could be placed.

Each ClaimResult carries:

  • claim – the extracted factual claim.

  • grounded – True if a supporting source was found above min_grounding_score.

  • confidence – polarity confidence for the best supporting chunk (0.0 when contradicted or ungrounded).

  • source_id – the document ID of the matched source (or None).

  • source_excerpt – the excerpt the LLM cited as evidence.

  • contradiction – True if the claim was contradicted.

  • polarity – a Polarity value (SUPPORTS, CONTRADICTS, UNRELATED).

  • similarity – the retrieval score of the matched chunk.

  • original_sentence – the sentence in the input text the claim came from.

  • database_name – the name of the database the source was found in.

result = checker.check(answer_text)

for claim in result.claims:
    status = (
        "CONTRADICTED" if claim.contradiction
        else "grounded" if claim.grounded
        else "ungrounded"
    )
    print(f"[{status}] {claim.claim}")
    if claim.source_id:
        print(f"    source: {claim.source_id} ({claim.database_name})")
        print(f"    excerpt: {claim.source_excerpt}")

# Human-readable summary of sources, contradictions, and ungrounded claims
print(result.citation_text)

# Inline-cited version of the original text (may be None)
if result.annotated_text:
    print(result.annotated_text)

The Polarity enum is a str enum, so its members compare and serialise as plain strings:

from localvectordb import Polarity

assert Polarity.SUPPORTS == "supports"
assert Polarity.CONTRADICTS.value == "contradicts"

Scoping to Source Documents

If you already know which documents were used to generate the text (for example, the chunks your RAG pipeline retrieved), pass their document IDs via sources. This enables a two-phase search:

  1. Scoped phase – only the listed source documents are searched. If a claim is supported here above min_grounding_score, the check short-circuits and returns immediately.

  2. Expanded phase – if the scoped search finds no support (or the claim is unsupported), the checker falls back to searching all documents across all databases. This catches claims that are unsupported by, or even contradicted by, the wider knowledge base.

# These were the documents the LLM was given as context
context_doc_ids = ["policy-2026", "handbook-benefits"]

result = checker.check(
    "The policy allows 10 days of PTO per year.",
    sources=context_doc_ids,
)

This is the recommended pattern for validating a specific RAG response: it verifies the answer against its own cited context first, then widens the net only when that context is insufficient.

Tuning Thresholds

Two thresholds control sensitivity:

  • similarity_threshold (default 0.3) is the minimum retrieval score for a chunk to be considered at all. Raise it to only classify high-relevance chunks; lower it to surface more candidate evidence.

  • min_grounding_score (default 0.7) is the minimum polarity confidence for a claim to be marked grounded. Raise it for stricter grounding; lower it to be more permissive.

search_type and top_k control retrieval breadth, and max_concurrent bounds how many claims are classified in parallel:

checker = FactChecker(
    databases=[db_docs, db_wiki],
    llm=client,
    similarity_threshold=0.4,   # stricter retrieval
    min_grounding_score=0.8,    # stricter grounding
    search_type="vector",       # pure semantic search
    top_k=8,                     # more evidence per claim
    max_concurrent=10,           # more parallelism
)

Selecting and Customising the LLM Provider

The provider is auto-detected from the client object’s module, so passing a native SDK client is usually all you need:

# Anthropic
import anthropic
checker = FactChecker(databases=db, llm=anthropic.Anthropic())

# OpenAI
import openai
checker = FactChecker(databases=db, llm=openai.OpenAI())

# Google GenAI
from google import genai
checker = FactChecker(databases=db, llm=genai.Client())

Both synchronous and asynchronous SDK clients are supported (for example anthropic.AsyncAnthropic or openai.AsyncOpenAI); the adapter detects which flavour it was given.

Override the model with the model parameter. When left as None, each provider uses its own default (claude-haiku-4-5-20251001 for Anthropic, gpt-4o-mini for OpenAI, gemini-2.0-flash for Gemini):

checker = FactChecker(
    databases=db,
    llm=anthropic.Anthropic(),
    model="claude-sonnet-4-5",
)

You can also construct a provider adapter explicitly:

from localvectordb.validation import AnthropicProvider

provider = AnthropicProvider(anthropic.Anthropic(), model="claude-sonnet-4-5")
checker = FactChecker(databases=db, llm=provider)

Custom Providers

To use any other LLM, implement the LLMProvider protocol – a single async complete method that takes a system prompt and a user prompt and returns the model’s text response:

class MyProvider:
    async def complete(self, system: str, user: str) -> str:
        # call your model however you like and return its text
        ...

checker = FactChecker(databases=db, llm=MyProvider())

Because LLMProvider is a runtime-checkable Protocol, any object with a matching complete signature is accepted without subclassing.

Local-Only in v0.1.0

Fact-checking is a local-only feature: it runs in-process against one or more LocalVectorDB instances via the FactChecker class shown above. The LocalVectorDB HTTP server does not expose fact-check endpoints, and RemoteVectorDB has no fact_check method. To fact-check against data served remotely, retrieve or open the database locally and construct a FactChecker around it.