Trimwise treats truncation as evidence selection under a hard budget. It does not rewrite, summarize, or generate replacement text. Instead, it finds meaningful units in the supplied source, estimates their usefulness, chooses complementary units, restores source order, and measures the complete excerpt before returning it.

source text
    │
    ├─ validate and measure
    ├─ segment into exact source spans
    ├─ score spans for salience or query relevance
    ├─ select useful, less-repetitive evidence
    ├─ compose in original order with affordable gap markers
    └─ remeasure ──> TrimResult

Trimwise operates only on the string you provide. It does not search, retrieve documents, query an index, or create a RAG pipeline.

The five-stage pipeline¶

Every strategy follows the same high-level path:

1. Measure: validate the request and return immediately when the source already fits.
2. Segment: map Markdown or plain text into complete, non-overlapping source units.
3. Rank: estimate usefulness with document centrality, BM25, embeddings, or hybrid fusion.
4. Select: balance relevance against similarity to evidence already retained.
5. Compose: restore source order, add affordable omission markers, and enforce the exact limit.

The strategy changes ranking and selection behavior. Measurement, source fidelity, composition, fallback, and the final budget guarantee stay shared.

Stage 1: validate and measure¶

Trimwise resolves public arguments before doing document work:

• auto becomes structural without a nonblank query.
• auto becomes lexical with a nonblank query.
• Explicit lexical, semantic, and hybrid calls require a nonblank query.
• Explicit structural calls ignore a supplied query for ranking.
• The text, limit, budget unit, strategy, query, and custom counter are validated.

The complete input is then measured in the requested unit:

If limit == 0, Trimwise returns empty text after measuring the input. If the input count is at most the limit, it returns the original string exactly and stops before Markdown parsing, ranking, callback invocation, or FastEmbed loading.

from trimwise import Trimmer

source = "  # Heading\r\n\r\nExact text.  "
result = Trimmer().trim(source, len(source), unit="characters")

assert result.text == source
assert result.trimmed is False

This fast path matters for agentic pipelines that trim many already-small sources: a configured semantic backend does not turn every call into an embedding call.

Stage 2: map structure without rewriting source¶

For oversized input, Trimwise creates a fresh markdown-it-py parser for that call. It uses the CommonMark rules plus table support. Parser block tokens provide source line ranges; Trimwise maps those line ranges back to character offsets and slices the original string.

Each candidate stores:

start offset  ─┐
end offset     ├─ exact source identity
source text   ─┘
block kind
section number
nearest preceding heading

The important invariant is:

candidate.text == original_text[candidate.start:candidate.end]

No Markdown renderer is involved, so selected wording, whitespace, indentation, link syntax, list markers, HTML, and code-fence language tags are not normalized on the way out.

Recognized source units¶

Whitespace-only input becomes one raw source unit. Empty input produces no candidates and normally returns through the fitting-input path.

Parser spans can overlap or nest. Trimwise removes contained duplicates, prefers richer block types for identical spans, clips unusual partial overlaps, and inserts raw units for nonblank ranges that no parser block covered. The final candidate list is ordered and non-overlapping.

Sections and headings¶

Every recognized heading starts a section. Following candidates remember the nearest heading until another heading begins. That relationship supports two later behaviors:

• Structural mode can give each section an initial share of the budget.
• Query-aware modes can attach the selected passage's heading when both still fit.

One enormous plain-text paragraph¶

Markdown may see a long plain-text document as one paragraph. A single candidate gives structural ranking nothing to compare, so Trimwise expands that sole paragraph into independently rankable complete sentences or source lines when boundaries exist.

Sentence boundaries include ordinary ., !, and ?, plus the Unicode ellipsis, ideographic full stop, and full-width exclamation and question marks used in CJK text. Source lines also provide boundaries when punctuation is absent.

from trimwise import Trimmer

source = (
    "Opening context. "
    "Repeated operating detail alpha beta. "
    "A decision was approved for September 2026. "
    "Repeated operating detail alpha gamma. "
    "Closing context."
)

result = Trimmer().trim(source, 85, unit="characters")

assert result.output_count <= 85
assert result.text.startswith("Opening context.")
assert result.text.endswith("Closing context.")

This expansion currently belongs to structural selection. If the source is one unpunctuated line, there is still no smaller complete unit to rank; a tight budget eventually requires exact prefix fallback.

Stage 3A: build ranking context¶

A short candidate can be ambiguous in isolation. For scoring only, Trimwise gives each candidate a small local window containing:

1. The candidate itself, placed first.
2. Its nearest section heading, when present.
3. The previous candidate from the same section.
4. The candidate in its source position.
5. The following candidate from the same section.

Duplicate indexes are removed from the window. The candidate is intentionally repeated at the front when a larger window exists, preserving a stronger signal for words found directly in that candidate rather than only in a neighbor.

For example, It increased by 18%. becomes easier to score when the ranking text also contains a heading such as ## Database latency and the adjacent explanation. Selection still retains only exact source candidates; the contextual window is never emitted as synthetic output.

Ranking-only normalization¶

Structural and lexical scoring normalize their ranking text with:

• Unicode NFKC normalization.
• Unicode case folding.
• Collapsed whitespace.
• One consistent leading space before tiktoken encoding.

The normalized text becomes tiktoken subword IDs. Using subwords instead of whitespace-only terms keeps lexical evidence available in scripts such as Chinese and Japanese and handles technical identifiers more consistently. Original source slices are never normalized.

Stage 3B: estimate usefulness¶

The resolved strategy provides one primary score for every candidate:

These papers motivate individual ideas. Trimwise adapts them to source-backed fragments, exact budgets, structural coverage, and deterministic composition rather than reproducing any complete paper system.

Structural centrality¶

Trimwise builds an L2-normalized sparse TF-IDF vector for every contextual candidate. Term frequency is adjusted by candidate length; inverse document frequency rewards terms that do not appear in every candidate. The mean of all candidate vectors becomes the document centroid, and candidate-centroid cosine similarity estimates how representative each unit is.

Centrality is useful without a query, but it can undervalue a rare critical fact. Structural anchors, section shares, MMR, and the fact-like signal reduce that risk without claiming to remove it.

BM25 relevance¶

Lexical mode compares query subword IDs with every contextual candidate using Robertson IDF and BM25 length normalization. Rare matching terms contribute more than terms repeated throughout the document, making IDs, error strings, dates, and names strong evidence.

BM25 matches lexical evidence; it does not understand that two differently worded sentences can mean the same thing.

Semantic relevance¶

Semantic mode obtains one query vector and one vector per contextual candidate. Trimwise converts them to one float32 matrix, validates dimensions and finite values, normalizes each row once, and computes all query-passage cosine scores with a matrix-vector operation.

Semantic quality depends on the supplied embedding model. Similarity measures representation, not truth, currency, authority, or answer completeness.

Hybrid fusion¶

When both lexical and semantic score rows are finite and contain variation, Trimwise min-max normalizes each row and blends them equally:

hybrid = 0.5 × normalized BM25 + 0.5 × normalized semantic score

If either row is flat or non-finite, score magnitude cannot be compared meaningfully. Trimwise then uses stable one-indexed Reciprocal Rank Fusion with k=60, following RRF. RRF is a defensive fallback, not the normal hybrid path.

Small structural and fact-like signal¶

After min-max normalization, the strategy's primary score receives 90% of final relevance. The remaining 10% is the mean of four binary indicators:

• Candidate is a heading or belongs under one.
• Candidate contains a URL.
• Candidate contains a number or date-like value.
• Candidate contains a snake_case or camelCase-style identifier.

This language-neutral signal helps operational evidence survive when primary scores are close. It does not override a large relevance difference, and it cannot determine whether a number, URL, or identifier is useful to the task.

Stage 4: choose complementary evidence¶

Trimwise selects candidates greedily. After each choice, it updates the greatest similarity between every remaining candidate and anything already selected. The next choice maximizes:

mmr_lambda × relevance − (1 − mmr_lambda) × maximum selected similarity

The default mmr_lambda=0.7 gives relevance 70% of the objective and the similarity penalty 30%. This follows the relevance-versus-redundancy idea from Maximal Marginal Relevance.

Structural and lexical modes use TF-IDF cosine similarity for MMR. Semantic and hybrid modes use nonnegative embedding cosine similarity and update all remaining similarities with vectorized matrix operations.

MMR is a duplicate-control proxy, not factual analysis. Similar wording can describe separate facts, while different wording can repeat one fact.

Structural selection path¶

Structural mode selects in three phases:

1. Protect anchors. Try the first and last complete candidates together. If they do not fit, try the more relevant one, breaking equal-score ties toward the first.
2. Cover sections. Divide the remaining measured capacity equally across all Markdown sections and select fitting MMR candidates inside each provisional share.
3. Redistribute globally. Spend unused room on the strongest fitting candidates from any section.

The shares are provisional, not hard quotas. A section whose units are too large does not waste its unused room; later global selection can use it elsewhere.

This is why structural mode is not a 50/25/25 slice. Position provides orientation through anchors, while centrality, section coverage, similarity, and exact cost decide the rest.

Query-aware selection path¶

Lexical, semantic, and hybrid modes do not force opening or closing anchors. They first remove standalone heading candidates when non-heading evidence exists, then bound the candidate pool:

1. Order candidates by the strategy's primary query score.
2. Search the first 90% of score gaps for the largest drop, avoiding an extreme tail outlier.
3. Keep candidates through that boundary plus a five-candidate recall buffer.
4. Run MMR inside the bounded pool.
5. Try each selected passage with its nearest heading; if the pair is too large, try the passage alone.

The cutoff depends on score shape rather than an absolute BM25 or cosine threshold. This matters because different models and queries produce different score scales. It adapts the central idea of query-specific evidence counts from Adaptive-k context selection without implementing that paper's exact threshold method.

When the bounded pool is exhausted, query-aware selection stops even if capacity remains. The limit means “at most,” not “fill with weak evidence.” The five-candidate buffer protects recall, so the pool is not a strict relevance guarantee.

Deterministic ties and complexity¶

Equal ranking outcomes break toward earlier source candidates. Given the same text, configuration, counter behavior, and semantic vectors, Trimwise selection is deterministic.

The sparse MMR path keeps the straightforward exact O(selected × candidates) comparison loop. Semantic and hybrid modes maintain the same exact objective but update remaining similarities in a vectorized NumPy operation. Approximate-neighbor selection is intentionally deferred until profiling representative workloads proves it necessary.

Stage 5: compose within the exact budget¶

Ranking order never becomes output order. For every proposed addition, Trimwise:

1. Sorts all selected candidates by their original source position.
2. Inserts exact whitespace for untouched gaps or minimal separators for omitted nonblank gaps.
3. Composes the retained fragments without optional omission markers.
4. Measures that complete proposal.
5. Rejects the addition if retained content and required separators exceed the limit.
6. Otherwise, tries optional omission markers one gap at a time in source-gap order.
7. Carries the retained original-input ranges into the final result.

This trial composition accounts for the actual cost of headings, separators, and markers instead of estimating candidate cost in isolation.

Leading, internal, and trailing gaps¶

For every nonblank omitted range, Trimwise can place the configured marker—[…omitted…] by default—before, between, or after retained fragments. Markers are attempted only after the retained content already fits.

If a marker would exceed the limit, Trimwise leaves it out rather than discarding source evidence. Internal nonblank gaps still receive a minimal newline separator so distant fragments do not run together. Marker boundaries use no more than one blank line while preserving retained fragment content and whitespace.

from trimwise import TrimConfig, Trimmer

source = "FIRST important.\n\nMiddle filler.\n\nLAST important."
trimmer = Trimmer(TrimConfig(omission_marker="<cut>"))
result = trimmer.trim(source, 39, unit="characters")

assert result.output_count <= 39

Omission markers are the only configured explanatory text Trimwise adds. They indicate a source gap; they are not summaries of what was removed.

When no complete candidate fits¶

If selection cannot retain any complete candidate, Trimwise chooses the strongest candidate by final relevance, breaking ties toward the earlier source position, and progressively shrinks it.

For ordinary text, fallback prefers:

1. The longest fitting complete paragraph prefix.
2. The longest fitting complete sentence or source-line prefix.
3. The longest exact source prefix accepted by the active measurer.

This preserves readable boundaries when possible without violating the limit.

Closed and unclosed code fences¶

For a genuinely closed backtick or tilde fence, Trimwise checks that the closing marker uses the same character and at least the opening marker's length. If the original opening-and-closing shell fits, it removes whole body lines until the fenced fragment fits.

from trimwise import Trimmer

source = "```python\n" + "print('large')\n" * 20 + "```\n"
result = Trimmer().trim(source, 35, unit="characters")

assert result.text.startswith("```python\n")
assert "\n```" in result.text[len("```python\n") :]
assert result.output_count <= 35

If even the fence shell is too large, the ordinary source-prefix fallback applies. A fence that was unclosed in the source remains unclosed; Trimwise does not invent a closing marker.

Exact prefix measurement¶

Character fallback can slice directly by code-point position. Word fallback uses complete whitespace-delimited spans. Built-in token fallback uses tiktoken offsets when the encoded text round-trips exactly; otherwise it scans source prefixes.

A custom token counter may be non-monotonic—for example, adding one character can merge several tokens. Trimwise therefore scans candidate prefixes rather than assuming a binary-search boundary, keeping the final fit exact at the cost of more callback calls on this rare path.

After content fallback, affordable leading and trailing omission markers are tried without removing the retained fragment.

Final measurement and result¶

Trimwise measures the finished string one final time. If an internal bug ever produced a result above the requested limit, the implementation raises RuntimeError instead of returning an invalid TrimResult.

The normal result records:

text          final extractive excerpt
input_count   original measured size
output_count  final measured size
limit         caller-defined ceiling
unit          concrete budget unit
strategy      resolved concrete strategy
trimmed       whether output text differs from input
spans         ordered original-input ranges retained in the output

Span offsets follow Python string slicing: starts are inclusive and ends are exclusive. Adjacent source-backed ranges are merged; overlapping ranges cannot arise from the nonoverlapping source candidates. Generated omission markers and minimal separators have no source span.

The guarantee applies to result.text, not to the larger prompt assembled around it. Source labels, instructions, examples, tool schemas, separators, and model output still need their own context space.

What source fidelity means¶

Retained candidate text always comes from exact source slices and returns in original order. The complete result may differ from one contiguous source substring because Trimwise can:

• Select non-adjacent fragments.
• Add the configured omission marker where it fits.
• Add minimal newlines between separated fragments.
• Attach an original section heading to a selected query-aware passage.
• Shorten the strongest candidate through source-prefix fallback when no complete unit fits.

Trimwise never paraphrases retained text, synthesizes a transition, repairs malformed Markdown, or combines distant facts into a new sentence.

Research-backed components and product behavior¶

Research provides scoring and selection ideas; it does not provide Trimwise's complete product contract.

Trimwise adds Markdown source spans, section shares, anchors, fact-like signals, contextual scoring, source-order composition, omission behavior, exact measurement, and progressive fallback. Those adaptations are implementation choices tested by the package; citing the component research does not prove that Trimwise is optimal for every downstream LLM task.

Important algorithmic limits¶

• Usefulness scores do not understand factual truth, authority, recency, or completeness.
• MMR similarity discourages repeated-looking evidence but does not prove factual diversity.
• Greedy selection does not solve a global token-cost knapsack or facility-location optimum.
• Structural centroid scoring can undervalue rare but critical details.
• Lexical scoring can miss paraphrases; semantic scoring inherits model quality and bias.
• A single unpunctuated line eventually requires prefix truncation.
• Extractive output cannot synthesize two distant facts into one denser sentence.

At extreme compression ratios, a trained token-level or abstractive compressor may preserve more task-relevant meaning per token. It also adds model cost and gives up Trimwise's exact-source guarantee. Evaluate the final LLM answers—not only the selected excerpts—on representative data.

Continue exploring¶

• Return to the Trimwise overview.
• Follow the Getting Started guide.
• Compare ranking paths in Choosing a Strategy.
• Configure models and callbacks in Semantic Models and Async Usage.
• Review all public settings in Configuration and API Reference.
• See deferred quality work in the roadmap.