Skip to main content

logsumexp

Function logsumexp 

Source
pub fn logsumexp(logprobs: &[f64]) -> f64
Expand description

Combine logprobs of several tokens that mean the same answer, in probability space.

Tokenisers spell one answer many ways. Measured on gemma4:26b, asking for a yes/no in German surfaced Ja, JA, Ja and ja as four separate entries, each holding part of the mass — and Nein was not a token at all, arriving as Ne + in. Single-letter labels avoid the split entirely, but a leading-space variant such as " A" still shows up on some tokenisers, so every variant of a label is folded together here rather than one of them being picked and the rest discarded.