pub fn logsumexp(logprobs: &[f64]) -> f64Expand description
Combine logprobs of several tokens that mean the same answer, in probability space.
Tokenisers spell one answer many ways. Measured on gemma4:26b, asking for a yes/no in
German surfaced Ja, JA, Ja and ja as four separate entries, each holding part of the
mass — and Nein was not a token at all, arriving as Ne + in. Single-letter labels avoid
the split entirely, but a leading-space variant such as " A" still shows up on some
tokenisers, so every variant of a label is folded together here rather than one of them being
picked and the rest discarded.