From the paper: Step 5: Update confidence during sampling. At the end of _update_sample_logprobs(...) (after appending the logprob dict)
# logprobs[0] is the sampled token; use the remaining candidates
if len(logprobs) > 1:
new_conf = -sum(logprobs[1:]) / len(logprobs[1:])
else:
new_conf = 0.0
So the sampled token is exclued when compute confidence, but as i print the vllm log, logprobs[0]==logprobs[1]
Code:
if len(logprobs) > 1:
logger.info("check_conf_stop logprobs %f, %f, %s", logprobs[0], max(logprobs[2:]), logprobs)
Output:
(APIServer pid=4094802) INFO 05-21 11:06:58 [logprobs.py:197] check_conf_stop logprobs -0.000000, -16.000000, [-4.768370445162873e-07, -4.768370445162873e-07, -16.0, -16.5, -16.875, -17.0, -17.25, -17.5, -17.75, -18.0, -18.25, -18.75, -19.125, -19.25, -20.125, -20.375, -20.4375, -21.0625, -22.125, -22.4375, -22.8125]
So actully the implement is still using sampled token logprobs when calc confidence.
From the paper: Step 5: Update confidence during sampling. At the end of _update_sample_logprobs(...) (after appending the logprob dict)
# logprobs[0] is the sampled token; use the remaining candidates
if len(logprobs) > 1:
new_conf = -sum(logprobs[1:]) / len(logprobs[1:])
else:
new_conf = 0.0
So the sampled token is exclued when compute confidence, but as i print the vllm log, logprobs[0]==logprobs[1]
Code:
if len(logprobs) > 1:
logger.info("check_conf_stop logprobs %f, %f, %s", logprobs[0], max(logprobs[2:]), logprobs)
Output:
(APIServer pid=4094802) INFO 05-21 11:06:58 [logprobs.py:197] check_conf_stop logprobs -0.000000, -16.000000, [-4.768370445162873e-07, -4.768370445162873e-07, -16.0, -16.5, -16.875, -17.0, -17.25, -17.5, -17.75, -18.0, -18.25, -18.75, -19.125, -19.25, -20.125, -20.375, -20.4375, -21.0625, -22.125, -22.4375, -22.8125]
So actully the implement is still using sampled token logprobs when calc confidence.