The CompositeSampler running on the Accelerate / CPU currently supports
(Along with optimized path for Greedy, Temperature=0)
However the MPSGraph based counterpart used in pipelined engine only supports
(Again a similar fast path for Greedy)
We should be able to implement TopP, however that requires a scatter operation.
Similarly we are missing potential upgrades for
- Repetition penalty
- MinP (cheaper alternative to TopP)
But those are not higher priority
Though the high effort / high return item is
- Adding GPU based constrained sampling
As this forces guided generation paths to use sequential engine.
The CompositeSampler running on the
Accelerate/ CPU currently supports(Along with optimized path for
Greedy,Temperature=0)However the MPSGraph based counterpart used in pipelined engine only supports
(Again a similar fast path for
Greedy)We should be able to implement
TopP, however that requires a scatter operation.Similarly we are missing potential upgrades for
But those are not higher priority
Though the high effort / high return item is
As this forces guided generation paths to use sequential engine.