centersl/ollama
1
0
Fork 0
mirror of https://github.com/ollama/ollama.git synced 2026-03-28 03:08:44 +07:00
Code Issues Packages Projects Releases Wiki Activity
4,950 Commits 374 Branches 473 Tags
parth/decrease-concurrent-download-hf
Commit Graph

2 Commits

Author SHA1 Message Date
Jeffrey Morgan
e093db92c4 sample: temporarily use grammars for constrained generation in new engine (#9586) 2025-03-10 16:17:39 +01:00
Parth Sareen
0682dae027 sample: improve ollama engine sampler performance (#9374) 
This change bring in various interface cleanups along with greatly improving the performance of the sampler.

Tested with llama3.2 on local machine.
Improves performance from ~ 70 tokens/s -> 135 tokens/s with topK(40) enabled.
Without topK performance is ~ 110 tokens/s
 2025-03-07 12:37:48 -08:00