Posted inLocal LLMs
I forked ik_llama.cpp and built my own quant format for “landfill” GPUs — a 35B MoE now beats upstream by +88% prefill / +30% decode on a $150 Tesla P100 (full benches + methodology inside)
Enhancing AI Performance on Salvaged Data Center Hardware: A Deep Dive into Custom Quantization for "Landfill" GPUs As artificial intelligence models continue to grow in complexity and size, leveraging existing…
