Resources on deciding what hardware to use for powering your local LLMs.
Relatively maintained resources:
- Tim Dettmers keeps a relatively up-to-date guide of recommendations: Which GPU(s) to Get for Deep Learning: My Experience and Advice for Using GPUs in Deep Learning
- LLM Utils is a documentation resource similar to llm-tracker that has a dedicated subsite GPU Utils that has GPU recommendations and requirements for various use-cases.
- r/LocalLLaMA Wiki Models has a list of memory requirements for different model types, parameters, quantizes
Possibly out of date articles:
- 2023-07-06 LLaMa 65B GPU benchmarks - comparison of 3090, 4090, A6000, A600 Ada w/ exllama and llama.cpp
- 2023-06-30 MosaicML compares training with MI250X vs A100 and H100
- 2023-06-15 r/LocalLLaMA: Free GPU options for LlaMA model experimentation