For best price/perf, Dual Socket EPYC ROME is probably the way to go.
If you have the cash:
- Cheap dual core 9004 chips. The going rate for a 9334 QS/ES chip is 1200 and should give you about 400GB/s
- For dual socket you’re probably looking at a Gigabyte MZ73-LM1 or AsRock Rack TURIN2D16-2T - looks like those aren’t going for about $1000-1300
- It’s about $1500-1800 for 384GB (24x16GB or 12x32GB DDR5-4800 ECC); 30% more for DDR5-5600 but maybe worth it if you’re going to drop in a 9005 upgrade at some point
- 200 for good quality risers
- $400 for a good 1300-1500W power supply
Either way you’ll want GPUs, depending on your price:
| Cost | GPU | Memory | MBW | FP16 TFLOPS | Notes | |
|---|---|---|---|---|---|---|
| $8200 | Nvidia RTX PRO 6000 | 96GB | 1.79 TB/s | 209.5 | Only buy through Nvidia Inception/Connect discount program | |
| $2000 | Nvidia RTX 5090 | 32GB | 1.79 TB/s | 209.5 | IMO don’t pay more than list price | |
| Used | $1000 | AMD MI100 | 32GB | 1.23 TB/s | 184.6 | Can be selling for more, not really worth more due to real world perf and lack of support |
| Used | $500 | AMD MI60 | 32GB | 1.02 TB/s | 29.49 | Linux only, limited support, ROCm unsupported as of 6.4 - tbt, not recommended |
| Used | $800 | Nvidia RTX 3090 | 24GB | 936.2 GB/s | 71.0 | Best bang/buck |
| $800 | AMD 7900 XTX | 24GB | 960 GB/s | 122.8 | Despite better on paper specs, at least 50% slower than 3090 in real world perf; only recommend if you can get it cheap/new | |
| $600 | AMD 9070XT | 16GB | 644.6 GB/s | 194.6 | Only worth it at MSRP | |
| $450 | Nvidia 5060 Ti | 16GB | 448 GB/s | 45.2 | personally, I’d go for a 3090 | |
| $250 | Intel B580 | 12GB | 456 | 116.8 | Keep an eye out for 24GB/48GB versions and see where that slots (not until Q4 2025) |
- If you are doing text inference you have options, but for video/image generation you will probably want CUDA
- I wouldn’t get the MI60 (or even the MI100) unless you knew what are doing (able to build your own ROCm and support libs if necessary); also some things like FA may simply not work. Be sure to carefully check https://rocm.docs.amd.com/en/latest/compatibility/compatibility-matrix.html and to search online for features that you require, but in general the 7900 XT/XTX / W7900 is or maybe soon the 9070XT / R9700 (and MI300X on server side, but those are aren’t reasonably available for purchase) are the only AMD cards I’d recommend for compatibility.