Running DeepSeek R1 671B Locally on EPYC Servers

Viewed 459
The post discusses the feasibility and performance of running the DeepSeek R1 671B model on budget-friendly EPYC servers. Comments highlight the performance dynamics of different hardware configurations, including CPU types, RAM specifications, and quantization methods (Q4 vs. Q8) impacting the TPS (tokens per second) rate. The challenges of operating with high RAM requirements for advanced AI models were noted alongside the potential for optimized setups for smaller models. Users expressed concern over hardware costs, power consumption, and the evolving demand for better systems akin to how gaming hardware has progressed over the years.
0 Answers