deepspeed docker集群实现多机多卡训练----问题记录及解决方案资源汇总
. Docker中实现Deepspeed多机多卡训练
- 【掘金-雨田君的记事本】docker容器中deepspeed多机多卡集群分布式训练大模型
. 问题记录及解决方案资源汇总
-
问题1：deepspeed socketStartConnect: Connect to 172.18.0.3<54379> failed : Software caused connection abort
有效方案：【博客园-高颜值的杀生丸】deepspeed 训练多机多卡报错 ncclSystemError Last error -
问题2：NCCL WARN Error while creating shared memory segment
有效方案：【简书-Aiah_Wang】NCCL分布式训练报错 -
问题3：docker swarm: Error response from daemon: rpc error: code = Unavailable desc = connection error
有效方案：【CSDN-鳄鱼儿】Docker Swarm 解决报错Error response from daemon: rpc error: code = Unavailable desc = connection error: -
问题4：ImportError: /root/.cache/torch_extensions/py310_cu121/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory
有效方案：【Github】[BUG][Upstream] py310_cu117/fused_adam/fused_adam.so: cannot open shared object file: No such file or directory #2