【CUDA显存不足的问题】

CUDA显存不足的问题

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 3.91 GiB. GPU 0 has a total capacty of 15.70 GiB of which 3.88 GiB is free. Process 1672845 has 3.56 GiB memory in use. Including non-PyTorch memory, this process has 8.24 GiB memory in use. Of the allocated memory 4.98 GiB is allocated by PyTorch, and 2.96 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

解决办法

• 降低批次,控制并发数

• 将--batchsize 256降为--batchsize 32,或者更低.

• 并发数从10降为3.
  app.conf.worker_concurrency = 3

到底多少合适,需要一点一点试,没有固定答案.

关于koila库

• 代码地址: https://github.com/rentruewang/koila

在stackoverflow中,发现一个很不错的推荐,但是这个应该更加适用于在显卡显存不足的情况下进行大模型训练.
我是用已训练好的大模型跑任务.
然后koila 不支持 python 3.7 版本,其他py版本可以试一下这个.