# Model not copied to multiple GPUs when using DDP (using trainer)

**URL:** <https://discuss.huggingface.co/t/model-not-copied-to-multiple-gpus-when-using-ddp-using-trainer/71576>\
**Category:** 🤗Accelerate\
**Created:** [February 2, 2024, 8:23am UTC](https://discuss.huggingface.co/t/model-not-copied-to-multiple-gpus-when-using-ddp-using-trainer/71576 "2024-02-02T08:23:03Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![saukum](https://avatars.discourse-cdn.com/v4/letter/s/ac8455/32.png) [@saukum](https://discuss.huggingface.co/u/saukum)\
**Post date:** [February 2, 2024, 8:23am UTC](https://discuss.huggingface.co/t/model-not-copied-to-multiple-gpus-when-using-ddp-using-trainer/71576/1 "2024-02-02T08:23:03Z")

</div>

I am trying to finetune mT5 model on a single node with 8 Nvidia A100s. I load the model with

> datatype = torch.bfloat16  
> model = AutoModelForSeq2SeqLM.from\_pretrained(model\_checkpoint, cache\_dir=cache\_dir,torch\_dtype=datatype, device\_map=“auto”)

and use Seq2SeqTrainer

> num\_gpus = torch.cuda.device\_count()  
> args = Seq2SeqTrainingArguments(  
> output\_dir=f"/outdir",  
> evaluation\_strategy=“epoch”,  
> learning\_rate=learning\_rate,  
> per\_device\_train\_batch\_size=batch\_size//num\_gpus,  
> per\_device\_eval\_batch\_size=batch\_size//num\_gpus,  
> weight\_decay=0.01,  
> save\_total\_limit=1,  
> num\_train\_epochs=num\_train\_epochs,  
> predict\_with\_generate=True,  
> logging\_steps=logging\_steps,  
> push\_to\_hub=False,  
> )  
> trainer = Seq2SeqTrainer(  
> model,  
> args,  
> train\_dataset=tokenized\_datasets[“train”],  
> eval\_dataset=tokenized\_datasets[“validation”].select(range(32)),  
> data\_collator=data\_collator,  
> tokenizer=tokenizer,  
> compute\_metrics=compute\_metrics,  
> )

python = 3.8.10  
torch==2.0.1+cu117  
accelerate==0.26.1  
CUDA Version: 11.7

When I check the compute and memory usage on GPUs using nvidia-smi, I see that only one GPU is being used for compute. There is small memory usage on others (possibly data batches being copied). But the GPU utilization stays 0% for the other 7 GPUs. I can see in the training logs that 8 different losses are calculated.  
I am not able figure out why this is so. Could somebody please provide a hint?

---

<div class="post-metadata">

**Author:** ![saukum](https://avatars.discourse-cdn.com/v4/letter/s/ac8455/32.png) [@saukum](https://discuss.huggingface.co/u/saukum)\
**Post date:** [February 5, 2024, 11:32am UTC](https://discuss.huggingface.co/t/model-not-copied-to-multiple-gpus-when-using-ddp-using-trainer/71576/2 "2024-02-05T11:32:57Z")

</div>

Starting the training using torchrun and specifying nproc\_per\_node fixed it.  
@muellerzr : A suggestion - Adding this to the ‘Using Trainer’ section in the blog [From PyTorch DDP to Accelerate to Trainer](https://huggingface.co/blog/pytorch-ddp-accelerate-transformers) would be helpful for others.  
torchrun --nproc\_per\_node 8 \<trainingfile.py\>

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [February 5, 2024, 11:33pm UTC](https://discuss.huggingface.co/t/model-not-copied-to-multiple-gpus-when-using-ddp-using-trainer/71576/3 "2024-02-05T23:33:15Z")

</div>

Closed on February 5, 2024, 11:33pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.