RuntimeError: Expected all tensors to be on the same device

[chioujryu]

I encountered this error after execution. How should I resolve it?

Traceback (most recent call last):
  File "/root/vlm_fine_tune/gemma3-object-detection/train.py", line 156, in <module>
    run_training_phase(model, processor, cfg, train_dataloader, train_keys=["embed_tokens", "attn"], phase_name="embed_attn_embed_tokens", val_dataloader=val_dataloader)
  File "/root/vlm_fine_tune/gemma3-object-detection/train.py", line 108, in run_training_phase
    train_model(model, optimizer, cfg, train_dataloader, val_dataloader)
  File "/root/vlm_fine_tune/gemma3-object-detection/train.py", line 73, in train_model
    outputs = model(**batch.to(model.device))
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/accelerate/hooks.py", line 175, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 1077, in forward
    outputs = self.model(
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/transformers/utils/generic.py", line 940, in wrapper
    output = func(self, *args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 888, in forward
    inputs_embeds = self.get_input_embeddings()(llm_input_ids)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1773, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1784, in _call_impl
    return forward_call(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/accelerate/hooks.py", line 175, in new_forward
    output = module._old_forward(*args, **kwargs)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/transformers/models/gemma3/modeling_gemma3.py", line 114, in forward
    return super().forward(input_ids) * self.embed_scale.to(self.weight.dtype)
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/modules/sparse.py", line 192, in forward
    return F.embedding(
  File "/root/miniforge3/envs/gemma3/lib/python3.10/site-packages/torch/nn/functional.py", line 2546, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
RuntimeError: Expected all tensors to be on the same device, but got index is on cuda:7, different from other tensors on cuda:0 (when checking argument in method wrapper_CUDA__index_select)