Cls loss & Total loss too big

[ChuyiZhong]

Hi all,
I am training it on my own dataset with one class, but the cls loss and total loss seem strange to me. How to get this right?
Thanks in advance.

[zylo117]

what's the training parameter and command? Try update the latest code and train with --head_only True

[ChuyiZhong]

what's the training parameter and command? Try update the latest code and train with --head_only True

python train.py -c 4 -p crowdhuman --batch_size 8 --lr 1e-5 --num_epochs 100
--load_weights /home/fudan/zhongchuyi/Yet-Another-EfficientDet-Pytorch/weights/efficientdet-d4.pth

[zylo117]

I think there is something wrong with the dataset. Can you run the tutorial? Try training on the shape dataset. If cls loss drop to less than 10 in a few epochs, the code is fine. It's take about 5 minutes.

[ggaziv]

@ChuyiZhong see #252

[sourabhyadav]

Hi all,
I am training it on my own dataset with one class, but the cls loss and total loss seem strange to me. How to get this right?
Thanks in advance.

I am facing a similar issue. Class loss is in range of hundreds after 2 epochs.

[zylo117]

It's not normal. It should drop to under 1.0 real soon.
Please check your dataset.
Can you get a good result from the tutorial on your server?

[sourabhyadav]

It's not normal. It should drop to under 1.0 real soon.
Please check your dataset.
Can you get a good result from the tutorial on your server?

Here is the results after 10 iterations:

Step: 12709. Epoch: 9/10. Iteration: 1271/1271. Cls loss: 0.46582. Reg loss: 0.02957. Total loss: 0.49539: 100%|█| 1271/1271 [05:45<00:00,  3.68it/
Val. Epoch: 9/10. Classification loss: 0.53922. Regression loss: 0.03294. Total loss: 0.57216

For 1-class model. I have cross-checked my category_id = 1.

The command used for training is:

python3 train.py -c 1 -p pc_loockdown --data_path /home/object_det/datasets/  --batch_size 8 --lr 1e-5 --num_epochs 10 --head_only True --load_weights weights/efficientdet-d1.pth

@zylo117 Does it look normal?

[zylo117]

Yes, now it is.
BTW, did you update the latest code. In the previous version, head_only may not work as expected.

[sourabhyadav]

Yes I cloned today's repo:

One doubt still remaining:
Why my training classification loss started with 1000?

[Info] loaded weights: efficientdet-d1.pth, resuming checkpoint from step: 0
[Info] freezed backbone
Step: 499. Epoch: 0/10. Iteration: 500/1271. Cls loss: 1132.17944. Reg loss: 0.06527. Total loss: 1132.24475:  39%|▍| 499/1271 [02:16<03:11,  4.04icheckpoint...
Step: 999. Epoch: 0/10. Iteration: 1000/1271. Cls loss: 866.44366. Reg loss: 0.02792. Total loss: 866.47156:  79%|▊| 999/1271 [04:27<00:56,  4.78itcheckpoint...
Step: 1270. Epoch: 0/10. Iteration: 1271/1271. Cls loss: 901.25433. Reg loss: 0.02943. Total loss: 901.28375: 100%|█| 1271/1271 [05:37<00:00,  3.76
Val. Epoch: 0/10. Classification loss: 1058.99283. Regression loss: 0.03893. Total loss: 1059.03176
Step: 1499. Epoch: 1/10. Iteration: 229/1271. Cls loss: 693.96698. Reg loss: 0.03183. Total loss: 693.99878:  18%|▏| 228/1271 [01:05<04:42,  3.69itcheckpoint...
Step: 1999. Epoch: 1/10. Iteration: 729/1271. Cls loss: 616.92957. Reg loss: 0.03471. Total loss: 616.96429:  57%|▌| 728/1271 [03:21<02:08,  4.22itcheckpoint...
Step: 2499. Epoch: 1/10. Iteration: 1229/1271. Cls loss: 294.32889. Reg loss: 0.05017. Total loss: 294.37906:  97%|▉| 1228/1271 [05:36<00:11,  3.77checkpoint...
Step: 2541. Epoch: 1/10. Iteration: 1271/1271. Cls loss: 367.30658. Reg loss: 0.03857. Total loss: 367.34515: 100%|█| 1271/1271 [05:46<00:00,  3.67
Val. Epoch: 1/10. Classification loss: 385.38531. Regression loss: 0.03736. Total loss: 385.42267
Step: 2999. Epoch: 2/10. Iteration: 458/1271. Cls loss: 228.76134. Reg loss: 0.03491. Total loss: 228.79625:  36%|▎| 457/1271 [02:05<03:08,  4.32itcheckpoint...
Step: 3499. Epoch: 2/10. Iteration: 958/1271. Cls loss: 126.56357. Reg loss: 0.03671. Total loss: 126.60027:  75%|▊| 957/1271 [04:23<01:10,  4.43itcheckpoint...
Step: 3812. Epoch: 2/10. Iteration: 1271/1271. Cls loss: 92.13811. Reg loss: 0.03446. Total loss: 92.17256: 100%|█| 1271/1271 [05:44<00:00,  3.68it
Val. Epoch: 2/10. Classification loss: 115.62846. Regression loss: 0.03631. Total loss: 115.66477

@zylo117 Is this normal behaviour for this repo?

[zylo117]

It is. It might be caused by a lower lr.
For your case, I think 1e-5 is a little bit too low.
But considering you are doing great so far, don't change it.

[sourabhyadav]

@zylo117 Thanks for proactive replies.
I did the inference on my custom testset I find lower AP than pre-trained network? Pre-trained (80 class) person AP = 64.69% and fine-tuned (1-class) person AP = 30.26%.

Basically the person in the dataset is in lookdown or top-down view. I have only 10k training images. I trained with the above command.

Why this would have happened? My intuition is since it is the person class but from a different view it should have improved at least better than pre-trained model?

[zylo117]

if it's not overfitting like your logs say. it's underfitting.
Maybe it's not converged.
Maybe increase learning rate and keep training

[sourabhyadav]

Yeah, after 100 epochs AP results were improved.
Thanks for awesome repo.

[lzh18628137361]

Yeah, after 100 epochs AP results were improved.
Thanks for awesome repo.

after 100 epochs ,loss = ? ,thanks

[sourabhyadav]

Yeah, after 100 epochs AP results were improved.
Thanks for awesome repo.

after 100 epochs ,loss = ? ,thanks

@lzh18628137361 approx after 50-60 epochs the losses are in the range of: classification loss: 0.00256, Reg loss: 0.0018 .