【天池学习赛 语义分割】自定义数据集时报错处理

【天池学习赛 语义分割】自定义数据集时报错处理 项目场景：

在自定义数据集的MMSegmentation中，运行一个模型
是一个天池的练习赛：地表建筑物识别

---

问题描述：

对于数据集，在MMSeg中并没有和他对应的数据集格式，我就自己写了一个数据集，需要的config文件可以私我或者访问github
报错信息

File "/home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 446, in forward
    return self._conv_forward(input, self.weight, self.bias)
  File "/home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 443, in _conv_forward
    self.padding, self.dilation, self.groups)
RuntimeError: cuDNN error: CUDNN_STATUS_INTERNAL_ERROR
You can try to repro this exception using the following code snippet. If that doesn't trigger the error, please include your original repro script when reporting this issue.

import torch
torch.backends.cuda.matmul.allow_tf32 = True
torch.backends.cudnn.benchmark = True
torch.backends.cudnn.deterministic = False
torch.backends.cudnn.allow_tf32 = True
data = torch.randn([2, 128, 256, 256], dtype=torch.float, device='cuda', requires_grad=True)
net = torch.nn.Conv2d(128, 64, kernel_size=[3, 3], padding=[1, 1], stride=[1, 1], dilation=[1, 1], groups=1)
net = net.cuda().float()
out = net(data)
out.backward(torch.randn_like(out))
torch.cuda.synchronize()

继续报错

  File "/home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmsegmentation/mmseg/models/losses/cross_entropy_loss.py", line 203, in forward
    **kwargs)
  File "/home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/mmsegmentation/mmseg/models/losses/cross_entropy_loss.py", line 25, in cross_entropy
    ignore_index=ignore_index)
  File "/home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/nn/functional.py", line 2846, in cross_entropy
    return torch._C._nn.cross_entropy_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index, label_smoothing)
RuntimeError: CUDA error: an illegal memory access was encountered
terminate called after throwing an instance of 'c10::CUDAError'
  what():  CUDA error: an illegal memory access was encountered
Exception raised from create_event_internal at ../c10/cuda/CUDACachingAllocator.cpp:1211 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::string) + 0x42 (0x7fa24efebd62 in /home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10.so)
frame #1:  + 0x1c5f3 (0x7fa2926a15f3 in /home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
frame #2: c10::cuda::CUDACachingAllocator::raw_delete(void*) + 0x1a2 (0x7fa2926a2002 in /home/%%%%%/anaconda3/envs/open-mmlab/lib/python3.7/site-packages/torch/lib/libc10_cuda.so)
Process finished with exit code 134 (interrupted by signal 6: SIGABRT)

报错显示是： torch._C._nn.cross_entropy_loss的错误

这里发现是CE loss出错了，发现是分割的类别有问题
！！！这里发现天池给的数据集（地表建筑物识别）有大坑！！！
数据集是.jpg格式，不仅会产生像素细微差异（每次会有像素偏差），改为png格式

---

原因分析：

这里是显示CUDA ERROR ，因为CUDA是并行工作，具有异步性，要关闭CUDA的并行性

---

解决方案：

1. 设置 CUDA_LAUNCH_BLOCKING=1

这里是让cuda停止异步，方便查看程序的错误点

2. 看到CELoss错误，首先考虑是不是数据标签的类别出错了！（编写代码修改labels的像素值）

def data_inspect():
    list=glob.glob('/data/tianchi_SegGame/ann_dir/train/*')
    for i in tqdm(range(len(list))):
        img =cv2.imread(list[i])
        _, dst = cv2.threshold(img, 127, 255, cv2.THRESH_TOZERO)
        _, dst = cv2.threshold(dst, 127, 255, cv2.THRESH_BINARY)
        img=dst[:,:,0]
        # cv2.imwrite(list[i],img)
        cv2.imwrite(list[i].replace('val1','val').replace('.jpg', '.png'),img)

3. 最后修改dataset Config文件（数据集后缀名）

import os.path as osp

import mmcv
import numpy as np
from PIL import Image

from .builder import DATASETS
from .custom import CustomDataset

@DATASETS.register_module()
class mydata_TCDataset(CustomDataset):
    CLASSES = ('background', 'change')
    PALETTE = [[0,0,0], [6, 230, 230]]
    def __init__(self, **kwargs):
        super(mydata_TCDataset, self).__init__(
            img_suffix='.jpg',
            seg_map_suffix='.jpg',
            reduce_zero_label=False,
            **kwargs)
        assert osp.exists(self.img_dir)