Node: ‘conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter‘No algorithm worked Error messages

情况说明

用tensorflow在GPU上跑一个fit，模型里用到了两个卷积函数layers.Conv2D和layers.Conv2DTranspose
epoch时出错，具体错误日志见下文，一开始Conv2D位置报错

解决过程

（1）搜索到一个类似的：No algorithm worked！，即代码开头加上以下设置

from tensorflow.config.experimental import list_physical_devices, set_memory_growth
physical_devices = list_physical_devices('GPU')
set_memory_growth(physical_devices[0], True)

（2）现在Conv2DTranspose位置报错，据错误日志推测大概是调用cudnn跑卷积转置失败，使用错误日志搜索无果后，使用关键词“cudnn运行反卷积gpu失败”，同样参考类似问题：Could not load library cudnn_cnn_infer64_8.dll，可能是cudnn的底层cnn计算库本身有问题，学博主的降一个cudnn版本

总结

• 解决方案就是【加代码+cudnn降版本】
• 当前版本环境：window10、tensorflow2.8.0、cuda11.2.2、cudnn8.1.0.77，下载地址如下：
  • tensorflow官方文档

    pip install -i https://pypi.tuna.tsinghua.edu.cn/simple tensorflow
    

    这里临时使用了清华镜像站作为源地址，因为官方源地址下载非常慢

    #  查看pip当前设置的源地址
    pip config list
    # 升级pip
    python -m pip install --upgrade pip
    # 默认使用清华镜像站，详情网址https://mirror.tuna.tsinghua.edu.cn/help/pypi/
    pip config set global.index-url https://pypi.tuna.tsinghua.edu.cn/simple
    # 官方源站https://pypi.org/simple
    

  • cuda11.2.2 我本来装的当前最新版cuda11.6+cudnn8.2，考虑到tensorflow官网上跑GPU的软件要求，就开始换cuda版本，先在控制面板里卸载带有CUDA名字的，大概有6个，再用CCleaner排查注册表问题，再安装11.2
    
  • cudnn，cudnn8.1.0.77
• 其间接连从anaconda里卸载tensorflow，直接安装到终端之类的，然并卵
• 其间还尝试只用CPU跑来排除到底是不是GPU环境配置的问题

  import os
  os.environ["CUDA_VISIBLE_DEVICES"]="-1"
  

错误日志

NotFoundError: Graph execution error: Detected at node ‘gradient_tape/denoise/sequential_1/conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter’ defined at (most recent call last):
…
Node: ‘gradient_tape/denoise/sequential_1/conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter’
No algorithm worked! Error messages:
Profiling failure on CUDNN engine 1#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 1: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 0#TC: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 0: UNKNOWN: CUDNN_STATUS_EXECUTION_FAILED
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 3#TC: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
Profiling failure on CUDNN engine 3: UNKNOWN: CUDNN_STATUS_INTERNAL_ERROR
in tensorflow/stream_executor/cuda/cuda_dnn.cc(4177): ‘cudnnConvolutionBackwardFilter( cudnn.handle(), alpha, input_nd_.handle(), input_data.opaque(), output_nd_.handle(), output_data.opaque(), conv_.handle(), ToConvBackwardFilterAlgo(algo), scratch_memory.opaque(), scratch_memory.size(), beta, filter_.handle(), filter_data.opaque())’
[[{{node gradient_tape/denoise/sequential_1/conv2d_transpose_1/conv2d_transpose/Conv2DBackpropFilter}}]] [Op:__inference_train_function_973]