YOLOv3_目标检测

本文主要是介绍YOLOv3_目标检测，希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

YOLOv3_目标检测

YOLOv1最初是由Joseph Redmon实现的，和大型NLP transformers不同，YOLOv1设计的很小，可为设备上的部署提供实时检测速度。

YOLO-9000是Joseph Redmon实现的第二个版本YOLOv2目标检测器，它对YOLOv1做了很多技巧上的改进，并强调该检测器能够推广到检测世界上的任何物体。

YOLOv3对YOLOv2做了进一步的改进，引入多尺度特征融合，针对不同网格尺寸并行处理，大大提升了不同尺寸目标的检测精度。

一、VOC2007数据集

本次实验我采用的是VOC2007数据集。

我只选用了其中的JEPGImages、Annotations两个文件夹。JEPGImages文件夹含有9963张RGB图片，总共20个类别，Annotations文件夹含有9963个xml文件，分别记录了每张图片中目标物体的类别与信息。

class_dictionary = {'aeroplane': 0, 'bicycle': 1, 'bird': 2, 'boat': 3, 'bottle': 4, 'bus': 5,'car': 6, 'cat': 7, 'chair': 8, 'cow': 9, 'diningtable': 10, 'dog': 11,'horse': 12, 'motorbike': 13, 'person': 14, 'pottedplant': 15, 'sheep': 16,'sofa': 17, 'train': 18, 'tvmonitor': 19}

二、先验框Anchors聚类

在基于anchor的目标检测网络中，一个至关重要的步骤就是科学的设置anchor，可以说anchor设置的合理与否，极大的影响着检测模型最终性能的好坏。

anchor到底是什么？如果用一句话概括，就是在图像上预设好的不同大小、不同长宽比的参照框。如果我们想要去检测某个目标物体，很容易可以想到，该目标物体的长宽比大致是固定的，比如说想要检测人脸，人脸的长宽比会近似在1:1.5。如果我们往模型里预先加入这个先验知识，网络就不必再大费力气重头学起，可以轻松舍弃掉一些不合理的学习结果，模型学习起来更加简单，检测效果自然会更好。

假设有一张416x416大小的图片，经过卷积上采样，得到13x13、26x26、52x52大小的特征图。我们在这三个特征图的每个点都设置三个不同大小的anchor。第一个13 x 13的特征图上设置了13 x 13 x 3 = 507个anchor，尺寸分别是：[179 265]、[303 170]、[371 318]。第二个26 x 26的特征图上设置了26 x 26 x 3 = 2028个anchor，尺寸分别是：[69 52]、[100 196]、[149 112]。第三个52 x 52的特征图上设置了52 x 52 x 3 = 8112个anchor，尺寸分别是：[25 35]、[34 77]、[64 112]。

这9个anchor尺寸是利用k_means算法聚类得到的。

从8000张训练数据集图片中取出所有bounding box，可以得到19670个box，其记录着每个目标物体的w和h。我们对这个二维数据集进行k_means聚类，得到9个聚类中心，并按w和h的大小依次排序，后续在YOLOv3模型中用来设置anchor。

三、数据结构编码

在read_data_path.py文件中，read_image_path()函数用来读取存储图片的绝对路径，read_coordinate_txt()函数用来读取目标物体的位置、种类，都以字符串的形式记录。整个数据集划分成三部分：8000个训练样本、1000个验证样本、963个测试样本。

train_x : （8000，），列表形式
train_y : （8000，），列表形式
val_x : （1000，），列表形式
val_y : （1000，），列表形式
test_x :（963，），列表形式
test_y :（963，），列表形式

在train.py文件中，data_encoding()函数对数据结构进行第一次编码。对于一个batch的train_x，先用opencv读取图片，再像素值除以255归一化，最后resize到(416, 416)尺寸。对于一个batch的train_y，首先将x1、y1、x2、y2转化为center_x、center_y、w、h，然后将center_x、center_y、w、h除以图片长和宽，转化成占整张图片的比例，最后针对每个目标物体的边框，根据IOU计算出最近似的那个anchor，将坐标值存储在那个位置，其他位置全部置0，得到(batch, 13, 13, 3, 25)、(batch, 26, 26, 3, 25)、(batch, 52, 52, 3, 25)的数据结构。

在这里，center_x_ratio、center_y_ratio记录的是占整张图片的比例，而不是针对于每个grid的比例。这一点还需要后续再处理，因为YOLOv3有三种不同尺寸的grid，不适合在这里立马就转换好，最好放在loss函数中用for循环分三种情况分别处理。

在yolov3_loss.py文件中，yolo_loss()函数对数据结构进行第二次编码。对(batch, 13, 13, 3, 25)、(batch, 26, 26, 3, 25)、(batch, 52, 52, 3, 25)的y_true再做处理。针对(13, 13)、(26, 26)、(52, 52)三种的网格，center_x_ratio、center_y_ratio转化成相对于不同小格子的偏移，w_ratio、h_ratio要转化为相对于anchor长宽的偏移。

四、Bounding Box变换

首先必须想明白一个问题：YOLOv3卷积网络最终输出的结果，到底是在拟合什么？

YOLOv1网络最终输出的结果，其实就是目标物体位置的直接刻画；VGG网络最终输出的结果，其实就是分类结果的直接刻画。但YOLOv3完全不同，它借助残差网络的思想，网络学习的不是目标物体位置的直接刻画，而是目标物体位置的残差，这一点可以在解码函数中完全体现出来。

YOLOv3借用了bounding box回归的思想。对真实数据y进行bounding box变换后，拟合起来会更简单，网络学习效果更好。

对于真实y_true：xy要转化成相对于每个小格子的比例，wh要除以最接近的anchor的长宽再取ln，confidence、class不做处理。

对于网络最终输出结果y_pre：xy、confidence、class进行sigmid变换，wh不做处理，此时它们就可以作为真实数据bounding box变换后的数值拟合。

五、Loss损失函数

进行bounding box变换后的y_true，与进行sigmoid变换后的y_pre计算损失。代码中y_pre的xy、confidence、class的sigmoid变换，隐藏在k.binary_crossentropy函数的from_logits=True中。

xy、confidence、class采用二项交叉熵损失，wh采用均方误差损失。

六、YOLOv3网络结构

YOLOv3网络主要由两部分构成：第一部分是Darknet_53特征提取网络，第二部分是FPN多尺度特征融合网络。

Darknet_53特征提取网络：由DBL、res1、…、res8构成。DBL可作为YOLOv3网络的最小组建，由一个卷积层一个BN层一个LeakyReLu层组成。整个Darknet_53网络没有池化层和全连接层，所有下采样过程通过卷积stride = 2实现。

FPN多尺度特征融合网络：输出三个尺度的预测结果。对于Darknet_53提取到的特征，先经过一系列的卷积变换，第一个分支输出(13, 13, 75)的预测结果。然后对特征图进行上采样，与之前提取到的特征进行融合，经过一系列的卷积变换，第二个分支输出(26, 26, 75)的预测结果。最后再对特征图进行上采样，与更之前提取到的特征进行融合，经过一系列的卷积变换，第三个分支输出(52, 52, 75)的预测结果。

七、训练过程

网上下载得到yolo_weights.h5网络权重，原模型总共有252层，载入权重后将前249层网络参数冰冻起来，针对特定的数据集，只训练最后3层的网络权重。

总共训练了三天时间，每个epoch花费1.5个小时。

第一轮训练：Adam(lr=1e-3)，epoch = 10。train loss从15844.1下降到31.6，val loss从95.7下降到33.5，此时目标检测效果已经不错。

第二轮训练：sgd = optimizers.SGD(lr=1e-4, momentum=0.9)，epoch=15。
train loss从32.40下降到26.35，val loss从31.03下降到28.37，此时训练已经达到瓶颈，损失函数很难再降下去。

第三轮训练：sgd = optimizers.SGD(lr=1e-8, momentum=0.9)，epoch = 10。
我将学习率调整到一个非常低的水平，期望损失函数能缓慢下降，但结果不尽如人意，train loss和val loss均达到瓶颈，无法再进一步降低了。

八、实验结果

从测试集中随意选取10张图片，检测结果如下：

可以看出，YOLOv3的目标检测效果的确比YOLOv1好太多，在引入anchor机制之后，网络学习出来的边框长宽更加精确，检测效果更好。

九、深入思考

Ques1：YOLOv2对YOLOv1做了哪些改进？

YOLOv2的论文全名为YOLO9000: Better, Faster, Stronger，它斩获了CVPR 2017 Best Paper Honorable Mention。这篇文章包含了两个模型：YOLOv2和YOLO9000，二者的主体结构是一致的，只是YOLO9000采用了一种混合训练的方式，号称能实现9000个类别的目标检测，所以作者给它取名为YOLO9000。

YOLOv2是在YOLOv1的基础上改进得来的，模型主体思路没什么变化，主要是引入了很多cv领域的trick，这些trick大幅提高了目标检测精度。

（1）引入BN层。Batch Normalization可以提升模型收敛速度，而且可以起到一定的正则化效果，降低模型的过拟合。YOLOv2的每个卷积层后面都添加了BN层，使用BN层后，YOLOv2的mAP提升了2.4%。

（2）Anchor Boxes。YOLOv1采用全连接层直接对边框进行预测，边框的宽与高是相对整张图片大小的，由于各个图片中存在不同尺度的物体，训练过程中学习不同物体的形状非常困难，这也导致了YOLOv1在精确定位方面表现较差。YOLOv2借鉴了RPN网络的先验框策略，使得模型更容易学习。使用Anchor boxes之后，YOLOv2的召回率由原来的81%提升至88%。

（3）Dimension Clusters。在Faster RCNN、SSD中，先验框的长宽都是手动设定的，带有一定的主观性，而如果选取的先验框长宽比较合适，模型会更容易学习。YOLOv2利用k-means算法对训练集的边界框做了聚类分析。

（4）Darknet-19特征提取器。YOLOv2采用了一个新的特征提取器Darknet-19，包括19个卷积层和5个maxpooling层，而之前一直都是使用VGG16进行特征提取。

Ques2：YOLOv3对YOLOv2做了哪些改进？

YOLOv3：An Incremental Improvement，按照原文说法，这仅是他们近一年的一个工作报告，并不算一个完整的paper，只是把其它论文的一些工作在YOLO上尝试了一下。相比于YOLOv2，YOLOv3最大的改进包括两点：使用残差模型和采用FPN架构。

YOLOv3的特征提取器是一个包含53个卷积层的残差模型，称为Darknet-53。相比于Darknet-19特征提取器，Darknet-53采用了残差单元所以可以构建得更深。YOLOv3采用FPN实现多尺度检测，通过金字塔特征融合策略，提取得到更好的图像特征。

Ques3：深度学习网络层数构建太深会出现什么问题？

VGG卷积网络达到了19层，GoogleNet卷积网络达到了22层，但实际上算法精度并不会随着网络层数的增多而提高，网络层数过多反而会出现以下问题： 计算资源的消耗、模型容易过拟合、梯度消失、梯度爆炸。

但更重要的是，随着网络层数的增加，网络会发生退化（degradation）现象：

随着网络层数的增多，训练集loss逐渐下降，然后趋于饱和，但如果再增加网络深度的话，训练集loss反而会增大。注意这并不是过拟合，因为过拟合中训练loss是一直减小的，这里训练loss莫名其妙反而变大了。

Ques4：残差网络的思想是什么，为什么能提取得到更好的特征？

当网络发生退化时，浅层网络能够达到比深层网络更好的特征提取效果，这时如果我们把浅层的特征也传递到深层，那么最终效果至少不会比浅层网络差。如果在VGG的第98层和VGG的第14层之间添加一条直接映射，网络的特征提取效果会更佳。

从信息论的角度理解，前向传输过程中随着层数的加深，Feature Map包含的图像信息会逐层减少，而残差网络直接映射的加入，保证了第l+1层的网络一定比第l层包含更多的图像信息。基于这种直接映射跨层连接的思想，残差网络应运而生。

Ques5：YOLOv3为什么能解决同一位置检测多个重叠的目标？

YOLOv1算法中图片被划分为(7, 7)的网格，每个grid只能检测该网格内一个目标物体。

YOLOv3算法引入了多尺度检测，图片具有(13, 13)、(26, 26)、(52, 52)三种网格划分。目标物体重叠情况下，虽然目标物体的位置相同，但它们的长宽尺寸往往不同。尽管此时每个grid还是只负责一个目标物体的检测，但借助三种不同尺度，YOLOv3可以实现同一位置多个重叠目标的检测。

Ques6：YOLOv3是如何融入Anchors思想的？

关于如何将Anchors思想融入模型算法中，YOLOv3和Faster RCNN处理方法完全不同。

Faster RCNN类似于RCNN的思路，先在原图上生成许许多多的anchors，把它们作为候选区域，一个一个的进行分类和回归调整。而YOLOv3完全不同，它直接在输出数据结构上融入anchors，网络最终输出的结果，就代表着针对不同anchors的位置调整情况。

Ques7：总结一下自己复现YOLOv3时代码出的BUG。

（1）训练数据y结构出错，代码无法运行。最后找到错误原因，针对训练数据y我之前设置的是(batch, 3)的结构，其中每个代表的含义是zeros(13, 13, 3, 25), zeros(26, 26, 3, 25), zeros(52, 52, 3, 25)。实际上要改成[zeros(batch, 13, 13, 3, 25), zeros(batch, 13, 13, 3, 25), zeros(batch, 13, 13, 3, 25)]的结构。

（2）训练起始阶段损失函数值巨大，50万的loss。查找后发现，训练数据结构转换中出了问题，写成了center_x / w， center_y / h，改成center_x / size[1]， center_y / size[2]后训练就正常了，起始阶段loss在15000左右。

（3）训练过程中，train loss能不断减小到几十，但val loss依然是上千之巨降低不了。之前在YOLOv1训练过程中也遇到这个问题，最后发现是权重文件加载不匹配，特征提取层未能读取到正确权重就被冰冻起来了。

我重新运行convert.py文件，转化生成yolo_weights.h5，权重文件被成功加载。

之前权重文件加载失败：

之后权重文件加载成功：

（4）最后我还发现源码逻辑上的一个BUG。

源码在生成Anchors时，是直接对图片数据集中目标物体的边框进行聚类。但在实际网络拟合中，图片被resize到(416, 416)大小，我由原边框长宽设计出来的先验信息，并没起到想要的效果。因此在生成Anchors时，应该先把图片数据集中目标物体边框长宽先缩放，让它与网络实际处理过程所拟合的数值相匹配。

十、源码

数据集读取：

import os
import xml.etree.ElementTree as ETclass_dictionary = {'aeroplane': 0, 'bicycle': 1, 'bird': 2, 'boat': 3, 'bottle': 4, 'bus': 5,'car': 6, 'cat': 7, 'chair': 8, 'cow': 9, 'diningtable': 10, 'dog': 11,'horse': 12, 'motorbike': 13, 'person': 14, 'pottedplant': 15, 'sheep': 16,'sofa': 17, 'train': 18, 'tvmonitor': 19}
class_list = list(class_dictionary.keys())def read_image_path():data_x = []filename = os.listdir('/home/archer/CODE/YOLOv3/JPEGImages')filename.sort()for name in filename:path = '/home/archer/CODE/YOLOv3/JPEGImages/' + namedata_x.append(path)print('JPEGImages has been download ! ')return data_xdef read_coordinate_txt():data_y = []filename = os.listdir('/home/archer/CODE/YOLOv3/Annotations')filename.sort()for name in filename:tree = ET.parse('/home/archer/CODE/YOLOv3/Annotations/' + name)root = tree.getroot()coordinate = ''for obj in root.iter('object'):difficult = obj.find('difficult').textcls = obj.find('name').textif cls not in class_list or int(difficult) == 1:continuecls_id = class_list.index(cls)xml_box = obj.find('bndbox')x_min = int(xml_box.find('xmin').text)y_min = int(xml_box.find('ymin').text)x_max = int(xml_box.find('xmax').text)y_max = int(xml_box.find('ymax').text)loc = (str(x_min) + ',' + str(y_min) + ',' + str(x_max) + ',' + str(y_max) + ',' + str(cls_id) + '  ')coordinate = coordinate + locdata_y.append(coordinate)print('Object Coordinate has been download ! ')return data_ydef make_data():data_x = read_image_path()data_y = read_coordinate_txt()n = len(data_x)train_x = data_x[0:8000]train_y = data_y[0:8000]val_x = data_x[8000:9000]val_y = data_y[8000:9000]test_x = data_x[9000:n]test_y = data_y[9000:n]return train_x, train_y, val_x, val_y, test_x, test_y

Anchors生成：

import numpy as np
import cv2# the anchor_iou between 19670 box and 9 clusters
def anchor_iou(boxes, clusters):n = boxes.shape[0]    # 19670k = clusters.shape[0]    # 9box_area = boxes[:, 0] * boxes[:, 1]    # 19670# repeat function : [1 1 1 2 2 2 3 3 3]box_area = box_area.repeat(k)    # 19670 * 9 = 177030box_area = np.reshape(box_area, (n, k))    # (19670, 9), every column is the box_area vectorcluster_area = clusters[:, 0] * clusters[:, 1]    # 9# tile function : [1 2 3 1 2 3 1 2 3]cluster_area = np.tile(cluster_area, [1, n])    # 9 * 19670 = 177030cluster_area = np.reshape(cluster_area, (n, k))    # (19670, 9), every row is the cluster_area vectorbox_w_matrix = np.reshape(boxes[:, 0].repeat(k), (n, k))    # (19670, 9)cluster_w_matrix = np.reshape(np.tile(clusters[:, 0], (1, n)), (n, k))    # (19670, 9)min_w_matrix = np.minimum(cluster_w_matrix, box_w_matrix)    # (19670, 9)box_h_matrix = np.reshape(boxes[:, 1].repeat(k), (n, k))    # (19670, 9)cluster_h_matrix = np.reshape(np.tile(clusters[:, 1], (1, n)), (n, k))    # (19670, 9)min_h_matrix = np.minimum(cluster_h_matrix, box_h_matrix)    # (19670, 9)inter_area = np.multiply(min_w_matrix, min_h_matrix)    # (19670, 9)iou = inter_area / (box_area + cluster_area - inter_area)    # (19670, 9)return ioudef k_means(box, k):box_number = len(box)    # 19670last_nearest = np.zeros(box_number)# init k clustersnp.random.seed(1)clusters = box[np.random.choice(box_number, k, replace=False)]# (9, 2)while True:# calculate the iou_distance between 19670 boxes and 9 clusters# distance :  (19670, 9)distances = 1 - anchor_iou(box, clusters)# calculate the mum anchor index for 19670 boxes# current_nearest : (19670, 1)current_nearest = np.argmin(distances, axis=1)if (last_nearest == current_nearest).all():breakfor num in range(k):clusters[num] = np.median(box[current_nearest == num], axis=0)last_nearest = current_nearestreturn clustersdef calculate_anchor(train_x, train_y):box = []# len(train_y) : 19670for i in range(len(train_y)):img = cv2.imread(train_x[i])size = img.shapeobj_all = train_y[i].strip().split()for j in range(len(obj_all)):obj = obj_all[j].split(',')x1, y1, x2, y2 = [int(obj[0]), int(obj[1]), int(obj[2]), int(obj[3])]w = x2 - x1h = y2 - y1w = int(w / size[1] * 416)h = int(h / size[0] * 416)box.append([w, h])box = np.array(box)# box : (19670, 2)anchors = k_means(box, 9)# anchors : (9, 2)index = np.argsort(anchors[:, 0])# [0 5 6 8 2 7 4 3 1]anchors = anchors[index]# [[22  37]#  [26  82]#  [49  132]#  [56  57]#  [89  211]#  [113 108]#  [162 298]#  [238 177]#  [341 340]]return anchors

YOLOv3模型结构：

from functools import reduce
from keras.layers import Conv2D, Add, ZeroPadding2D, UpSampling2D, Concatenate, Input
from keras.layers.advanced_activations import LeakyReLU
from keras.layers.normalization import BatchNormalization
from keras.models import Model
from keras.regularizers import l2
from functools import wrapsdef compose(*funcs):# Compose arbitrarily many functions, evaluated left to right.# Reference: https://mathieularose.com/function-composition-in-python/# return lambda x: reduce(lambda v, f: f(v), funcs, x)if funcs:return reduce(lambda f, g: lambda *a, **kw: g(f(*a, **kw)), funcs)else:raise ValueError('Composition of empty sequence not supported.')@wraps(Conv2D)
def DarknetConv2D(*args, **kwargs):darknet_conv_kwargs = {'kernel_regularizer': l2(5e-4)}darknet_conv_kwargs['padding'] = 'valid' if kwargs.get('strides') == (2, 2) else 'same'darknet_conv_kwargs.update(kwargs)return Conv2D(*args, **darknet_conv_kwargs)def DarknetConv2D_BN_Leaky(*args, **kwargs):no_bias_kwargs = {'use_bias': False}no_bias_kwargs.update(kwargs)return compose(DarknetConv2D(*args, **no_bias_kwargs),BatchNormalization(),LeakyReLU(alpha=0.1))def resblock_body(x, num_filters, num_blocks):# Darknet uses left and top padding instead of 'same' modex = ZeroPadding2D(((1, 0), (1, 0)))(x)x = DarknetConv2D_BN_Leaky(num_filters, (3, 3), strides=(2, 2))(x)for i in range(num_blocks):y = compose(DarknetConv2D_BN_Leaky(num_filters//2, (1, 1)),DarknetConv2D_BN_Leaky(num_filters, (3, 3)))(x)x = Add()([x, y])return xdef darknet_body(x):x = DarknetConv2D_BN_Leaky(32, (3, 3))(x)x = resblock_body(x, 64, 1)x = resblock_body(x, 128, 2)x = resblock_body(x, 256, 8)x = resblock_body(x, 512, 8)x = resblock_body(x, 1024, 4)return xdef make_last_layers(x, num_filters, out_filters):x = compose(DarknetConv2D_BN_Leaky(num_filters, (1, 1)),DarknetConv2D_BN_Leaky(num_filters*2, (3, 3)),DarknetConv2D_BN_Leaky(num_filters, (1, 1)),DarknetConv2D_BN_Leaky(num_filters*2, (3, 3)),DarknetConv2D_BN_Leaky(num_filters, (1, 1)))(x)y = compose(DarknetConv2D_BN_Leaky(num_filters*2, (3, 3)),DarknetConv2D(out_filters, (1, 1)))(x)return x, ydef create_yolo_model():# Create YOLO_V3 model CNN body in Kerasinputs = Input((416, 416, 3))darknet = Model(inputs, darknet_body(inputs))# darknet.summary()x, y1 = make_last_layers(darknet.output, 512, 75)x = compose(DarknetConv2D_BN_Leaky(256, (1, 1)), UpSampling2D(2))(x)x = Concatenate()([x, darknet.layers[152].output])x, y2 = make_last_layers(x, 256, 75)x = compose(DarknetConv2D_BN_Leaky(128, (1, 1)), UpSampling2D(2))(x)x = Concatenate()([x, darknet.layers[92].output])x, y3 = make_last_layers(x, 128, 75)yolo3_model = Model(inputs, [y1, y2, y3])yolo3_model.summary()return yolo3_model

损失函数：

from keras import backend as k
import numpy as npwith open("yolo_anchors.txt", "r") as f:string = f.read().strip().split(',')
anchors = [int(s) for s in string]
anchors = np.reshape(anchors, (9, 2))def yolo_head(y_pre_part):# y_pre_part : Tensor, shape=(batch, 13, 13, 75), float32grid_shape = k.shape(y_pre_part)[1:3]# Tensor, shape=(2, ), [13 13]grid_y = k.tile(k.reshape(k.arange(0, stop=grid_shape[0]), [-1, 1, 1, 1]), [1, grid_shape[1], 1, 1])# Tensor, shape=(13, 13, 1, 1), int32grid_x = k.tile(k.reshape(k.arange(0, stop=grid_shape[1]), [1, -1, 1, 1]), [grid_shape[0], 1, 1, 1])# Tensor, shape=(13, 13, 1, 1), int32grid = k.concatenate([grid_x, grid_y])# Tensor, shape=(13, 13, 1, 2), int32grid = k.cast(grid, k.dtype(y_pre_part))# Tensor, shape=(13, 13, 1, 2), float32return griddef yolo_loss(args):y_pre = args[:3]y_true = args[3:]# y_pre :  [ Tensor (batch, 13, 13, 75) , Tensor (batch, 26, 26, 75) , Tensor (batch, 52, 52, 75) ]# y_true : [ Tensor (batch, 13, 13, 3, 25) , Tensor (batch, 26, 26, 3, 25) , Tensor (batch, 52, 52, 3, 25) ]# little grid predict large bounding box and# large grid predict little bounding boxanchor_mask = [[6, 7, 8], [3, 4, 5], [0, 1, 2]]input_shape = k.shape(y_pre[0])[1:3] * 32input_shape = k.cast(input_shape, k.dtype(y_true[0]))# [13, 13] * 32 = [416, 416] - tensor, floatgrid_shapes = [k.cast(k.shape(y_pre[l])[1:3], k.dtype(y_true[0])) for l in range(3)]# [[13, 13]-tensor, [26, 26]-tensor, [52, 52]-tensor]loss = 0m = k.cast(k.shape(y_pre[0])[0], k.dtype(y_pre[0]))    # batch size, tensor-32, floatfor l in range(3):# y_pre[l] : shape=(batch, ?, ?, 75), float32# single_y_pre : shape=(batch, ?, ?, 3, 25), float32# grid : shape=(?, ?, 1, 2), float32grid_shape = k.shape(y_pre[l])[1:3]    # Tensor, shape=(2, ), [13, 13] or [26, 26] or [52, 52]single_y_pre = k.reshape(y_pre[l], [-1, grid_shape[0], grid_shape[1], 3, 25])# Tensor, shape=(batch, ?, ?, 3, 25), float32grid = yolo_head(y_pre[l])# Tensor, shape=(?, ?, 1, 2), float32# calculate the y_true_confidence, y_true_class, y_true_xy, y_true_wh---------------------------------------y_true_confidence = y_true[l][..., 4:5]    # Tensor, (batch, 13, 13, 3, 1), float32object_mask = y_true_confidencey_true_class = y_true[l][..., 5:]    # Tensor, (batch, 13, 13, 3, 20), float32y_true_xy = y_true[l][..., :2]*grid_shapes[l][::-1] - grid    # Tensor, shape=(batch, 13, 13, 3, 2), float32y_true_wh = k.log(y_true[l][..., 2:4] / anchors[anchor_mask[l]] * input_shape[::-1])y_true_wh = k.switch(object_mask, y_true_wh, k.zeros_like(y_true_wh))# avoid log(0)=-inf# Tensor, shape=(batch, 13, 13, 3, 2), float32box_loss_scale = 2 - y_true[l][..., 2:3] * y_true[l][..., 3:4]# Tensor, shape=(batch, 13, 13, 3, 1), float32# calculate the y_pre_confidence, y_pre_class, y_pre_xy, y_pre_wh-----------------------------------------y_pre_confidence = single_y_pre[..., 4:5]# Tensor, shape=(batch, 13, 13, 3, 1), float32y_pre_class = single_y_pre[..., 5:]# Tensor, shape=(batch, 13, 13, 3, 20), float32y_pre_xy = single_y_pre[..., 0:2]# Tensor, shape=(batch, 13, 13, 3, 2), float32y_pre_wh = single_y_pre[..., 2:4]# Tensor, shape=(batch, 13, 13, 3, 2), float32# calculate the sum loss ---------------------------------------------------------------------------------xy_loss = object_mask * box_loss_scale * k.binary_crossentropy(y_true_xy, y_pre_xy, from_logits=True)wh_loss = object_mask * box_loss_scale * 0.5 * k.square(y_true_wh - y_pre_wh)confidence_loss1 = object_mask * k.binary_crossentropy(y_true_confidence, y_pre_confidence, from_logits=True)confidence_loss2 = (1-object_mask) * k.binary_crossentropy(y_true_confidence, y_pre_confidence, from_logits=True)confidence_loss = confidence_loss1 + confidence_loss2class_loss = object_mask * k.binary_crossentropy(y_true_class, y_pre_class, from_logits=True)xy_loss = k.sum(xy_loss) / mwh_loss = k.sum(wh_loss) / mconfidence_loss = k.sum(confidence_loss) / mclass_loss = k.sum(class_loss) / mloss += xy_loss + wh_loss + confidence_loss + class_lossreturn loss

训练函数：

import numpy as np
from keras.models import load_model
import cv2
import yolov3_model
from yolov3_loss import yolo_loss
from keras.utils import Sequence
import math
from keras.callbacks import ModelCheckpoint
from keras import optimizers
from keras.models import Model
from keras.layers import Input, Lambda
from keras.optimizers import Adamwith open("yolo_anchors.txt", "r") as f:string = f.read().strip().split(',')
anchors = [int(s) for s in string]
anchors = np.reshape(anchors, (9, 2))# [[22  37]
#  [26  82]
#  [49  132]
#  [56  57]
#  [89  211]
#  [113 108]
#  [162 298]
#  [238 177]
#  [341 340]]def max_iou_index(w, h):# (w, h) : 253 177box_area = np.array([w * h]).repeat(9)# box_area : [44781 44781 44781 44781 44781 44781 44781 44781 44781]anchor_area = anchors[:, 0] * anchors[:, 1]# anchor_area : [875 2618 7168 3588 19600 16688 47435 51510 117978]anchor_w_matrix = anchors[:, 0]# anchor_w_matrix : [25 34 64 69 100 149 179 303 371]min_w_matrix = np.minimum(anchor_w_matrix, w)# min_w_matrix : [25 34 64 69 100 149 179 253 253]anchor_h_matrix = anchors[:, 1]# anchor_h_matrix : [35 77 112 52 196 112 265 170 318]min_h_matrix = np.minimum(anchor_h_matrix, h)# min_h_matrix : [35 77 112 52 177 112 177 170 177]inter_area = np.multiply(min_w_matrix, min_h_matrix)# inter_area : [875 2618 7168 3588 17700 16688 31683 43010 44781]iou = inter_area / (box_area + anchor_area - inter_area)# iou : [0.01953954 0.05846229 0.16006789 0.08012327 0.37916926 0.37265805 0.52340046 0.80722959 0.37957077]index = np.argmax(iou)return indexdef data_encoding(batch_image_path, batch_true_boxes):# batch_image_path : (32, str)， str recorded the image path# batch_true_boxes : (32, str)， str has absolute x_min, y_min, x_max, y_max, class_id# encoding_y : x、y、w、h are relative valueanchor_mask = [[6, 7, 8], [3, 4, 5], [0, 1, 2]]grid_shapes = [[13, 13], [26, 26], [52, 52]]batch_encoding_y = [np.zeros((32, 13, 13, 3, 25)), np.zeros((32, 26, 26, 3, 25)), np.zeros((32, 52, 52, 3, 25))]batch_images = []for i in range(len(batch_true_boxes)):img = cv2.imread(batch_image_path[i])size = img.shapeimg1 = img / 255resize_img = cv2.resize(img1, (416, 416), interpolation=cv2.INTER_AREA)batch_images.append(resize_img)obj_all = batch_true_boxes[i].strip().split()for j in range(len(obj_all)):obj = obj_all[j].split(',')x1, y1, x2, y2 = [int(obj[0]), int(obj[1]), int(obj[2]), int(obj[3])]category = int(obj[4])    # 0 - 19center_x = (x1 + x2) / 2center_y = (y1 + y2) / 2w = x2 - x1h = y2 - y1center_x_ratio = center_x / size[1]center_y_ratio = center_y / size[0]w_ratio = w / size[1]h_ratio = h / size[0]anchor_index = max_iou_index(w, h)for num in range(3):if anchor_index in anchor_mask[num]:# anchor_mask[0] : [6, 7, 8]# anchor_mask[1] : [3, 4, 5]# anchor_mask[0] : [0, 1, 2]inner_index = anchor_mask[num].index(anchor_index)   # 0 or 1 or 2grid_x = int(center_x / size[1] * grid_shapes[num][1])grid_y = int(center_y / size[0] * grid_shapes[num][0])batch_encoding_y[num][i, grid_y, grid_x, inner_index, 0:4] = np.array([center_x_ratio,center_y_ratio,w_ratio, h_ratio])batch_encoding_y[num][i, grid_y, grid_x, inner_index, 4] = 1batch_encoding_y[num][i, grid_y, grid_x, inner_index, 5 + category] = 1return batch_images, batch_encoding_yclass SequenceData(Sequence):def __init__(self, data_x, data_y, batch_size):self.batch_size = batch_sizeself.data_x = data_xself.data_y = data_yself.indexes = np.arange(len(self.data_x))def __len__(self):return math.floor(len(self.data_x) / float(self.batch_size))def on_epoch_end(self):np.random.shuffle(self.indexes)def __getitem__(self, idx):batch_index = self.indexes[idx * self.batch_size:(idx + 1) * self.batch_size]batch_x = [self.data_x[k] for k in batch_index]batch_y = [self.data_y[k] for k in batch_index]x, y = data_encoding(batch_x, batch_y)x = np.array(x)return [x, *y], np.zeros(32)# create model and train and save
def train_network(train_generator, validation_generator, epoch):model_body = yolov3_model.create_yolo_model()model_body.load_weights('/home/archer/CODE/YOLOv3/yolo_weights.h5', by_name=True, skip_mismatch=True)print('model_body layers : ', len(model_body.layers))for i in range(249):model_body.layers[i].trainable = Falseprint('249 Layers has been frozen ! ')grid_shape = np.array([[13, 13], [26, 26], [52, 52]])y_true = [Input(shape=(grid_shape[l, 0], grid_shape[l, 1], 3, 25)) for l in range(3)]# [Tensor-(?, 13, 13, 3, 25), Tensor-(?, 26, 26, 3, 25), Tensor-(?, 52, 52, 3, 25)]model_loss = Lambda(yolo_loss, output_shape=(1, ), name='yolo_loss')([*model_body.output, *y_true])model = Model([model_body.input, *y_true], model_loss)# model_body.input : Tensor, shape=(?, 416, 416, 3), float32model.compile(optimizer=Adam(lr=1e-3), loss={'yolo_loss': lambda y_true, y_pre: y_pre})checkpoint = ModelCheckpoint('/home/archer/CODE/YOLOv3/best_weights.hdf5', monitor='val_loss',save_weights_only=True, save_best_only=True)model.fit_generator(train_generator,steps_per_epoch=len(train_generator),epochs=epoch,validation_data=validation_generator,validation_steps=len(validation_generator),callbacks=[checkpoint])model_body.save_weights('first_weights.hdf5')def load_network_then_train(train_generator, validation_generator, epoch, input_name, output_name):model_body = yolov3_model.create_yolo_model()model_body.load_weights(input_name, by_name=True, skip_mismatch=True)for i in range(249):model_body.layers[i].trainable = Falsegrid_shape = np.array([[13, 13], [26, 26], [52, 52]])y_true = [Input(shape=(grid_shape[l, 0], grid_shape[l, 1], 3, 25)) for l in range(3)]model_loss = Lambda(yolo_loss, output_shape=(1,), name='yolo_loss')([*model_body.output, *y_true])model = Model([model_body.input, *y_true], model_loss)sgd = optimizers.SGD(lr=1e-8, momentum=0.9)model.compile(optimizer=sgd, loss={'yolo_loss': lambda y_true, y_pre: y_pre})checkpoint = ModelCheckpoint('/home/archer/CODE/YOLOv3/best_weights.hdf5', monitor='val_loss',save_weights_only=True, save_best_only=True)model.fit_generator(train_generator,steps_per_epoch=len(train_generator),epochs=epoch,validation_data=validation_generator,validation_steps=len(validation_generator),callbacks=[checkpoint])model.save_weights(output_name)

main函数调用：

import numpy as np
import cv2
import read_data_path as rp
import get_anchors as ga
import yolov3_model as ym
import train as trclass_dictionary = {'aeroplane': 0, 'bicycle': 1, 'bird': 2, 'boat': 3, 'bottle': 4, 'bus': 5,'car': 6, 'cat': 7, 'chair': 8, 'cow': 9, 'diningtable': 10, 'dog': 11,'horse': 12, 'motorbike': 13, 'person': 14, 'pottedplant': 15, 'sheep': 16,'sofa': 17, 'train': 18, 'tvmonitor': 19}
class_list = list(class_dictionary.keys())def txt_document(matrix):f = open("yolo_anchors.txt", 'w')row = np.shape(matrix)[0]for i in range(row):if i == 0:x_y = "%d,%d" % (matrix[i][0], matrix[i][1])else:x_y = ", %d,%d" % (matrix[i][0], matrix[i][1])f.write(x_y)f.close()def sigmoid(x):return 1/(1 + np.exp(-x))if __name__ == "__main__":train_x, train_y, val_x, val_y, test_x, test_y = rp.make_data()anchors = ga.calculate_anchor(train_x, train_y)# [[22  37]#  [26  82]#  [49  132]#  [56  57]#  [89  211]#  [113 108]#  [162 298]#  [238 177]#  [341 340]]txt_document(anchors)print('The anchors have been documented ! ')train_generator = tr.SequenceData(train_x, train_y, 32)validation_generator = tr.SequenceData(val_x, val_y, 32)# tr.train_network(train_generator, validation_generator, epoch=10)# tr.load_network_then_train(train_generator, validation_generator, epoch=15,#                            input_name='first_weights.hdf5', output_name='second_weights.hdf5')yolo3_model = ym.create_yolo_model()yolo3_model.load_weights('second_weights.hdf5')for i in range(len(train_x) - 1010, len(train_x) - 999):img1 = cv2.imread(train_x[i])size = img1.shapeimg2 = img1 / 255img3 = cv2.resize(img2, (416, 416), interpolation=cv2.INTER_AREA)img4 = img3[np.newaxis, :, :, :]pre = yolo3_model.predict(img4)pre_13 = np.reshape(pre[0][0], [13, 13, 3, 25])pre_26 = np.reshape(pre[1][0], [26, 26, 3, 25])pre_52 = np.reshape(pre[2][0], [52, 52, 3, 25])grid_y_13 = np.tile(np.reshape(np.arange(0, 13), [-1, 1]), [1, 13]) * size[0] / 13# 0   0   0   ...# 32  32  32  ...# ... ...     ...# 384 384 384 ...grid_x_13 = np.tile(np.reshape(np.arange(0, 13), [1, -1]), [13, 1]) * size[1] / 13# 0  32  64 ...# 0  32  64 ...# ...  ...  ...# 0  32  64 ...grid_y_26 = np.tile(np.reshape(np.arange(0, 26), [-1, 1]), [1, 26]) * size[0] / 26   # (26, 26)grid_x_26 = np.tile(np.reshape(np.arange(0, 26), [1, -1]), [26, 1]) * size[1] / 26   # (26, 26)grid_y_52 = np.tile(np.reshape(np.arange(0, 52), [-1, 1]), [1, 52]) * size[0] / 52   # (52, 52)grid_x_52 = np.tile(np.reshape(np.arange(0, 52), [1, -1]), [52, 1]) * size[1] / 52   # (52, 52)for j in range(3):pre_13[:, :, j, 0] = sigmoid(pre_13[:, :, j, 0]) * size[1] / 13 + grid_x_13pre_13[:, :, j, 1] = sigmoid(pre_13[:, :, j, 1]) * size[0] / 13 + grid_y_13pre_13[:, :, j, 2] = np.exp(pre_13[:, :, j, 2]) * anchors[j + 6, 0]pre_13[:, :, j, 3] = np.exp(pre_13[:, :, j, 3]) * anchors[j + 6, 1]pre_13[:, :, j, 4] = sigmoid(pre_13[:, :, j, 4])pre_13[:, :, j, 5:] = sigmoid(pre_13[:, :, j, 5:])pre_26[:, :, j, 0] = sigmoid(pre_26[:, :, j, 0]) * size[1] / 26 + grid_x_26pre_26[:, :, j, 1] = sigmoid(pre_26[:, :, j, 1]) * size[0] / 26 + grid_y_26pre_26[:, :, j, 2] = np.exp(pre_26[:, :, j, 2]) * anchors[j + 3, 0]pre_26[:, :, j, 3] = np.exp(pre_26[:, :, j, 3]) * anchors[j + 3, 1]pre_26[:, :, j, 4] = sigmoid(pre_26[:, :, j, 4])pre_26[:, :, j, 5:] = sigmoid(pre_26[:, :, j, 5:])pre_52[:, :, j, 0] = sigmoid(pre_52[:, :, j, 0]) * size[1] / 52 + grid_x_52pre_52[:, :, j, 1] = sigmoid(pre_52[:, :, j, 1]) * size[0] / 52 + grid_y_52pre_52[:, :, j, 2] = np.exp(pre_52[:, :, j, 2]) * anchors[j, 0]pre_52[:, :, j, 3] = np.exp(pre_52[:, :, j, 3]) * anchors[j, 1]pre_52[:, :, j, 4] = sigmoid(pre_52[:, :, j, 4])pre_52[:, :, j, 5:] = sigmoid(pre_52[:, :, j, 5:])candidate_box = []for k1 in range(13):for k2 in range(13):for k3 in range(3):if pre_13[k1, k2, k3, 4] > 0.1:center_x = pre_13[k1, k2, k3, 0]center_y = pre_13[k1, k2, k3, 1]w = pre_13[k1, k2, k3, 2]h = pre_13[k1, k2, k3, 3]confidence = pre_13[k1, k2, k3, 4]category = np.argmax(pre_13[k1, k2, k3, 5:])x1 = center_x - w/2y1 = center_y - h/2x2 = center_x + w/2y2 = center_y + h/2category = int(category)candidate_box.append([x1, y1, x2, y2, confidence, category])print('Grid 13 * 13 :', x1, y1, x2, y2, confidence, category)for k1 in range(26):for k2 in range(26):for k3 in range(3):if pre_26[k1, k2, k3, 4] > 0.1:center_x = pre_26[k1, k2, k3, 0]center_y = pre_26[k1, k2, k3, 1]w = pre_26[k1, k2, k3, 2]h = pre_26[k1, k2, k3, 3]confidence = pre_26[k1, k2, k3, 4]category = np.argmax(pre_26[k1, k2, k3, 5:])x1 = center_x - w / 2y1 = center_y - h / 2x2 = center_x + w / 2y2 = center_y + h / 2category = int(category)candidate_box.append([x1, y1, x2, y2, confidence, category])print('Grid 26 * 26 :', x1, y1, x2, y2, confidence, category)for k1 in range(52):for k2 in range(52):for k3 in range(3):if pre_52[k1, k2, k3, 4] > 0.1:center_x = pre_52[k1, k2, k3, 0]center_y = pre_52[k1, k2, k3, 1]w = pre_52[k1, k2, k3, 2]h = pre_52[k1, k2, k3, 3]confidence = pre_52[k1, k2, k3, 4]category = np.argmax(pre_52[k1, k2, k3, 5:])x1 = center_x - w / 2y1 = center_y - h / 2x2 = center_x + w / 2y2 = center_y + h / 2category = int(category)candidate_box.append([x1, y1, x2, y2, confidence, category])print('Grid 52 * 52 :', x1, y1, x2, y2, confidence, category)candidate_box = np.array(candidate_box)for num in range(len(candidate_box)):a1 = int(candidate_box[num, 0])b1 = int(candidate_box[num, 1])a2 = int(candidate_box[num, 2])b2 = int(candidate_box[num, 3])confidence = str(candidate_box[num, 4])index = int(candidate_box[num, 5])pre_class = class_list[index]cv2.rectangle(img1, (a1, b1), (a2, b2), (0, 0, 255), 2)cv2.putText(img1, pre_class, (a1, int((b1+b2)/2)), 1, 1, (0, 0, 255))cv2.putText(img1, confidence, (a2, int((b1+b2)/2)), 1, 1, (0, 0, 255))cv2.namedWindow("Final_Image")cv2.imshow("Final_Image", img1)cv2.waitKey(0)cv2.imwrite("/home/archer/CODE/YOLOv3/demo/" + str(i) + '.jpg', img1)

十一、项目链接

如果代码跑不通，或者想直接使用训练好的模型，可以去下载项目链接：
https://blog.csdn.net/Twilight737

这篇关于YOLOv3_目标检测的文章就介绍到这儿，希望我们推荐的文章对编程师们有所帮助！

---