【Tensorflow tf 掏粪记录】笔记二——用SVM打造全连接神经网络识别MNIST数据集

由于我个人的原因，在写了第一篇tensorflow安装教程后就暂停了更新博客。可是我并没有停止前进的步伐，因为我习惯了做书面的笔记了。可是书面笔记，容易破损啊啊啊啊！！！

在深度学习中，入门的项目是MNIST数据集的分类识别。就好比我们学各种编程语言时候的Hello World！

下载MNIST数据集

从这里下载MNIST数据集。点击如下图所示按钮下载：
训练图片，训练标签，测试图片，测试标签

入门级别数据集，不大才11MB。很快就下载好了。

当然我们也可以用如下代码在程序中下载，不过我当时做的项目中没有用此方法。

from tensorflow.examples.tutorials.mnist import input_data
mnist = input_data.read_data_sets( 'MNIST_data' )

在这个项目里，我并没有用多少tensorflow提供的API接口来实现，更多是用tf的运算来实现整个神经网络。（真的累）

开车先造轮(加载MNIST数据)

想代码跑起来，轮子必不可少。读取数据方面的轮子是参考一个关于MNIST开源的项目写的。

调包

首先调包。(每个程序猿都是一名合格的调包侠)

import math
import numpy as np
import h5py
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow.python.framework import ops
from LoadData import DataUtils

数据的处理

读取数据

def main():
trainFile_x = 'D:/PycharmProjects/MNIST/MNIST_data/train-images.idx3-ubyte'    # 读取训练集文件，后面为训练集的存放地址trainFile_y = 'D:/PycharmProjects/MNIST/MNIST_data/train-labels.idx1-ubyte'    # 读取训练集标签文件，后面为训练集标签的存放地址testFile_x = 'D:/PycharmProjects/MNIST/MNIST_data/t10k-images.idx3-ubyte'    # 读取测试集文件，后面为测试集的存放地址testFile_y = 'D:/PycharmProjects/MNIST/MNIST_data/t10k-labels.idx1-ubyte'    # 读取测试集标签文件，后面为测试集标签的存放地址train_X = DataUtils( fileName = trainFile_x ).getImage()    # 获取训练集train_Y = DataUtils( fileName = trainFile_y ).getLabel()    # 获取训练集标签test_X = DataUtils( testFile_x ).getImage()    # 获取测试集test_Y = DataUtils( testFile_y ).getLabel()    # 获取测试集标签return train_X, train_Y, test_X, test_Y

one_hot_matrix

把训练集与测试集的标签按指定的个数大小进行tf.one_hot()
处理。

def one_hot_matrix( labels, C ):C = tf.constant( C, name = "C" )one_hot_matrix = tf.one_hot( labels, C, axis = 0 )one_hot = sess.run( one_hot_matrix )return one_hot

数组的转置

我们要把训练集与测试集转置下，因为我个人喜欢一张图片放为一个列。
同时我们也对标签进行了one_hot_matrix()处理

def fit_data():train_X, train_Y, test_X, test_Y = main()train_X, test_X, = train_X.T, test_X.T'''--------进行one_hot处理--------'''train_Y = one_hot_matrix( train_Y, 10 )test_Y = one_hot_matrix( test_Y, 10 )

随机mini_batches

def random_mini_batches( X, Y, mini_batch_size ):m = X.shape[1]    # 获取MNIST数据集中的训练集的个数mini_batches = []'''-------实现随机操作--------'''permutation = list( np.random.permutation( m ) )shuffled_X = X[:, permutation]shuffled_Y = Y[:, permutation]'''--------进行mini_batches分配--------'''num_complete_minibatches = math.floor( m / mini_batch_size )for k in range( 0, num_complete_minibatches ):mini_batch_X = shuffled_X[:, k * mini_batch_size : mini_batch_size * ( k + 1 )]mini_batch_Y = shuffled_Y[:, k * mini_batch_size : mini_batch_size * ( k + 1 )]mini_batch = ( mini_batch_X, mini_batch_Y )mini_batches.append( mini_batch )'''--------把剩下的训练集分到一个batch中--------'''if m % mini_batch_size != 0:mini_batch_X = shuffled_X[:, mini_batch_size * num_complete_minibatches : m]mini_batch_Y = shuffled_Y[:, mini_batch_size * num_complete_minibatches : m]'''--------把对应的训练集测试集按元组组成mini_batch并放入mini_batches中--------'''mini_batch = ( mini_batch_X, mini_batch_Y )mini_batches.append( mini_batch )return mini_batches

初始化参数

def initialize_parameters():'''--------初始化第一层神经网络--------'''W1 = tf.get_variable( "W1", [40, 784], initializer = tf.contrib.layers.xavier_initializer() )b1 = tf.get_variable( "b1", [40, 1], initializer = tf.zeros_initializer() )'''--------初始化第二层神经网络--------'''W2 = tf.get_variable( "W2", [20, 40], initializer = tf.contrib.layers.xavier_initializer() )b2 = tf.get_variable( "b2", [20, 1], initializer = tf.zeros_initializer() )'''--------初始化第三层神经网络--------'''W3 = tf.get_variable( "W3", [10, 20], initializer = tf.contrib.layers.xavier_initializer() )b3 = tf.get_variable( "b3", [10, 1], initializer = tf.zeros_initializer() )'''-------列了个参数表--------'''parameters = {"W1" : W1,"b1" : b1,"W2" : W2,"b2" : b2,"W3" : W3,"b3" : b3}return parameters

我初始化了3层40深的全连接神经网络，当时闹着玩的，初始化如此奇葩的全连接神经网络。结果训练结果还不错，没出现过拟合的情况。在这里我还列了个map方便自己以后调用参数。

向前传播

我是手撸SVM来实现的神经网络。
因为感觉用tf封装好的tf.layers.dense() 太简单了。
这里用relu激活函数并且实现了3层神经元的计算。

def forward_propagation( X, parameters ):'''--------读表--------'''W1 = parameters['W1']b1 = parameters['b1']W2 = parameters['W2']b2 = parameters['b2']W3 = parameters['W3']b3 = parameters['b3']'''--------实现神经元的计算--------'''Z1 = tf.matmul( W1, X ) + b1A1 = tf.nn.relu( Z1 )Z2 = tf.matmul( W2, A1 ) + b2A2 = tf.nn.relu( Z2 )Z3 = tf.matmul( W3, A2 ) + b3return Z3

计算损失

这里我用的是tf的tf.nn.softmax_cross_entropy_with_logits(logits, labels) 来计算训练时的生成的结果与标签的误差。

def compute_cost( Z3, Y ):logits = tf.transpose( Z3 )labels = tf.transpose( Y )'''--------损失计算--------'''cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits( logits = logits, labels = labels ) )return cost

训练的代码

我们每100epoch输出一次Cost，每5epoch记录一次Cost，最后用训练好的模型跑一遍测试集和训练集，来得出对不同数据集的准确率。

def model( train_X, train_Y, test_X, test_Y, learning_rate = 0.0001, num_epochs = 300, minibatch_size = 32, print_cost = True ):ops.reset_default_graph()( n_x, m ) = train_X.shapen_y = train_Y.shape[0]costs = []X, Y = create_placeholeers( n_x, n_y )parameters = initialize_parameters()Z3 = forward_propagation( X, parameters )cost = compute_cost( Z3, Y )'''--------用Adam正则化--------'''optimizer = tf.train.AdamOptimizer( learning_rate = learning_rate ).minimize( cost )init = tf.global_variables_initializer()with tf.Session() as sess:sess.run( init )for epoch in range( num_epochs ):epoch_cost = 0num_minibatches = int( m / minibatch_size )minibatches = random_mini_batches( train_X, train_Y, minibatch_size )for minibatch in minibatches:( minibatch_X, minibatch_Y ) = minibatch_, minibatch_cost = sess.run( [optimizer, cost], feed_dict = {X : minibatch_X, Y : minibatch_Y})epoch_cost += minibatch_cost / num_minibatches   #全局成本行函数if print_cost == True and epoch % 100 == 0:print ( "Cost after epoch %i: %f" % ( epoch, epoch_cost ) )if print_cost == True and epoch % 5 == 0:costs.append( epoch_cost )plt.plot( np.squeeze( costs ) )plt.ylabel( "cost" )plt.xlabel( "iteration ( per tens)" )plt.title( "Learning rate = " + str( learning_rate ) )plt.show()parameters = sess.run( parameters )print ( "Parameters have been trained" )correct_prediction = tf.equal( tf.argmax( Z3 ), tf.argmax( Y ) )accuracy = tf.reduce_mean( tf.cast( correct_prediction, "float" ) )print( "Train Accuracy:", accuracy.eval( {X : train_X, Y : train_Y} ) )print( "Test Accuracy:", accuracy.eval( {X : test_X, Y : test_Y} ) )sess.close()return parameters

用训练好的模型进行指定类型输出

这里我们用指定的数据集X1，希望从模型中得出属于index列的图片的数字
其中输出的Machine是机器识别图像预测的数字，Y为数据集带的标签

def predict( parameters, X1, index ):X, Y = create_placeholeers(X1.shape[0], 0)Z3 = forward_propagation(X, parameters)perdictions = tf.argmax(Z3[:, index])with tf.Session() as sess:print("Machine:", perdictions.eval( {X : X1} ) )

运行后会有如下图所示的cost显示

损失函数

完整代码

LoadData.py 代码

import numpy as np
import struct
import matplotlib.pyplot as plt
import osclass DataUtils( object ):def __init__( self, fileName = None, outPath = None ):self._fileName = fileNameself._outPath = outPathself._tag = '>'self._twoBytes = 'II'self._fourBytes = 'IIII'self._pictureBytes = '784B'self._labelBytes = '1B'self._twoBytes2 = self._tag + self._twoBytesself._fourBytes2 = self._tag + self._fourBytes    #>IIII’指的是使用大端法读取4个unsinged int 32 bit integerself._pictureBytes2 = self._tag + self._pictureBytes    #>784B’指的是使用大端法读取784个unsigned byteself._labelBytes2 = self._tag + self._labelBytesdef getImage( self ):"""将MNIST的二进制文件转换成新书特征数据"""binFile = open( self._fileName, 'rb' )    #以二进制的方式打开文件buffer = binFile.read()binFile.close()index = 0numMagic, numImages, numRows, numColumns = struct.unpack_from( self._fourBytes2, buffer, index )index += struct.calcsize( self._fourBytes )images = []for i in range( numImages ):imageValue = struct.unpack_from( self._pictureBytes2, buffer, index )     #struct.unpack_from( fmt, buffer, offser )从offset位置开始解包buffer，按照fmt格式输出index += struct.calcsize( self._pictureBytes2 )imageValue = list( imageValue )for j in range( len( imageValue ) ):if imageValue[j] > 1:imageValue[j] = 1images.append( imageValue )return np.array( images )def getLabel( self ):"""将MNIST中label二进制文件转化成对应的label数字特征"""binFile = open( self._fileName, 'rb' )buffer = binFile.read()binFile.close()index = 0magic, numItems = struct.unpack_from( self._twoBytes2, buffer, index )index += struct.calcsize( self._twoBytes2 )labels = [];for x in range( numItems ):im = struct.unpack_from( self._labelBytes2, buffer, index )index += struct.calcsize( self._labelBytes2 )labels.append( im[0] )return np.array( labels )def outImage( self, arrX, arrY ):"""根据生成的特征和数字标号，输出png图像"""m, n = np.shape( arrX )# 每张图是28 * 28 = 784bytefor i in range( 1 ):image = np.array( arrX[i] )image = image.reshape( 28, 28 )outFile = str( i ) + "_" + str( arrY[ i ] ) + '.png'plt.figure()plt.imshow( image, cmap = 'binary' ) #将图片黑白显示
plt.savefig( self._outPath + '/' + outFile )

Util.py 代码

import math
import numpy as np
import h5py
import matplotlib.pyplot as plt
import tensorflow as tf
from tensorflow.python.framework import ops
from LoadData import DataUtilssess = tf.InteractiveSession()def main():trainFile_x = 'D:/PycharmProjects/MNIST/MNIST_data/train-images.idx3-ubyte'trainFile_y = 'D:/PycharmProjects/MNIST/MNIST_data/train-labels.idx1-ubyte'testFile_x = 'D:/PycharmProjects/MNIST/MNIST_data/t10k-images.idx3-ubyte'testFile_y = 'D:/PycharmProjects/MNIST/MNIST_data/t10k-labels.idx1-ubyte'train_X = DataUtils( fileName = trainFile_x ).getImage()train_Y = DataUtils( fileName = trainFile_y ).getLabel()test_X = DataUtils( testFile_x ).getImage()test_Y = DataUtils( testFile_y ).getLabel()return train_X, train_Y, test_X, test_Ydef data_test():# Loading the datasettrain_X, train_Y, test_X, test_Y = fit_data()print(train_X.shape, train_Y.shape, test_X.shape, test_Y.shape)index = 0image = train_X[:, index]print( image.shape )image = image.reshape(28, -1)print( image.shape )plt.imshow( image )plt.show()print( "Y = " + str( np.squeeze( train_Y[:, index] ) ) )print( "Y = " + str( np.argmax( train_Y[:, index] ) ) )def fit_data():train_X, train_Y, test_X, test_Y = main()train_X, test_X, = train_X.T, test_X.Ttrain_Y = one_hot_matrix( train_Y, 10 )test_Y = one_hot_matrix( test_Y, 10 )print ( train_X.shape, train_Y.shape, test_X.shape, test_Y.shape )return train_X, train_Y, test_X, test_Ydef one_hot_matrix( labels, C ):C = tf.constant( C, name = "C" )one_hot_matrix = tf.one_hot( labels, C, axis = 0 )one_hot = sess.run( one_hot_matrix )return one_hotdef create_placeholeers( n_x, n_y ):X = tf.placeholder( tf.float32, [n_x, None] )Y = tf.placeholder( tf.float32, [n_y, None] )return X, Ydef initialize_parameters():W1 = tf.get_variable( "W1", [40, 784], initializer = tf.contrib.layers.xavier_initializer() )b1 = tf.get_variable( "b1", [40, 1], initializer = tf.zeros_initializer() )W2 = tf.get_variable( "W2", [20, 40], initializer = tf.contrib.layers.xavier_initializer() )b2 = tf.get_variable( "b2", [20, 1], initializer = tf.zeros_initializer() )W3 = tf.get_variable( "W3", [10, 20], initializer = tf.contrib.layers.xavier_initializer() )b3 = tf.get_variable( "b3", [10, 1], initializer = tf.zeros_initializer() )parameters = {"W1" : W1,"b1" : b1,"W2" : W2,"b2" : b2,"W3" : W3,"b3" : b3}return parametersdef forward_propagation( X, parameters ):W1 = parameters['W1']b1 = parameters['b1']W2 = parameters['W2']b2 = parameters['b2']W3 = parameters['W3']b3 = parameters['b3']Z1 = tf.matmul( W1, X ) + b1A1 = tf.nn.relu( Z1 )Z2 = tf.matmul( W2, A1 ) + b2A2 = tf.nn.relu( Z2 )Z3 = tf.matmul( W3, A2 ) + b3return Z3def compute_cost( Z3, Y ):logits = tf.transpose( Z3 )labels = tf.transpose( Y )cost = tf.reduce_mean( tf.nn.softmax_cross_entropy_with_logits( logits = logits, labels = labels ) )return costdef model( train_X, train_Y, test_X, test_Y, learning_rate = 0.0001, num_epochs = 300, minibatch_size = 32, print_cost = True ):ops.reset_default_graph()( n_x, m ) = train_X.shapen_y = train_Y.shape[0]costs = []X, Y = create_placeholeers( n_x, n_y )parameters = initialize_parameters()Z3 = forward_propagation( X, parameters )cost = compute_cost( Z3, Y )optimizer = tf.train.AdamOptimizer( learning_rate = learning_rate ).minimize( cost )init = tf.global_variables_initializer()with tf.Session() as sess:sess.run( init )for epoch in range( num_epochs ):epoch_cost = 0num_minibatches = int( m / minibatch_size )minibatches = random_mini_batches( train_X, train_Y, minibatch_size )for minibatch in minibatches:( minibatch_X, minibatch_Y ) = minibatch_, minibatch_cost = sess.run( [optimizer, cost], feed_dict = {X : minibatch_X, Y : minibatch_Y})epoch_cost += minibatch_cost / num_minibatches   #全局成本行函数if print_cost == True and epoch % 100 == 0:print ( "Cost after epoch %i: %f" % ( epoch, epoch_cost ) )if print_cost == True and epoch % 5 == 0:costs.append( epoch_cost )plt.plot( np.squeeze( costs ) )plt.ylabel( "cost" )plt.xlabel( "iteration ( per tens)" )plt.title( "Learning rate = " + str( learning_rate ) )plt.show()parameters = sess.run( parameters )print ( "Parameters have been trained" )correct_prediction = tf.equal( tf.argmax( Z3 ), tf.argmax( Y ) )accuracy = tf.reduce_mean( tf.cast( correct_prediction, "float" ) )print( "Train Accuracy:", accuracy.eval( {X : train_X, Y : train_Y} ) )print( "Test Accuracy:", accuracy.eval( {X : test_X, Y : test_Y} ) )sess.close()return parametersdef random_mini_batches( X, Y, mini_batch_size ):m = X.shape[1]mini_batches = []permutation = list( np.random.permutation( m ) )shuffled_X = X[:, permutation]shuffled_Y = Y[:, permutation]num_complete_minibatches = math.floor( m / mini_batch_size )for k in range( 0, num_complete_minibatches ):mini_batch_X = shuffled_X[:, k * mini_batch_size : mini_batch_size * ( k + 1 )]mini_batch_Y = shuffled_Y[:, k * mini_batch_size : mini_batch_size * ( k + 1 )]mini_batch = ( mini_batch_X, mini_batch_Y )mini_batches.append( mini_batch )if m % mini_batch_size != 0:mini_batch_X = shuffled_X[:, mini_batch_size * num_complete_minibatches : m]mini_batch_Y = shuffled_Y[:, mini_batch_size * num_complete_minibatches : m]mini_batch = ( mini_batch_X, mini_batch_Y )mini_batches.append( mini_batch )return mini_batchesdef predict( parameters, X1, index ):X, Y = create_placeholeers(X1.shape[0], 0)Z3 = forward_propagation(X, parameters)perdictions = tf.argmax(Z3[:, index])with tf.Session() as sess:print("Machine:", perdictions.eval( {X : X1} ) )def print_iamge( X, Y, index ):image = X[:, index]image = image.reshape( 28, -1 )plt.imshow(image)plt.show()print("Y = " + str( np.argmax( Y[:, index] ) ) )# init = tf.global_variables_initializer()
# sess.run(init)
# train_X, train_Y, test_X, test_Y = fit_data()
# predict( initialize_parameters(), test_X, 0 )
# print_iamge( test_X, test_Y, 6 )

MNIST.py代码

from Util import  fit_data, model, predict, print_iamgetrain_X, train_Y, test_X, test_Y = fit_data()
parameters = model( train_X, train_Y, test_X, test_Y )
predict( parameters, test_X, 6 )
print_iamge( test_X, test_Y, 6 )

项目代码

https://github.com/IronMastiff/MNIST

参考资料

https://www.coursera.org/specializations/deep-learning