Exploring the teaching of deep learning in neural networks

翻译：探索神经网络深度学习的教学 -李睿凡，王小捷，钟义信
摘自：超星期刊 第19期 2014年10月10日
栏目：计算机教育
文章编号：1672-5913(2014)19-0077-03
中图分类号：G642

原文

译文：

1. School of Computer Science and Technology, Beijing University of Posts and Telecommunications, Beijing 100876, China;
2. Information Network Engineering Research Center, Ministry of Education, Beijing Dianli University, Beijing 100876, China

Abstract:
  Deep learning is the latest breakthrough in the field of intelligent scienc-e and technology, which will be the basic concepts, models, and metho-ds of deep learning.It is urgent to introduce the basic concepts, models and methods of deep learning into intelligent science and technology c-ourses.The article discusses how to effectively carry out deep learning teaching in undergraduate and graduate courses and introduce the latest research results in the field of intelligence to the highly innovative und-ergraduate and graduate groups.In order to be able to reach the forefr-ont of the discipline early, enhance the interest in intelligent science an-d technology, and stimulate the spirit of innov-ation.

Key Words: intelligent science and technology; deep learning; teaching advice

0 Preface
  The main specialized courses in intelligent science and technology inclu-de machine intelligence, pattern analysis, machine learning, data minin-g and other courses.According to the speed of development of this dis-cipline, although the content of the teaching materials of the specialize-d courses has been updated very rapidly, we still have to sigh over the rapid development of the intelligent science and technology disciplines.This constantly challenges teachers who are engaged in the first line of intelligent science and technology teaching.

  Neural network depth learning is one of the latest and hottest advances in intelligent science and technology, and its starting point is to construct a neural network with a typical two-layer structure.Since 2006, professor Geoffrey Hington of the university of Toronto, Canada, has been a master of neural networks in Science in the United States. The magazine published the issue of deep learning milestones entitled “Reducing the Dimensionality of Data with Neural Networks”. Since the beginning of the paper, there has been a breakthrough in deep learning, which is destined to affect our generation of teachers engaged in intelligent science and technology teaching and scientific research.We should seize the opportunity to meet the challenges and advance the new development in this area of intelligence.

  Beijing university of posts and telecommunications computer college has a bachelor’s degree in intelligent science and technology as well as a corresponding master’s degree and doctor’s degree. At the same time take on the teaching work of undergraduate and graduate students’ intelligent courses. In 2013, the author has proposed in the literature that the latest achievement of deep learning be introduced into the specialized courses of intelligent science and technology. Including its necessity and feasibility and preliminary implementation recommendations.

1 Deep learning background

  The basic starting point of deep learning is to construct and implement a multilayered learning neural network, which has a deep historical origin. In the early 1980s, with multilayer sensing and reverse propagation algorithms as the focal point, researchers in neural networks saw the great potential of neural networks. Although it is inevitable to try to construct a multilayer neural structure with more than two layers of structure from the research results of biological neural network and cognitive neurology, most of them end up in failure. There are historical limitations and cognitive limitations.
  At that time, the computer speed and the development level of hardware storage equipment were not high, and it was very difficult to construct large scale neural networks and carry out effective training. At the same time, the difficulties of reverse propagation algorithm and multilayer neural network training are biased in understanding. Until 2006, professor Hinton and his doctoral student, Salakhutdinov, proposed the use of layers of unsupervised pre-training with limited Boltzmann machines. The traditional reverse propagation algorithm is used to train the effective method of multilayer neural network, which becomes a fuse in the study of deep learning, which quickly leads to a system.

  From doubt to doubt is an important characteristic in the history of deep learning. Since the introduction of intensive learning in 2006 by professor Hinton, The battle was fought by professor Jitendra Malik, a computer vision guru from the university of California, Berkeley. Although deep learning broke the test record on recognition’s hand-written digital task which name is MINIST, there was no knowledge of the performance of deep learning on the problem of large-scale visual recognition.Professor hington’s team has embraced the challenge of attending Stanford university in 2012.Recognition evaluation of large-scale image of ImageNet ILSVRC by professor FeiLi and others.The task consists of 1.2 million high-resolution pictures and 1,000 analogies.Professor Hinton’s team used the structure of a seven-layer convoluted neural network to make a surprising breakthrough, reducing recognition’s error rate from 26.2% to 15.3%.There is no doubt that the performance of deep learning will go down in history from this moment on.The skeptics of the effects of deep learning have to face the fact that deep learning is really a sword.

  In the academic field, the study of intensive learning spread from neural networks to many fields, including the international conference on machine learning (ICML), the conference on neural information processing (NIPS),Computer vision Congress (ICCV), acoustic speech and signal processing Congress (ICASSP), computational linguistics Congress (ACL), computer vision and mode recognition (CVPR),Multi-media conferences (MM) and other topics with deep learning and seminars.At home, in June 2013, the magazine of programmers interviewed three experts in the field of machine learning in China, Zhou Zhihua, a professor at Nanjing university. Li hang, chief researcher at the Noah’s ark laboratory of Huawei technology Co., Ltd., and Zhu Jun, deputy researcher at Tsinghua university, discussed their views on deep learning. They unanimously affirm the contribution of deep learning in the field of machine learning and expect it in the long term. At the ICML conference held in Beijing in 2014, in addition to hington, Another leader in deep learning, Yoshio Yoshio, and several researchers in other fields.

  Industry is very interested in the application of deep learning.In November 2012, Microsoft publicly demonstrated a fully automated simultaneous interpretation system in Tianjin, the key supporting technology of which is deep learning.In January 2013, Mr. Li Yanhong, chief executive of Baidu, announced the establishment of the Institute of deep learning.In March 2013, Google acquired a company founded by Geoffrey Hinton, founder of deep learning;In December 2013, Facebook set up an artificial intelligence laboratory and appointed professor LeCunYann, a leader in depth learning at New York university, as its director.In January 2014, Google acquired deep learning
start-ups for $400 million which name is DeepMind Technologies;In may 2014, Baidu announced the appointment of “father of Google’s brain” AndrewNg as the chief scientist of Baidu’s research Institute.From the interest of academic and industrial circles in deep learning, deep learning has become a hot spot in machine learning and mode, recognition and even in the field of intelligence.

2 Teaching suggestion

In addition, teachers should take full account of the students’ knowledge background and learning characteristics, and choose appropriate teaching content according to their own characteristics and interests.In addition, teachers must distinguish between undergraduate and graduate students in the teaching of neural network depth in order to achieve the desired results.

2.1 For Undergraduates

  For undergraduates, the teaching goal of neural network deep learning is to make students understand the basic starting point of deep learning and grasp the basic content and initial application of deep learning.To understand how deep learning can become a breakthrough in intelligent science and technology, and to stimulate students’ interest in the research of deep learning on neural networks.In view of the above objectives, we suggest that there is no need for a separate intensive neural network learning course, but that a number of teaching modules be added to the intelligent classes of senior undergraduates.As a rule, the hours of specialized courses in the senior grade are limited (usually 32 to 36 hours), and we suggest that we can set up depth learning links between 4 and 6 hours.

  Based on the above considerations, the teaching content mainly includes two main parts: multi-layer sensor and classic backpass algorithm, self-encoder and unsupervised feature learning.In short, we can first explain the limitations of the single neural cell model and learning algorithm and its representation ability, and then lead to the neural network model of multi-layer sensor and the classic back learning algorithm.In unit 2, we introduce the concept of unsupervised feature learning, the neural network model of the automatic encoder and the learning algorithm to let students understand the relationship between the automatic encoder and the main component analysis method.It is used to initialize the parameters of the depth neural network model, adjust the weight process as a whole, and understand the relationship between the depth learning and the representation learning.In addition, if time permits, the teacher can arrange a lecture on the limited Boltzmann machine random model and random learning algorithm in the third unit. In particular, a random approximation of the contrast dispersion algorithm and the limited Boltzmann machine for the deep neural network learning process. Finally, as an example of practical application, teachers can use these deep neural network models in the tasks of handwritten digital recognition. At the same time, it is possible to compare the backtracking algorithm using multi-layer sensor, and ask students to write the corresponding program and write the experimental report.

  In general, students are required to master the basic content of neural network depth learning through the above content. To further enable students to think about how to understand the necessity of the deep neural network from the perspective of biological neural network and human cognition, how to construct and learn the deep learning network and how to use the deep structure for vision, How to treat deep learning and original machine learning methods such as flow learning, probability graph model, The direct relationship between energy model and depth learning and other disciplines.According to the above questions, teachers appropriately guide students to think and consult the literature, thus stimulating students’ interest in deep learning.

2.2 In Postgraduate Teaching

  In postgraduate teaching, we must consider some issues, such as the possibility of offering a more independent neural network course and a special teaching course.Many students from non-computer studies have not even been exposed to intelligent science and technology.Graduate students, especially doctoral students, are transformed from knowledge learning to knowledge creation. Therefore, the goal of our program is to make students understand the basic starting point of deep learning, master the main content of deep learning and the typical application of a field. To stimulate students to have a strong interest in the study of neural network depth, and put forward research proposals in areas of interest to them.

  Based on the above objectives, as distinct from the teaching arrangements for undergraduate students, we recommend postgraduate courses for the characteristics of postgraduate education (for example, in 36 studies).The main contents include neural network and machine learning knowledge, depth neural network foundation, depth neural network paper reading list 3.The basic knowledge of neural network and machine learning (2 hours) mainly enables students to make a smooth transition from zero starting point to this course.Teachers explain the main objectives of machine learning and linear classifier, main component analysis and linear discriminant analysis, neural network basic units and so on.Part 2 explains the contents of undergraduate teaching arrangements (6 hours);Part 3, students read the essay section.For the third part of the students’ reading and interpretation, we further divided it into four units, with the theme mainly including limited Polzman machine and its extension, automatic encoder and extension,Deep structure model, computer vision, typical application of speech and language processing.Due to the differences in research and teaching interests, we did not give a detailed list of papers for each unit, which left more space for the teaching profession.

  In addition to the teaching links, we also emphasize the use of what we have learned in the course of our graduate students’ teaching, which includes two aspects. One is an experimental project, which includes the basic model of deep learning, the experiment of the automatic encoder and the limited Boltzmann machine model and training algorithm, the experiment of the depth neural network model and the experiment of the training algorithm. More important is how to use the learned neural network depth learning method to solve possible problems.In the end, two to three students will be formed into a group to give them a period of time, asking them to put forward their own questions of explanation, and by communicating with the teachers, the students group will determine a research topic. Write research proposals, complete the study at a later time, write the study report and report the research in groups at the final hours of the course.

  Teachers also need to pay attention to improving students’ learning research initiative.It is also easy to learn in depth.For example, it is easy for students to have access to the top conference courses mentioned in the background section;Some of the recent technical lectures sponsored by the Chinese computer society involved intensive learning.The home page of professor hinton, founder of intensive learning, is rich in information; The coursera website offers a free neural network course taught by professor hinton;Online tutorials provided by professor andrewng at Stanford university are also excellent materials;The paper by professor bengio of the university of Montreal, learningdep archetecuresforai, is a good entry point in this regard.These can fully mobilize students’ initiative and greatly improve the quality of teaching.

  All in all, students can master the main contents of neural network deep learning through classroom teaching, programming practice and project research.To further understand the research motivation of deep learning and learn how to use the deep learning structure to solve the problems of visual, speech and language applications.From a deeper level, we should think about how to construct and learn a deep learning network and how these models relate to traditional machine learning models such as flow learning, probability graph model and energy model.These teaching content is helpful for students to change from simple knowledge learning to innovation consciousness.

In conclusion
  In conclusion, we discuss how to effectively carry out the teaching of deep learning in undergraduate and postgraduate courses, including the guiding ideology, content organization, teaching methods, etc.It is expected that students will be able to reach the forefront of this subject early, enhance their interest in intelligent science and technology, and stimulate their innovative spirit.Deep learning is still at a high development stage, and it is still immature, and there are still many contents in dispute.Play the role of promoting the teaching of deep learning.

后记：个人能力有限，若有错误之处，欢迎指正！