Natural Neural Networks

Size: px

Start display at page:

Download "Natural Neural Networks"

牢宋
5 years ago
Views:

1 深度学习讨论班黄雷

2 内容大纲 1. 深度学习介绍神经网络的历史深度学习的应用 2. 多层感知机 (multi-layer perceptron machine) 前向神经网络 (feedforward neural network) 3. 卷积神经网络 (Convolution neural networks) 4. 递归神经网络 (Recursive neural networks) 5. 利用神经网络的针对具体问题建模如何设计损失函数是否采用端到端学习 6. 训练神经网络的实用技巧如何有效地训练如何提高模型的泛化能力深度学习知识层面深度学习应用实践层面

3 课程的目标 1. 在知识层面基本的术语三类经典的神经网络 2. 应用实践层面会针对具体的问题, 利用深度学习建模编程实践, 基于 torch 平台 ( 深度学习平台 )

4 相关资料深度学习课程牛津大学,Nando de Freitas, machinelearning/ Coursera, Geoffrey Hinton, Neural Network for Machine Learning 斯坦福大学,Fei-Fei Li,CS231

5 相关资料领域顶级会议 ICLR (International Conference on Learning Representation ) ICML CVPR, ACL, IJCAI, AAAI.

6 相关资料实验室 ftp: fileserver.nlsde.buaa.edu.cn/public/study/dee plearning/

7 Deep learning introduction Presented by Lei Huang November 29, 2016

8 Basic Concept Machine learning Neural network Deep network Outline History of neural network Perceptron BackPropagation Deep learning Application

9 Machine learning dataset D={X, Y} Goal Input: X Output: Y Learning: Y=F(X) or P(Y X) Automatically detect patterns in data Use the uncovered patters to predict future data Fitting and Generalization Y=F(X) P(Y X)

10 Types: view of data Machine learning Supervised Learning( 监督学习 ) D={X, Y} Learning: Y=F(X) or P(X,Y) Unsupervised Learning( 非监督学习 ) dataset D={X, X} Learning: X=F(g(X)), use G(x) as representation Types: view of models Non-parametric model ( 非参模型 ) Y=F(X; x 1, x 2 x n ) Parametric model( 参数化模型 ) Y=F(X; θ)

11 Neural network Neural network Y=F(X)=f T (f T 1 ( f 1 (X))) f i x = g(wx + b) Nonlinear activation sigmod Relu

12 Deep neural network Why deep? Powerful representation capacity( 函数表达能力 )

13 Key properties of Deep learning End to End learning ( 端到端学习 ) no distinction between feature extractor and classifier Deep architectures: cascade of simpler non-linear modules

14 Basic Concept Machine learning Neural network Deep network Outline History of neural network Perceptron BackPropagation Deep learning Application

15 The Perceptron 1957, Frank Rosenblatt, Perceptron( 感知机 ) Source: Intelligence artificielle Yann Le cun,

16 The Perceptron Source: Intelligence artificielle Yann Le cun,

17 The Perceptron Source: Intelligence artificielle Yann Le cun,

18 The Perceptron Source: Intelligence artificielle Yann Le cun,

19 The Perceptron Source: Intelligence artificielle Yann Le cun,

20 The Perceptron Source: Intelligence artificielle Yann Le cun,

21 The Perceptron Source: Intelligence artificielle Yann Le cun,

22 The Perceptron Source: Intelligence artificielle Yann Le cun,

23 The Perceptron Source: Intelligence artificielle Yann Le cun,

24 The Perceptron Source: Intelligence artificielle Yann Le cun,

25 AI Winter 1969, Minsky, perceptron XOR Two layers, computation impossible.

26 Conclusion for first phase Training by iteration Inference, calculate f(x) Compare the difference between the f(x) and y Adjust the weights. (gradient based) Source: Intelligence artificielle Yann Le cun,

27 Second: Back Propagation 1986, Backpropagation( 反向传播 ) Calculate gradient efficiently O(d^2) Routine of training Forward Back-propagation Update weights based on gradients

28 Beaten by SVM 1990s, Vapnik, support vector machines.( 支持向量机 ) y=f(wx+b) Globally optimization High efficiency, just one layer Loss function Function + loss Kernel trick for nonlinear

29 Third: deep learning 2006, Geoffery Hinton Deep Belief network Pre-training Fine-tuning

30 2011, audio Third: deep learning The task Hours of training data Deep neural network Gaussian Mixture Model GMM with more data Switchboard (Microsoft Research) English broadcast news (IBM) % 27.4% 18.6% (2000 hrs) % 18.8% Google voice search (android 4.1) 5, % (and falling) 16.0% (>>5,870 hrs)

31 2012,imageNet. Third: deep learning

32 2012,imageNet. Third: deep learning

33 Why deep learning grow so fast? Big Data More Powerful and cheaper machine Open Source Code: git-hub Paper: arxiv Source: 程序员的深度学习入门指南费良宏, 2016

34 Basic Concept Machine learning Neural network Deep network Outline History of neural network Perceptron BackPropagation Deep learning Application

35 Object Classification Application

36 Object detection Application

37 Scene Parsing Application

38 Application Automatic Image Caption Generation

39 Application Artistic style learning on images

40 Application Automatically Adding Sounds To Silent Movies

41 Application Automatic Handwriting Generation

42 Application Automatic Text Generation Shakespeare Wikipedia articles (including the markup) Algebraic Geometry (with LaTeX markup) Linux Source Code -effectiveness/

43 Alpha Go Application

44 Q&A

附件4

附件4 附件 2 2015 年省级精品资源共享课申报书 ( 本科 ) 申报学校西华师范大学课程名称数值方法与计算机技术课程类型公共基础课专业基础课专业课其它所属学科门类理学 ( 数学 ) 所属专业类名称课程负责人数学与应用数学信息与计