Deep Learning intro and hands-on tutorialusers.auth.gr/passalis/etc/ml_meetup.pdf2000-Σμ :Deep...

Deep Learning DL Frameworks Darknet Keras

Deep Learning intro and hands-ontutorial

Π[email protected]

Ε . ώ Πώ. Π ΠΘ

1 / 53


Deep Learning

2 / 53


Δ

▶ Έ ( ) ω πμ , ώπ ώ π ω( , ...), π ώ ππ π π ω ω

▶ ώπ ώ π , ’(biologically plausible)

▶ Δ π ώώ π ώ▶ Ε π θ !

3 / 53


Ε π Δ

▶ 3 main ingredients!▶ ( Δ )▶ Δ ώ▶ (loss/objective function)

4 / 53


▶ 1943: Π ώ θ ώ ώ(Warren McCulloch and Walter Pitts)

▶ 1940-50: Π π π θ πώ ( neuroscience-oriented)

▶ 1958: Perceptron (Frank Rosenblatt)▶ 1975: Backpropagation▶ 1980-2000: ώπ π

▶ ώ ππ π

▶ Ο ώ (π. . SVMs) θώ π π

▶ π π ώ▶ Δ π π ώ▶ Δ π ώ

5 / 53


▶ 2000-Σ μ : Deep Learning▶ π state-of-the-art ππ tasks

▶ Χ ώ π ώ π π πμ

▶ ώ (GPUs) π ππ ώ

▶ Μ π μ ω θ ώπ (big data)

▶ Ν μ π (relu, Adam,...)

6 / 53


Ε ώ

7 / 53


8 / 53


9 / 53


Deep Learning▶ A π ώ Π( ώ )

▶ π information retrieval routing algorithms....▶ Data mining, analytics, ...

▶ Έ π π valuable skills ώ

10 / 53


Deep Learning Startups

▶ Affectiva (emotion recognition from videos or images)

▶ Gridspace (Conversationally-aware software, speechrecognition, speaker identification, critical moment)

11 / 53



▶ Ditto Labs (brand identification)

▶ Nervana (silicon to software optimized framework for deeplearning)

▶ Finance, energy, online services, ...

12 / 53



▶ Deep Genomics (precision medicine, understanding diseases,developing genetic therapies)

▶ Indico (text+image analytics for developing new products)

13 / 53


Deep Learning Startups▶ Enlitic (medical image analysis)

▶ Deep Instinct ( cybersecurity, detects, predicts and preventsadvanced persistent threats in real time)

▶ MarianaIQ (B2B account-based marketing)

14 / 53


Deep Learning

▶ Η π π ώ θ

▶ Perceptron:

15 / 53


Deep Learning▶ 2-layer MLP:

16 / 53


Deep Learning▶ 3-layer MLP:

17 / 53


Deep Learning

▶ LeNet:

18 / 53


Deep Learning

▶ GoogleLeNet:

▶ ResNet-200: 200 layers ...

19 / 53


π Δ

▶ From scratch (C, Java, ...)▶ MATLAB▶ Ε ώ

▶ π π π π π (DeepLearning Frameworks)

▶ Torch, Theano, Tensorflow, ...▶ Caffe, CNTK, Darkenet, ...▶ π π Python

20 / 53


ώ π π Framework?

▶ Η π π π ώπ π

▶ π π ώ ,π

▶ Η ώ π

▶ Δ ώ ώ GPUs πdeployment ώώ π. . CUDA

▶ Έ ώ π ώ▶ Έ ώ ώ (π π )

21 / 53


GPU Computing▶ Ο GPUs π ώ hardware π ώ

ώ π π π ώ ππ (π. . π ώώ

, convolutions, ...)▶ π ώ▶ Δ π π▶ Η nvidia ώ π GPU computing ώ

( ώ , π. . bitcoin mining)

22 / 53


GPU Computing

▶ Ο π ώώ ώ π θ GPU▶ θ π θ π (cuBLAS,cuRAND, cuDNN, ... )

▶ θ ώ π π wrapper θπ π π (π. . python)

▶ Ε (ResNet-200, forward+backward pass):▶ Dual Xeon E5-2630 v3: 22425 ms▶ Titan X (Pascal): 297 ms▶ 75 π GPU▶ π π θ ώ ,

ώ ώ .

23 / 53


GPU Computing

▶ Δ ώ π , deployment!▶ Ο π π ώώ ώ π GPU-accelerated hardware(π.x. self-driving cars)

24 / 53


DL Frameworks

25 / 53


Deep Learning Frameworks

▶ Π π !▶ Caffe▶ CTNK▶ Tensorflow▶ Darknet▶ Theano▶ (Py)Torch▶ Keras, Blocks, Lasagne,

26 / 53


Caffe

▶ π π GPU-accelerated θ DeepLearning

▶ π ώ π θ ώ ωπ μμ μ

▶ Η μπ ώ ώ

▶ Δ π C++ Python▶ Π μ ω image analysis▶ Χ π projects large-scale μπ

μ▶ Caffe2 (supported by Facebook)

27 / 53


Tensorflow

▶ Google supported!▶ Η μ ώ▶ Π distributed▶ Yπ mobile▶ Π ώ π▶ APIs π (Python, C++, Java, ...)▶ Όώ :

▶ Π π π π π▶ Π ώ

28 / 53


CTNK

▶ Microsoft Cognitive Toolkit (CNTK)▶ Ε μ μ π ω μ ω▶ Π π ώ deployment▶ APIs C++, Python, C#

29 / 53


Darknet

▶ Lightweight minimal framework ώώ CCUDA

▶ π π ώ ώ▶ Δπ

▶

▶

▶

▶ ( ) π π▶ minimal dependencies ( )

▶ π ώ , θ ,

30 / 53


Theano

▶ θ ώώ Python▶ Π ώ▶ π ώ θ ώ

▶ Ο ώ ώ (ώ θ ώ )Theano ώ :

▶ π ώ ώ π π (C)▶ π π π π

▶ π CUDAGPU

▶ π handler python

▶ Large compilation cost, fast execution

31 / 53


PyTorch

▶ Π ώ Torch Python▶ π ώ θ π π

μ ( ώπ)

▶ Π ώ ( ώώπ ώ π θ π Caffe/Caffe2)

▶ π π π GPU▶ Π ώ π ώ ώ π

32 / 53


θ /Wrappers

▶ Π π π π πGPUs π deep learning models π

θ π π π π▶ Keras (Tensorflow, Theano, CTNK)▶ Lasagne (Theano)▶ Blocks (Theano)

33 / 53


Pre-trained models

▶ Η π ώώ π π ώθ ώ π ώ

▶ π ώ ώ π ώworkstations ώ 4 high end

▶ π π frameworks πώ pretrained ώ

▶ π ώ π internetώ π

34 / 53


Π Framework ώ ?

▶ Π !▶ π ώ π θώ ?▶ Δ θ ώ π ώ ?▶ Π θ ώ ?▶ Ε ?▶ Deployment target?▶ Χ pretrained models?

35 / 53


Darknet

36 / 53


Darknet

▶ Ε π π ώ π state-of-the-artπ ώ pretrained ώ

▶ Ε ( Linux)▶ π ώ π θ ώ

▶ http://pjreddie.com/darknet

37 / 53

http://pjreddie.com/darknet


ώ▶ http://pjreddie.com/darknet/yolo▶ ώ π pretrained ώ ώ (yolo-tiny,yolo, coco-tiny, coco-tiny)

▶ Η COCO ώ ώ ώ π ππ π ώ 80

38 / 53

http://pjreddie.com/darknet/yolo


ώ./darknet coco test cfg/yolo-coco.cfg data/yolo-coco.weightsimage.jpg

39 / 53


ώpredictions.png

40 / 53


Deep Dream (nightmares)./darknet-gpu nightmare cfg/vgg-conv.cfg data/vgg-conv.weightsimage.jpg 5

41 / 53



42 / 53



43 / 53



44 / 53


RNN text generation./darknet rnn generate cfg/rnn.cfg data/shakespeare.weights -srand0 -seed ”Hello there!” -len 500

1 Hel lo there ! There ’ s a good f r i end .2 COUNTESS. I am sure you can ; you have a dev i l were

made on3 her beauty , and the very night than he two f i gu r e s4 the worthy Achalle , and you sha l l make such a

messenger .5 I have not s ing l ed on the f i e l d ;6 The name of Caesar ’ s sun , when you should speak ,7 Which when they would be reveng ’d on him in th e i r

death8 But tha t I have rece iv ’d and say they9 Had s t range ly to h i s kingdom , and give her ,10 Upon the f i r s t here are both beams to love .

45 / 53


Keras

46 / 53


Keras

▶ θ Python▶ π backend Tensorflow (default), theanoCTNK

▶ Ε▶ pip install keras

▶ Ε backend (edit Α/.keras/keras.json)

47 / 53


Keras▶ Π π ώ well-known datasets (π. . MNIST)

1 from keras . da t a s e t s import mnist

▶ datasets ώ pre-processing

1 ( x_ t ra in , y_ t r a in ) , ( x_ t e s t , y _ t e s t ) = mnist . load_data( )

2 x_ t r a in = x_ t r a in . reshape (60000 , 784)3 x _ t e s t = x _ t e s t . reshape (10000 , 784)4 x_ t r a in = x_ t r a in . astype ( ’ f l oa t32 ’ ) / 255.05 x _ t e s t = x _ t e s t . as type ( ’ f l oa t32 ’ ) / 255.0

▶ keras π utilities ώ tasks (π. .ώ π labels binary )

1 y_ t ra in = keras . u t i l s . t o _ c a t e g o r i c a l ( y_ tra in , 10)2 y_ t e s t = keras . u t i l s . t o _ c a t e g o r i c a l ( y_ t e s t , 10)

48 / 53


Neural Networks in less than 10 lines▶ Model Definition

1 model = Sequent ia l ( )2 model . add(Dense (512 , a c t i v a t i on= ’ re lu ’ , input_shape

=(784 ,) ) )3 model . add(Dropout (0 .2 ) )4 model . add(Dense (512 , a c t i v a t i on= ’ re lu ’ ) )5 model . add(Dropout (0 .2 ) )6 model . add(Dense (10 , a c t i v a t i o n= ’ softmax ’ ) )7 model . compile ( l o s s= ’ c a t e go r i c a l _ c r o s s en t r opy ’ ,

opt imizer=Adam( ) , metr i c s =[ ’ accuracy ’ ] )▶ Training

1 h i s t o ry = model . f i t ( x_ t ra in , y_ tra in , ba t ch_s i z e =128 ,epochs=10 , verbose =2 , va l ida t i on_da ta =( x_ t e s t ,y _ t e s t ) )

▶ Testing1 score = model . eva lua te ( x_ t e s t , y_ t e s t , verbose=0)

49 / 53


Neural Networks in less than 10 lines▶ θ default θώ ώ θ

!

1 model . compile ( l o s s= ’ c a t e go r i c a l _ c r o s s en t r opy ’ ,opt imizer=Adam( l r =0.01) , metr i c s =[ ’ accuracy ’ ] )

learning rate epoch 1 epoch 5 epoch 100.0001 92.96% 96.91% 97.83%0.001 96.81% 97.95% 98.02%0.01 94.92% 96.04% 96.67%0.1 10.09% 10.09 % 10.09%

▶ GPU vs CPU▶ i5 4670 (4 cores, 32 bit): 19s per epoch▶ Geforce 1060: 2s per epoch

50 / 53


CNN in less than 10 lines

▶ Model Definition

1 model = Sequent ia l ( )2 model . add(Conv2D(32 , ke rne l _ s i z e =(3 , 3) , a c t i v a t i o n= ’

re lu ’ , input_shape=input_shape ) )3 model . add(Conv2D(64 , (3 , 3) , a c t i v a t i o n= ’ re lu ’ ) )4 model . add(MaxPooling2D ( poo l _ s i z e =(2 , 2) ) )5 model . add(Dropout (0 .25) )6 model . add( F la t t en ( ) )7 model . add(Dense (128 , a c t i v a t i on= ’ re lu ’ ) )8 model . add(Dropout (0 .5 ) )9 model . add(Dense ( num_classes , a c t i v a t i o n= ’ softmax ’ ) )

51 / 53


MLP vs CNN

▶ GPU vs CPU▶ i5 4670 (4 cores, 32 bit): 157s per epoch▶ Geforce 1060: 14s per epoch

▶ ώ MLP CNN▶ MLP: 98.02% (2s per epoch)▶ CNN: 99.19% (14s per epoch)▶ 10 π

52 / 53


Ε !

53 / 53

Deep Learning intro and hands-on tutorialusers.auth.gr/passalis/etc/ml_meetup.pdf2000-Σμ :Deep...

Documents

Transcript of Deep Learning intro and hands-on tutorialusers.auth.gr/passalis/etc/ml_meetup.pdf2000-Σμ :Deep...