Multiple GPUs and Parallelism
CPU, GPU, DPS, FPGA, ASIC(TPU).
Common CNN Models
Some commonly used CNN models: LeNet-5, AlexNet, VGG, NiN, GoogLeNet, ResNet.
CNN: Feature Extraction
CNN is a special kind of MLP. Why do we still need CNN even though MLP can work well? This involves a classic problem in the computer field: the trade-off between memory and computing speed. CNN is widely used in image processing. An image is characterized by its representation in the computer by millions of pixels. Each pixel is a feature of the image. It is unbearable for GPU to store so many model parameters for these features. Hence, we need CNN to compress the number of parameters and extract features from an image.
Discrete Fourier Transform
Discrete Fourier Transform (DFT) is a linear transform that converts a finite sequence of equally-spaced samples of a function in the time domain into a same-length sequence of equally-spaced samples in the frequency domain, which is a complex-valued function of frequency.
Maximum Mean Discrepancy
最大均值差异（Maximum Mean Discrepancy）是迁移学习（Transfer Learning）中常用的损失函数，它可以用来衡量两个随机变量分布的差异性。
Fourier Transform and CNN
Fourier transform has a strong conection with convolution.
Fourier Transform
Fourier transform is a transform that converts a function into a form that describes the frequencies present in the original function. To a certain extent, it can also be viewed as a coordinate transformation, which transform coordinate in time domain to frequency domain. It is the basis for understanding convolution and CNN.
Numerical Stability
In deep learning, numerical stability refers to making the training of the model stable and feasible. More spefically, numerical stability is the stability of model parameters, outputs and gradients.
Model Selection & Hyperparameter Optimization
Hyperparameters are parameters of neural networks other than w and b, including learning rate, depth of layer, size of each layer and so on. Compared to model parameters w and b, hyperparameters are determined entirely by ourselves. Tuning a model is to find better hyperparameters.
D2L: Environment Configuration and Introduction
Dive into deep learning (D2L) is a online deep learning course by Mu Li. In this course, Mu Li teaches the basic concept of DL and usage of pytorch. This post includes the environment configuration in windows, introduction of this course and some basic concept about matrix derivative.