JourneyToCoding

Some commonly used RNN models: GRU, LSTM, DRNN, BRNN...

CNN is good at processing spatial information but it is not good at processing sequence information. RNN (Recurrent Neural Network) can better process sequence information than other neural networks.

CPU, GPU, DPS, FPGA, ASIC(TPU).

Some commonly used CNN models: LeNet-5, AlexNet, VGG, NiN, GoogLeNet, ResNet.

CNN is a special kind of MLP. Why do we still need CNN even though MLP can work well? This involves a classic problem in the computer field: the trade-off between memory and computing speed. CNN is widely used in image processing. An image is characterized by its representation in the computer by millions of pixels. Each pixel is a feature of the image. It is unbearable for GPU to store so many model parameters for these features. Hence, we need CNN to compress the number of parameters and extract features from an image.

Discrete Fourier Transform (DFT) is a linear transform that converts a finite sequence of equally-spaced samples of a function in the time domain into a same-length sequence of equally-spaced samples in the frequency domain, which is a complex-valued function of frequency.

最大均值差异（Maximum Mean Discrepancy）是迁移学习（Transfer Learning）中常用的损失函数，它可以用来衡量两个随机变量分布的差异性。

Fourier transform has a strong conection with convolution.

Fourier transform is a transform that converts a function into a form that describes the frequencies present in the original function. To a certain extent, it can also be viewed as a coordinate transformation, which transform coordinate in time domain to frequency domain. It is the basis for understanding convolution and CNN.

In deep learning, numerical stability refers to making the training of the model stable and feasible. More spefically, numerical stability is the stability of model parameters, outputs and gradients.