| Architecture . | Representative Models . | Model Features . |
|---|---|---|
| Deep belief net | RBM (Zhang et al., 2018) | A generative graphic model that uses the energy to capture the probability distribution between visible units and hidden units. |
| SRBM (Chen et al., 2017) | A sparse variant that each hidden unit connects to part of the visible units, preventing the model overfitting based on hierarchical latent tree analysis. | |
| FRBM (Ning et al., 2018) | A fast variant trained by the lean CD algorithm in which the bounds-based filtering and delta product reduce the redundant dot product calculations. | |
| TTRBM (Ju et al., 2019) | A compact variant that the parameters between the visible layer and hidden layer are reduced by transforming into the tensor-train format. | |
| Stacked autoencoder | AE (Michael et al., 2018) | A basic fully connected network that uses the encoder-decoder strategy in an unsupervised manner to learn intrinsic features of data. |
| DAE (Vincent et al., 2008) | A denoising variant that reconstructs the clear data from the noising data. | |
| SAE (Makhzani & Frey, 2013) | A sparse variant that captures the sparse representations of the input by adding the constraint into the loss function. | |
| GAE (Hou et al., 2019) | An adversarial variant that the decoder subnetwork that is also regarded as the generator, adopting game theory to more consistent features with input data. | |
| FAE (Ashfahani et al., 2019) | An evolving variant that constructs an adaptive network structure in the learning of representations, based on the network significance. | |
| BAE (Angshul, 2019) | An evolving variant adding the path-loss term in the loss function based on dictionary learning. | |
| Convolutional neural network | Alexnet (Krizhevsky, Sutskever, & Hinton, 2012) | The nonsaturating neurons and the dropout are adopted in the nonlinear computational layers, based on a GPU implementation, respectively. |
| ResNet (He et al., 2016) | A shortcut connection is used to cross several layers to back propagate the network loss to previous layers. | |
| Inception (Christian et al., 2017) | A deeper and wider network is designed by using the uniform grid size for the blocks with auxiliary information. | |
| SEnet (Cao et al., 2019) | Informational embedding and adaption recalibration are regarded as self-attention operations. | |
| ECNN (Sandler et al., 2018) | The low-rank convolution replaces the full-rank convolution to improve the learning efficiency without much accuracy loss. | |
| Recurrent neural network | RNN (Zhang et al., 2014) | A fully connected network where the self-connection between hidden layers is used to model the time dependency. |
| BiRNN (Schuster & Paliwal, 1997) | Two independent computing processes are used to encode the forward and the backward dependency. | |
| LSTM (Hochreiter & Schmidhuber, 1997) | The memory block is introduced to model the long-time dependency well. | |
| SRNN (Lei et al., 2018) | A fast variant in which the light recurrence and highway network are proposed to improve the learning efficiency for a parallelized implementation. | |
| VRNN (Jang et al., 2019) | A variational variant that uses the variational encoder-decoder strategy to model the temporal intrinsic features. |
| Architecture . | Representative Models . | Model Features . |
|---|---|---|
| Deep belief net | RBM (Zhang et al., 2018) | A generative graphic model that uses the energy to capture the probability distribution between visible units and hidden units. |
| SRBM (Chen et al., 2017) | A sparse variant that each hidden unit connects to part of the visible units, preventing the model overfitting based on hierarchical latent tree analysis. | |
| FRBM (Ning et al., 2018) | A fast variant trained by the lean CD algorithm in which the bounds-based filtering and delta product reduce the redundant dot product calculations. | |
| TTRBM (Ju et al., 2019) | A compact variant that the parameters between the visible layer and hidden layer are reduced by transforming into the tensor-train format. | |
| Stacked autoencoder | AE (Michael et al., 2018) | A basic fully connected network that uses the encoder-decoder strategy in an unsupervised manner to learn intrinsic features of data. |
| DAE (Vincent et al., 2008) | A denoising variant that reconstructs the clear data from the noising data. | |
| SAE (Makhzani & Frey, 2013) | A sparse variant that captures the sparse representations of the input by adding the constraint into the loss function. | |
| GAE (Hou et al., 2019) | An adversarial variant that the decoder subnetwork that is also regarded as the generator, adopting game theory to more consistent features with input data. | |
| FAE (Ashfahani et al., 2019) | An evolving variant that constructs an adaptive network structure in the learning of representations, based on the network significance. | |
| BAE (Angshul, 2019) | An evolving variant adding the path-loss term in the loss function based on dictionary learning. | |
| Convolutional neural network | Alexnet (Krizhevsky, Sutskever, & Hinton, 2012) | The nonsaturating neurons and the dropout are adopted in the nonlinear computational layers, based on a GPU implementation, respectively. |
| ResNet (He et al., 2016) | A shortcut connection is used to cross several layers to back propagate the network loss to previous layers. | |
| Inception (Christian et al., 2017) | A deeper and wider network is designed by using the uniform grid size for the blocks with auxiliary information. | |
| SEnet (Cao et al., 2019) | Informational embedding and adaption recalibration are regarded as self-attention operations. | |
| ECNN (Sandler et al., 2018) | The low-rank convolution replaces the full-rank convolution to improve the learning efficiency without much accuracy loss. | |
| Recurrent neural network | RNN (Zhang et al., 2014) | A fully connected network where the self-connection between hidden layers is used to model the time dependency. |
| BiRNN (Schuster & Paliwal, 1997) | Two independent computing processes are used to encode the forward and the backward dependency. | |
| LSTM (Hochreiter & Schmidhuber, 1997) | The memory block is introduced to model the long-time dependency well. | |
| SRNN (Lei et al., 2018) | A fast variant in which the light recurrence and highway network are proposed to improve the learning efficiency for a parallelized implementation. | |
| VRNN (Jang et al., 2019) | A variational variant that uses the variational encoder-decoder strategy to model the temporal intrinsic features. |
Notes: RBM: restricted Boltzmann machine; SRBM: sparse restricted Boltzmann machine; FRBM: fast restricted Boltzmann machine; TTRBM: tensor-train restricted Boltzmann machine; AE: autoencoder; DAE: denoising autoencoder; SAE: K-sparse autoencoder; GAE: generative autoencoder; FAE: fast autoencoder; BAE: blind autoencoder; Alexnet: Alex convolutional net; ResNet: residual convolutional net; Inception: Inception; SEnet: squeeze excitation network; ECNN: efficient convolutional neural network; RNN: recurrent neural network; BiRNN: bidirectional recurrent neural network; LSTM: long short-term memory; SRNN: slight recurrent neural network; VRNN: variational recurrent neural network.