These results show that replacing Dropout with SDR in DenseNet tests yields various reductions in error in smaller networks, and $13%$ reduction in errors of top-5 accuracy on ImageNet (see Figure 3 and Table 1).
Figure 3:

Error plots for DenseNet-40 training error and DenseNet-BC 250 validation error. The training error for DenseNet-40 is reduced by 83%, and the validation error for DenseNet-BC 250 is reduced by 13%.

Figure 3:

Error plots for DenseNet-40 training error and DenseNet-BC 250 validation error. The training error for DenseNet-40 is reduced by 83%, and the validation error for DenseNet-BC 250 is reduced by 13%.

Table 1:
Top-1 Validation Error at End of Training of DenseNet-SDR Compared to DenseNet with Dropout ($p=0.2$).
Data Set
CIFAR-10CIFAR-100ImageNet
Dropout
DenseNet-40 6.88 28.31 –
DenseNet-100 BC 6.02 26.18 –
DenseNet-100 5.11 23.00 –
DenseNet 250 BC 5.18 22.44 –
DenseNet 121 BC – – 27.26/9.01
($k=32$
SDR-Decay
DenseNet-40 6.53 27.39 –
DenseNet-100 BC 5.24 ($-$13.0%) 23.45 –
DenseNet-100 4.87 22.10 ($-$3.9%) –
DenseNet 250 BC – 19.79 ($-$11.8%) –
SDR-Dynamic
DenseNet-40 6.10 ($-$11.3%) 25.63 ($-$9.5%) –
DenseNet-100 BC 5.56 23.19 ($-$11.4%) –
DenseNet-100 4.82 ($-$5.7%) 22.37 –
DenseNet 250 BC 4.68 (−9.7%) 21.45 –
DenseNet 121 BC – – 25.58 ($-$6.2%)/7.82 ($-$13.2%)
($k=32$
Data Set
CIFAR-10CIFAR-100ImageNet
Dropout
DenseNet-40 6.88 28.31 –
DenseNet-100 BC 6.02 26.18 –
DenseNet-100 5.11 23.00 –
DenseNet 250 BC 5.18 22.44 –
DenseNet 121 BC – – 27.26/9.01
($k=32$
SDR-Decay
DenseNet-40 6.53 27.39 –
DenseNet-100 BC 5.24 ($-$13.0%) 23.45 –
DenseNet-100 4.87 22.10 ($-$3.9%) –
DenseNet 250 BC – 19.79 ($-$11.8%) –
SDR-Dynamic
DenseNet-40 6.10 ($-$11.3%) 25.63 ($-$9.5%) –
DenseNet-100 BC 5.56 23.19 ($-$11.4%) –
DenseNet-100 4.82 ($-$5.7%) 22.37 –
DenseNet 250 BC 4.68 (−9.7%) 21.45 –
DenseNet 121 BC – – 25.58 ($-$6.2%)/7.82 ($-$13.2%)
($k=32$

Notes: ImageNet results include both Top-1 and Top-5 error. Values in parentheses indicate percentage decrease in error compared to dropout. Dashes indicate incompatible model or data set pairs. Values in bold show SDR error improvements relative to known best benchmarks.

Close Modal