Interface  Description 

Regularization 
Regularization API for magnitudebased regularization techniques such as:
L1Regularization L2Regularization WeightDecay Implementations should have the following features: 1. 
Class  Description 

L1Regularization 
L1 regularization: Implements updating as follows:
L = loss + l1 * sum_i abs(w[i]) {@code w[i] = updater(gradient[i] + l1 * sign(w[i]))  where sign(w[i]) is +/ 1 Note that L1 regularization is applied before the updater (Adam/Nesterov/etc) is applied. 
L2Regularization 
L2 regularization: very similar to
WeightDecay , but is applied before the updater is applied, not after. 
WeightDecay 
WeightDecay regularization: Updater is not applied to the regularization term gradients, and (optionally) applies the learning rate.

Enum  Description 

Regularization.ApplyStep 
ApplyStep determines how the regularization interacts with the optimization process  i.e., when it is applied
relative to updaters like Adam, Nesterov momentum, SGD, etc.

Copyright © 2020. All rights reserved.