gluonnlp.initializer

This page describes initializers that are useful for multiple NLP model architectures.

Highway Bias Initializer

We now provide Highway bias initializer defined in the following work.

@inproceedings{srivastava2015training,
     title={Training very deep networks},
     author={Srivastava, Rupesh K and Greff, Klaus and Schmidhuber, J{\"u}rgen},
     booktitle={Advances in neural information processing systems},
     pages={2377--2385},
     year={2015}}

HighwayBias

Initialize all biases of an Highway layer by setting the biases of nonlinear transformer and the transform gate differently.

API Reference

NLP initializer.

class gluonnlp.initializer.HighwayBias(nonlinear_transform_bias=0.0, transform_gate_bias=-2.0, **kwargs)

Initialize all biases of an Highway layer by setting the biases of nonlinear transformer and the transform gate differently. The dimension of the biases are identical and equals to the \(arr.shape[0]/2\), where \(arr\) is the bias tensor.

The definition of the biases follows the work:

@inproceedings{srivastava2015training,
 title={Training very deep networks},
 author={Srivastava, Rupesh K and Greff, Klaus and Schmidhuber, J{\"u}rgen},
 booktitle={Advances in neural information processing systems},
 pages={2377--2385},
 year={2015}
}

Parameters

• nonlinear_transform_bias (float, default 0.0) – bias for the non linear transformer. We set the default according to the above original work.

• transform_gate_bias (float, default -2.0) – bias for the transform gate. We set the default according to the above original work.

class gluonnlp.initializer.TruncNorm(mean=0, stdev=0.01, **kwargs)

Initialize the weight by drawing sample from truncated normal distribution with provided mean and standard deviation. Values whose magnitude is more than 2 standard deviations from the mean are dropped and re-picked..

Parameters

• mean (float, default 0) – Mean of the underlying normal distribution

• stdev (float, default 0.01) – Standard deviation of the underlying normal distribution

• **kwargs (dict) – Additional parameters for base Initializer.