Simple RNN

tflearn.layers.recurrent.simple_rnn (incoming, n_units, activation='sigmoid', dropout=None, bias=True, weights_init=None, return_seq=False, return_state=False, initial_state=None, dynamic=False, trainable=True, restore=True, reuse=False, scope=None, name='SimpleRNN')

Simple Recurrent Layer.

Input

3-D Tensor [samples, timesteps, input dim].

Output

if return_seq: 3-D Tensor [samples, timesteps, output dim]. else: 2-D Tensor [samples, output dim].

Arguments

  • incoming: Tensor. Incoming 3-D Tensor.
  • n_units: int, number of units for this layer.
  • activation: str (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: 'sigmoid'.
  • dropout: tuple of float: (input_keep_prob, output_keep_prob). The input and output keep probability.
  • bias: bool. If True, a bias is used.
  • weights_init: str (name) or Tensor. Weights initialization. (See tflearn.initializations)
  • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
  • return_state: bool. If True, returns a tuple with output and states: (output, states).
  • initial_state: Tensor. An initial state for the RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
  • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
  • trainable: bool. If True, weights will be trainable.
  • restore: bool. If True, this layer weights will be restored when loading a model.
  • reuse: bool. If True and 'scope' is provided, this layer variables will be reused (shared).
  • scope: str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name: str. A name for this layer (optional).

LSTM

tflearn.layers.recurrent.lstm (incoming, n_units, activation='tanh', inner_activation='sigmoid', dropout=None, bias=True, weights_init=None, forget_bias=1.0, return_seq=False, return_state=False, initial_state=None, dynamic=False, trainable=True, restore=True, reuse=False, scope=None, name='LSTM')

Long Short Term Memory Recurrent Layer.

Input

3-D Tensor [samples, timesteps, input dim].

Output

if return_seq: 3-D Tensor [samples, timesteps, output dim]. else: 2-D Tensor [samples, output dim].

Arguments

  • incoming: Tensor. Incoming 3-D Tensor.
  • n_units: int, number of units for this layer.
  • activation: str (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: 'tanh'.
  • inner_activation: str (name) or function (returning a Tensor). LSTM inner activation. Default: 'sigmoid'.
  • dropout: tuple of float: (input_keep_prob, output_keep_prob). The input and output keep probability.
  • bias: bool. If True, a bias is used.
  • weights_init: str (name) or Tensor. Weights initialization. (See tflearn.initializations).
  • forget_bias: float. Bias of the forget gate. Default: 1.0.
  • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
  • return_state: bool. If True, returns a tuple with output and states: (output, states).
  • initial_state: Tensor. An initial state for the RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
  • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
  • trainable: bool. If True, weights will be trainable.
  • restore: bool. If True, this layer weights will be restored when loading a model.
  • reuse: bool. If True and 'scope' is provided, this layer variables will be reused (shared).
  • scope: str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name: str. A name for this layer (optional).

References

Long Short Term Memory, Sepp Hochreiter & Jurgen Schmidhuber, Neural Computation 9(8): 1735-1780, 1997.

Links

http://deeplearning.cs.cmu.edu/pdfs/Hochreiter97_lstm.pdf

GRU

tflearn.layers.recurrent.gru (incoming, n_units, activation='tanh', inner_activation='sigmoid', dropout=None, bias=True, weights_init=None, return_seq=False, return_state=False, initial_state=None, dynamic=False, trainable=True, restore=True, reuse=False, scope=None, name='GRU')

Gated Recurrent Unit Layer.

Input

3-D Tensor Layer [samples, timesteps, input dim].

Output

if return_seq: 3-D Tensor [samples, timesteps, output dim]. else: 2-D Tensor [samples, output dim].

Arguments

  • incoming: Tensor. Incoming 3-D Tensor.
  • n_units: int, number of units for this layer.
  • activation: str (name) or function (returning a Tensor). Activation applied to this layer (see tflearn.activations). Default: 'tanh'.
  • inner_activation: str (name) or function (returning a Tensor). GRU inner activation. Default: 'sigmoid'.
  • dropout: tuple of float: (input_keep_prob, output_keep_prob). The input and output keep probability.
  • bias: bool. If True, a bias is used.
  • weights_init: str (name) or Tensor. Weights initialization. (See tflearn.initializations).
  • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
  • return_state: bool. If True, returns a tuple with output and states: (output, states).
  • initial_state: Tensor. An initial state for the RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
  • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
  • trainable: bool. If True, weights will be trainable.
  • restore: bool. If True, this layer weights will be restored when loading a model.
  • reuse: bool. If True and 'scope' is provided, this layer variables will be reused (shared).
  • scope: str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name: str. A name for this layer (optional).

References

Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation, K. Cho et al., 2014.

Links

http://arxiv.org/abs/1406.1078

Bidirectional RNN

tflearn.layers.recurrent.bidirectional_rnn (incoming, rnncell_fw, rnncell_bw, return_seq=False, return_states=False, initial_state_fw=None, initial_state_bw=None, dynamic=False, scope=None, name='BiRNN')

Build a bidirectional recurrent neural network, it requires 2 RNN Cells to process sequence in forward and backward order. Any RNN Cell can be used i.e. SimpleRNN, LSTM, GRU... with its own parameters. But the two cells number of units must match.

Input

3-D Tensor Layer [samples, timesteps, input dim].

Output

if return_seq: 3-D Tensor [samples, timesteps, output dim]. else: 2-D Tensor Layer [samples, output dim].

Arguments

  • incoming: Tensor. The incoming Tensor.
  • rnncell_fw: RNNCell. The RNN Cell to use for foward computation.
  • rnncell_bw: RNNCell. The RNN Cell to use for backward computation.
  • return_seq: bool. If True, returns the full sequence instead of last sequence output only.
  • return_states: bool. If True, returns a tuple with output and states: (output, states).
  • initial_state_fw: Tensor. An initial state for the forward RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
  • initial_state_bw: Tensor. An initial state for the backward RNN. This must be a tensor of appropriate type and shape [batch_size x cell.state_size].
  • dynamic: bool. If True, dynamic computation is performed. It will not compute RNN steps above the sequence length. Note that because TF requires to feed sequences of same length, 0 is used as a mask. So a sequence padded with 0 at the end must be provided. When computation is performed, it will stop when it meets a step with a value of 0.
  • scope: str. Define this layer scope (optional). A scope can be used to share variables between layers. Note that scope will override name.
  • name: str. A name for this layer (optional).