Joint CTC Conformer LSTM Model

Joint CTC Conformer LSTM Model

class openspeech.models.joint_ctc_conformer_lstm.model.JointCTCConformerLSTMModel(configs: omegaconf.dictconfig.DictConfig, vocab: openspeech.vocabs.vocab.Vocabulary)[source]

Conformer encoder + LSTM decoder.

Parameters
  • configs (DictConfig) – configuraion set

  • vocab (Vocabulary) – vocab of training data

Inputs:
inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded

FloatTensor of size (batch, seq_length, dimension).

input_lengths (torch.LongTensor): The length of input tensor. (batch)

Returns

Result of model predictions.

Return type

  • y_hats (torch.FloatTensor)

forward(inputs: torch.Tensor, input_lengths: torch.Tensor) → Dict[str, torch.Tensor][source]

Forward propagate a inputs and targets pair for inference.

Inputs:
inputs (torch.FloatTensor): A input sequence passed to encoders. Typically for inputs this will be a padded

FloatTensor of size (batch, seq_length, dimension).

input_lengths (torch.LongTensor): The length of input tensor. (batch)

Returns

Result of model predictions that contains y_hats, logits,

encoder_outputs, encoder_logits, encoder_output_lengths.

Return type

  • dict (dict)

set_beam_decoder(batch_size: int, beam_size: int = 3)[source]

Setting beam search decoder

test_step(batch: tuple, batch_idx: int)collections.OrderedDict[source]

Forward propagate a inputs and targets pair for test.

Inputs:

batch (tuple): A train batch contains inputs, targets, input_lengths, target_lengths batch_idx (int): The index of batch

Returns

loss for training

Return type

loss (torch.Tensor)

training_step(batch: tuple, batch_idx: int)collections.OrderedDict[source]

Forward propagate a inputs and targets pair for training.

Inputs:

batch (tuple): A train batch contains inputs, targets, input_lengths, target_lengths batch_idx (int): The index of batch

Returns

loss for training

Return type

loss (torch.Tensor)

validation_step(batch: tuple, batch_idx: int)collections.OrderedDict[source]

Forward propagate a inputs and targets pair for validation.

Inputs:

batch (tuple): A train batch contains inputs, targets, input_lengths, target_lengths batch_idx (int): The index of batch

Returns

loss for training

Return type

loss (torch.Tensor)

Joint CTC Conformer LSTM Model Configuration

class openspeech.models.joint_ctc_conformer_lstm.configurations.JointCTCConformerLSTMConfigs(model_name: str = 'joint_ctc_conformer_lstm', encoder_dim: int = 512, num_encoder_layers: int = 17, num_attention_heads: int = 8, feed_forward_expansion_factor: int = 4, conv_expansion_factor: int = 2, input_dropout_p: float = 0.1, feed_forward_dropout_p: float = 0.1, attention_dropout_p: float = 0.1, conv_dropout_p: float = 0.1, conv_kernel_size: int = 31, half_step_residual: bool = True, num_decoder_layers: int = 2, decoder_dropout_p: float = 0.1, num_decoder_attention_heads: int = 1, max_length: int = 128, teacher_forcing_ratio: float = 1.0, rnn_type: str = 'lstm', decoder_attn_mechanism: str = 'loc', optimizer: str = 'adam')[source]

This is the configuration class to store the configuration of a JointCTCConformerLSTM.

It is used to initiated an JointCTCConformerLSTM model.

Configuration objects inherit from :class: ~openspeech.dataclass.configs.OpenspeechDataclass.

Configurations:

model_name (str): Model name (default: joint_ctc_conformer_lstm) encoder_dim (int): Dimension of encoder. (default: 512) num_encoder_layers (int): The number of encoder layers. (default: 17) num_attention_heads (int): The number of attention heads. (default: 8) feed_forward_expansion_factor (int): The expansion factor of feed forward module. (default: 4) conv_expansion_factor (int): The expansion factor of convolution module. (default: 2) input_dropout_p (float): The dropout probability of inputs. (default: 0.1) feed_forward_dropout_p (float): The dropout probability of feed forward module. (default: 0.1) attention_dropout_p (float): The dropout probability of attention module. (default: 0.1) conv_dropout_p (float): The dropout probability of convolution module. (default: 0.1) conv_kernel_size (int): The kernel size of convolution. (default: eq) half_step_residual (bool): Flag indication whether to use half step residual or not (default: True) num_decoder_layers (int): The number of decoder layers. (default: 2) decoder_dropout_p (float): The dropout probability of decoder. (default: 0.1) max_length (int): Max decoding length. (default: 128) teacher_forcing_ratio (float): The ratio of teacher forcing. (default: 1.0) rnn_type (str): Type of rnn cell (rnn, lstm, gru) (default: lstm) decoder_attn_mechanism (str): The attention mechanism for decoder. (default: loc) optimizer (str): Optimizer for training. (default: adam)