Uploaded image for project: 'SystemDS'
  1. SystemDS
  2. SYSTEMDS-3303

NN Builtin: Attention Layer

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • SystemDS 3.1
    • SystemDS 3.1
    • None

    Description

      Add an attention layer to scripts/nn/layers.

      Have a look at how the existing layers are added, for instance scripts/nn/layers/lstm.dml.

      A quick reference to what an attention layer is: https://analyticsindiamag.com/a-beginners-guide-to-using-attention-layer-in-neural-networks/

      The origin of the attention model: https://arxiv.org/pdf/1409.0473.pdf

      Todo:

      1. Add attention.dml to scripts/nn/layers/ with the implementation of the layer.
      2. Add an AttentionExample.dml file to scripts/nn/examples where the attention layer is used with an lstm-layer to demonstrate that an attention model can be trained and used for predictions.

      Attachments

        Activity

          People

            Unassigned Unassigned
            sebwrede Sebastian Wrede
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: