Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
SystemDS 3.1
-
None
Description
Add an attention layer to scripts/nn/layers.
Have a look at how the existing layers are added, for instance scripts/nn/layers/lstm.dml.
A quick reference to what an attention layer is: https://analyticsindiamag.com/a-beginners-guide-to-using-attention-layer-in-neural-networks/
The origin of the attention model: https://arxiv.org/pdf/1409.0473.pdf
Todo:
- Add attention.dml to scripts/nn/layers/ with the implementation of the layer.
- Add an AttentionExample.dml file to scripts/nn/examples where the attention layer is used with an lstm-layer to demonstrate that an attention model can be trained and used for predictions.