Lambda Layer
Lambda Layer
**Lambda layers** are a building block for modeling long-range dependencies in data. They consist of long-range interactions between a query and a structured set of context elements at a reduced memory cost. Lambda layers transform each available context into a linear function, termed a lambda, which is then directly applied to the corresponding query. Whereas self-attention defines a similarity kernel between the query and the context elements, a lambda layer instead summarizes contextual information into a fixed-size linear function (i.e. a matrix), thus bypassing the need for memory-intensive attention maps.
LambdaNetworks: Modeling Long-Range Interactions Without Attention
https://paperswithcode.com/paper/lambdanetworks-modeling-long-range-1
LambdaNetworks: Modeling Long-Range Interactions Without Attention