What is the difference between global and local attention?
Global attention looks at all words in the input, while local attention only looks at a sliding window of neighboring words.
Verify This Answer
Cross-check this information using these trusted sources:
More FAQs in Understanding Attention Mechanism in Deep Learning
An attention calculation that divides scores by the square root of the vector dimension to prevent gradients from exploding.
Because it relates positions of a single sequence to calculate a representation of the same sequence.
Yes, Vision Transformers (ViTs) apply self-attention to image patches instead of text tokens.
Still have questions?
Browse all our FAQs or reach out to our support team