What is scaled dot-product attention?
An attention calculation that divides scores by the square root of the vector dimension to prevent gradients from exploding.
Verify This Answer
Cross-check this information using these trusted sources:
More FAQs in Understanding Attention Mechanism in Deep Learning
Global attention looks at all words in the input, while local attention only looks at a sliding window of neighboring words.
Because it relates positions of a single sequence to calculate a representation of the same sequence.
Yes, Vision Transformers (ViTs) apply self-attention to image patches instead of text tokens.
Still have questions?
Browse all our FAQs or reach out to our support team