Attentions Under the Microscope: A Comparative Study of Resource Utilization for Variants of Self-Attention
Research benchmarks eight attention mechanisms' energy usage and hardware resource demands, addressing high memory and time complexity in LLMs and VLMs.