Sparse Query Attention (SQA): A Computationally Efficient Attention Mechanism with Query Heads Reduction
Paper
•
2510.01817
•
Published
•
13
Sparse Query Attention (SQA) Research by Reactive AI
Experimental models with Sparse Query Attention layers. Reducing training time/cost by ~3-10% compared to GQA & MQA, with the same level performance
-
-
ReactiveAI/sSQAT-mm
Text Generation • 8.62M • Updated -
ReactiveAI/SQAT-mm
Text Generation • 8.57M • Updated -
ReactiveAI/xSQAT-mm
Text Generation • 8.52M • Updated -
ReactiveAI/GQA-Ref-Micro
Text Generation • 8.67M • Updated -
ReactiveAI/MQA-Ref-Micro
Text Generation • 8.64M • Updated -
ReactiveAI/SQAT-m
Text Generation • 10.7M • Updated -
ReactiveAI/xSQAT-m
Text Generation • 10.4M • Updated -
ReactiveAI/sSQAT-m
Text Generation • 10.9M • Updated -
ReactiveAI/xSMQAT-m
Text Generation • 10.2M • Updated