Post by Kasper N.

Wrote a preprint on sparse attention together with Amelie Dittmann :) it is a training-free method that clusters keys and queries in transformed spaces, to then prune token interactions. It targets feasibility under large token counts like in pathology imagery and genomics. Check it out at https://lnkd.in/gnHUhSGT