[Question]: when searching the best sparse attention type ,why to caculate the score just pick the 2500 cols?

[unicorneeee]

Describe the issue

I hope this messages find u well!
def stream_llm(vertical_size, slash_size):
q_len = q.shape[2]

    mask = torch.triu(torch.tril(torch.ones(q_len, q_len), 0), -slash_size).to(q)
    mask[:,:vertical_size] = 1
    mask = mask.unsqueeze(0).unsqueeze(1)

    est_attn = torch.tril(mask)
    attn_weights_x = attn_weights * est_attn
    res3 = attn_weights_x[:,:,2500:].sum(-1).mean(-1).squeeze().float().detach().cpu().numpy()
    return res3

why this only pick the 2500: columns? If the head is A-shape, this may lead to mis-classification to V-S type?

[iofu728]

Hi @unicorneeee, thanks for your interest in MInference.

The value 2500 here is used to exclude the dense upper triangular part of the lower triangle, which doesn’t affect any results.

[unicorneeee]

Thanks for your answering,! I wander why pick the 2500? Because I draw the attn_weights on the top 30000 tokens, and I notice that there isn't show the dense part in the top 2500, the attn_weights distribute very discrete. Is there any experiment to show pick the 2500 is the best number?