What are attention masks in transformer models and how do they function?

Attention masks are binary or float vectors that indicate which tokens in the input sequence should be attended to and which should not. In most transformer implementations, they are used to prevent the model from attending to padding tokens or future tokens in the case of decoder models. For example, in sequence-to-sequence tasks, attention masks ensure that the model processes only the relevant parts of the input, thereby enhancing performance and efficiency.