# Nice infrence trick

**URL:** <https://discuss.huggingface.co/t/nice-infrence-trick/72278>\
**Category:** 🤗Transformers\
**Created:** [February 7, 2024, 2:43pm UTC](https://discuss.huggingface.co/t/nice-infrence-trick/72278 "2024-02-07T14:43:28Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![nevakrien](https://avatars.discourse-cdn.com/v4/letter/n/46a35a/32.png) [@nevakrien](https://discuss.huggingface.co/u/nevakrien)\
**Post date:** [February 7, 2024, 2:43pm UTC](https://discuss.huggingface.co/t/nice-infrence-trick/72278/1 "2024-02-07T14:43:28Z")

</div>

so I am working with nllb and what I noticed is that it tends to do this thing of repeating the same tokens over and over consecutively. now what people usually do here is pass no\_repeat\_ngrams but that checks for ngrams on the entire text and thats not the issue here. names for instance should probably repeate.

here is my solution:  
just check for the first few tokens if u see a repeat and act just on that

> class StopRepeats(LogitsProcessor):  
> #stop repeating values of ngram\_size or more inside the context  
> #for instance abcabc is repeating twice has an ngram\_size of 3 and fits in a context of 6  
> def **init** (self, count,ngram\_size,context):  
> self.count = count  
> self.ngram\_size=ngram\_size  
> self.context = context
> 
> ```
> def __call__ (self, input_ids, scores):#encoder_input_ids
> if input_ids.size(1) > self.context:
> input_ids = input_ids[:, -self.context:]
>     
> for step in range(self.ngram_size, self.context // 2+ 1):                
>         
> cuts=[input_ids[:,i:i+step] for i in range(len(input_ids[0])-step,0,-step)]
> cuts=cuts[:self.count-1]
> if(len(cuts)!=self.count-1):
> continue
> 
> matching = torch.ones(input_ids.shape[0], dtype=torch.bool,device=input_ids.device)
> for cut in cuts[1:]:
> matching&= (cut==cuts[0]).all(dim=1)
> 
> scores[matching,cuts[0][matching,-1]]=float("-inf")
>            
> 
> return scores
> 
> ```

I thought maybe I pull request this into hf so people can use it but idk if thats the type of thing u pull request