# BERT next sentence prediction: bert-base always returns false

**URL:** <https://discuss.huggingface.co/t/bert-next-sentence-prediction-bert-base-always-returns-false/37819>\
**Category:** Models\
**Created:** [April 26, 2023, 3:37pm UTC](https://discuss.huggingface.co/t/bert-next-sentence-prediction-bert-base-always-returns-false/37819 "2023-04-26T15:37:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![anentropic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anentropic/32/15354_2.png) [@anentropic](https://discuss.huggingface.co/u/anentropic)\
**Post date:** [April 26, 2023, 3:37pm UTC](https://discuss.huggingface.co/t/bert-next-sentence-prediction-bert-base-always-returns-false/37819/1 "2023-04-26T15:37:05Z")

</div>

I am experimenting with the various `BertFor*` model classes with pre-trained models

Using `bert-base-uncased` with `BertForNextSentencePrediction` I have the following code which seems to work nicely:

```python
from transformers import BertTokenizer, BertForNextSentencePrediction
import torch

# Load pre-trained tokenizer and model
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForNextSentencePrediction.from_pretrained('bert-base-uncased').eval()

# Tokenize input sentences
sentence_a = "The cat sat on the mat."
sentence_b = "It was a nice day outside."
encoded = tokenizer.encode_plus(sentence_a, sentence_b, return_tensors='pt')

# Get model predictions
with torch.no_grad():
    outputs = model(**encoded)
logits = outputs[0]
probs = torch.softmax(logits, dim=1)
is_next_sentence = torch.argmax(probs, dim=1).bool().item()

```

`is_next_sentence` is either `True` or `False`

The only problem is that I can’t seem to find any pair of sentences which return `True`.

(Does anybody have any examples? Is the code above wrong?)

I am guessing that maybe the base model is just not able to do this task and I need to use a fine-tuned variant?

Does anyone have a `bert-base` variant on HF Hub that is fine-tuned on NSP?

---

<div class="post-metadata">

**Author:** ![anentropic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anentropic/32/15354_2.png) [@anentropic](https://discuss.huggingface.co/u/anentropic)\
**Post date:** [April 26, 2023, 4:11pm UTC](https://discuss.huggingface.co/t/bert-next-sentence-prediction-bert-base-always-returns-false/37819/2 "2023-04-26T16:11:12Z")

</div>

Or… training my own would be a good exercise I guess

Can anyone recommend an HF dataset for NSP training?

---

<div class="post-metadata">

**Author:** ![anentropic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anentropic/32/15354_2.png) [@anentropic](https://discuss.huggingface.co/u/anentropic)\
**Post date:** [April 26, 2023, 4:15pm UTC](https://discuss.huggingface.co/t/bert-next-sentence-prediction-bert-base-always-returns-false/37819/3 "2023-04-26T16:15:53Z")

</div>

I am starting to think maybe no one actually has a use for NSP inference… is this just a pre-training technique?

---

<div class="post-metadata">

**Author:** ![anentropic](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/anentropic/32/15354_2.png) [@anentropic](https://discuss.huggingface.co/u/anentropic)\
**Post date:** [April 26, 2023, 4:44pm UTC](https://discuss.huggingface.co/t/bert-next-sentence-prediction-bert-base-always-returns-false/37819/4 "2023-04-26T16:44:41Z")

</div>

Turns out the code above is wrong somehow

With help from [happy-transformer/happy\_next\_sentence.py at master · EricFillion/happy-transformer · GitHub](https://github.com/EricFillion/happy-transformer/blob/master/happytransformer/happy_next_sentence.py) I found working code:

```python
encoded = tokenizer(sentence_a, sentence_b, return_tensors='pt')

with torch.no_grad():
    outputs = model(**encoded)

probabilities = torch.softmax(outputs[0], dim=1)
prediction = torch.argmax(probabilities)
# 0 = yes, 1 = no
is_next_sentence = prediction.item() == 0

```

The main difference I see in this code is using `tokenizer(...)` rather than `tokenizer.encode_plus`