# SQuAD with BERT tokenizer: Mismatch between span and token boundaries

**URL:** <https://discuss.huggingface.co/t/squad-with-bert-tokenizer-mismatch-between-span-and-token-boundaries/11644>\
**Category:** Models\
**Created:** [November 12, 2021, 9:28pm UTC](https://discuss.huggingface.co/t/squad-with-bert-tokenizer-mismatch-between-span-and-token-boundaries/11644 "2021-11-12T21:28:23Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![mgreenbe](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/mgreenbe/32/2827_2.png) [@mgreenbe](https://discuss.huggingface.co/u/mgreenbe)\
**Post date:** [November 12, 2021, 9:28pm UTC](https://discuss.huggingface.co/t/squad-with-bert-tokenizer-mismatch-between-span-and-token-boundaries/11644/1 "2021-11-12T21:28:23Z")

</div>

I’m trying to train a BERT model on the SQuAD dataset. I’ve found several instances where the answer text doesn’t start or end at a token boundary. I’m using `bert-base-uncased`, but I think my question is valid for any tokenizer based on the word-part algorithm.

To find the indices `start_position` and `end_position` of the starting and ending tokens of the answer span, I use the offsets returned by the tokenizer. Here’s an example where everything works “as expected”:

```auto
question_text = "Which band recorded the album 'Thank You 4 Your Service'?"
context = "Thank You 4 Your Service is the sixth and final studio album by American hip hop group A Tribe Called Quest."
answer_text = "A Tribe Called Quest"
answer_start = context.index(answer_text)
answer_end = answer_start + len(answer_text)
x = tokenizer(context, return_offsets_mapping=True)
starts, ends = zip(*x.offset_mapping)
start_position = starts.index(answer_start)
end_position = ends.index(answer_end)
tokenizer.convert_ids_to_tokens(x.input_ids[start_position:end_position + 1])
# ['a', 'tribe', 'called', 'quest']

```

Things go awry, however, if we set

```auto
question_text = "How many studio albums did A Tribe Called Quest release?"
answer_text = "six"

```

The line

```auto
end_position = ends.index(answer_end)

```

raises an exception because the answer “six” is part of the word “sixth” which is, itself, a token (id 4369).

What’s the “right” thing to do here? By “right”, I mean how do people-who-know-what-they’re-doing resolve this. What did Devlin et al. do when fine-tuned BERT on SQuAD in the original BERT paper? I couldn’t find those details in the paper.
