# Using UDOP for layout analysis

**URL:** <https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871>\
**Category:** 🤗Transformers\
**Created:** [March 11, 2024, 10:34am UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871 "2024-03-11T10:34:41Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![annakrogager](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@annakrogager](https://discuss.huggingface.co/u/annakrogager)\
**Post date:** [March 11, 2024, 10:34am UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/1 "2024-03-11T10:34:41Z")

</div>

Hello,

I am very interested in the newly added [UDOP model](https://huggingface.co/docs/transformers/main/en/model_doc/udop). I would like to use it for layout analysis and by reading the documentation + paper, I found that it should be already pre-trained for this task using a prompt like this one: “Layout Analysis. Title”. However, when I try to run inference with this prompt, using the [example inference noteboook](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/UDOP/Inference_with_UDOP%2C_a_generative_document_AI_model.ipynb), the model just seems to do document classification instead - no layout analysis.

So, I thought it should probably be fine-tuned instead and I attempted to fine-tune it on the DocLayNet dataset. The layout analysis output is supposed to consist of so-called “layout tokens” (from the paper), which is a string representation of the layout bboxes after discretising them to a certain layout vocab size. In the paper, they mention the following as an example of a layout token: `<50><100><250><300>`

However, the UDOP processor doesn’t seem to recognize this text format - it tokenizes this into `['<unk>', '50', '>', '<unk>', '100', '>', '<unk>', '250', '>', '<unk>', '300', '>']`, so it doesn’t even have a token for `<`. I think the “layout tokens” should be part of the tokenizer vocabulary for this task to make sense but there are no tokens in the vocabulary that look like layout tokens.

I found a linked issue in the microsoft codebase ([here](https://github.com/microsoft/i-Code/issues/35)) but no answer. In this issue, there is a linked notebook where there does appear to be layout tokens in the tokenizer vocabulary (`<loc_0>, ..., <loc_499>`) . However, these tokens are not there in the Hugging Face implementation.

Any suggestions how I can proceed from here? How can I use layout tokens in the Hugging Face implementation and consequently use the model for layout analysis? Any help is greatly appreciated 🙂

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [March 11, 2024, 6:21pm UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/2 "2024-03-11T18:21:59Z")

</div>

[Transformers-Tutorials/UDOP/Layout\_analysis\_with\_UDOP.ipynb at master · NielsRogge/Transformers-Tutorials · GitHub](https://github.com/NielsRogge/Transformers-Tutorials/blob/master/UDOP/Layout_analysis_with_UDOP.ipynb) 😉

---

<div class="post-metadata">

**Author:** ![annakrogager](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@annakrogager](https://discuss.huggingface.co/u/annakrogager)\
**Post date:** [March 12, 2024, 8:15am UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/3 "2024-03-12T08:15:11Z")

</div>

This is great, thanks for the quick response Niels 😃

So there were 2 things wrong in my code actually:

1. The model “microsoft/udop-large” doesn’t have any layout tokens in the vocabulary, but “microsoft/udop-large-512” does.
2. I need to set `skip_special_tokens=False` when generating the output.

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [March 12, 2024, 8:24am UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/4 "2024-03-12T08:24:38Z")

</div>

Hi,

Thanks for investigating! Indeed, still need to push the special tokens to the vocabulary of microsoft/udop-large, can do soon.

I have a PR open for it: [[UDOP] Add special tokens to tokenizer by NielsRogge · Pull Request #29594 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/pull/29594).

And indeed, one needs to skip the special tokens when decoding as otherwise those are not included in the final output.

I also saw not that great performance of the layout analysis, this may be due to the fact that I’m using the Tesseract OCR engine, whereas the authors probably used Azure’s Read API.

---

<div class="post-metadata">

**Author:** ![annakrogager](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@annakrogager](https://discuss.huggingface.co/u/annakrogager)\
**Post date:** [March 12, 2024, 8:34am UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/5 "2024-03-12T08:34:57Z")

</div>

I noticed that your visualisation of the output bboxes seems to be a bit off. I used the following code instead to unnormalize the output bboxes:

```auto
LAYOUT_VOCAB_SIZE = 500

def unnormalize_box(box, image_width, image_height):
    x1 = box[0] / LAYOUT_VOCAB_SIZE * image_width
    y1 = box[1] / LAYOUT_VOCAB_SIZE * image_height
    x2 = box[2] / LAYOUT_VOCAB_SIZE * image_width
    y2 = box[3] / LAYOUT_VOCAB_SIZE * image_height
    return [x1, y1, x2, y2]

```

and then the layout analysis results look better 🙂

But you’re probably right that a better OCR engine would improve the results 👍

Thanks again for looking into this!

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [March 12, 2024, 9:12am UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/6 "2024-03-12T09:12:00Z")

</div>

Oh thanks, still had to look into the paper regarding what these layout tokens represent. Will update my notebook!

Also, I’ve pushed the special tokens for `microsoft/udop-large`, so it should work: [Upload processor · microsoft/udop-large at 95cde2f](https://huggingface.co/microsoft/udop-large/commit/95cde2f07406fdb942f5277f4c40f365ed8a9d84).

---

<div class="post-metadata">

**Author:** ![annakrogager](https://avatars.discourse-cdn.com/v4/letter/a/ac8455/32.png) [@annakrogager](https://discuss.huggingface.co/u/annakrogager)\
**Post date:** [March 12, 2024, 9:30am UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/7 "2024-03-12T09:30:15Z")

</div>

Another thing I just learned: using the prompt “Layout analysis on PubLayNet. Text” seems to work better than just “Layout analysis. Text”

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [March 12, 2024, 9:31pm UTC](https://discuss.huggingface.co/t/using-udop-for-layout-analysis/76871/8 "2024-03-12T21:31:12Z")

</div>

Closed on March 12, 2024, 9:31pm UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.