# Transformers, am i only using a Encoder for Binary Classification?

**URL:** <https://discuss.huggingface.co/t/transformers-am-i-only-using-a-encoder-for-binary-classification/3045>\
**Category:** Beginners\
**Created:** [January 4, 2021, 11:07am UTC](https://discuss.huggingface.co/t/transformers-am-i-only-using-a-encoder-for-binary-classification/3045 "2021-01-04T11:07:04Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![unknownTransformer](https://avatars.discourse-cdn.com/v4/letter/u/9f8e36/32.png) [@unknownTransformer](https://discuss.huggingface.co/u/unknownTransformer)\
**Post date:** [January 4, 2021, 11:07am UTC](https://discuss.huggingface.co/t/transformers-am-i-only-using-a-encoder-for-binary-classification/3045/1 "2021-01-04T11:07:04Z")

</div>

Hi Guys, i’ve got a basic, begginers Questions. If im doing something like a Binary Classification (Sentiment Analysis) of Text and im using Transformers (like Bert for example). Am i using both Encoder, and the Decoder Part of the Transformer Network? If i understood the basics right im just using an Encoder for Sequences to Binary - and Sequences to Sequence would use both Decoder and Encoder?

Ty in advanced.

---

<div class="post-metadata">

**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\
**Post date:** [January 4, 2021, 3:34pm UTC](https://discuss.huggingface.co/t/transformers-am-i-only-using-a-encoder-for-binary-classification/3045/2 "2021-01-04T15:34:28Z")

</div>

Hi @unknownTransformer,

BERT uses only the Encoder.

See this page of the docs: [https://huggingface.co/transformers/model\_summary.html](https://huggingface.co/transformers/model_summary.html)

See also the Devlin paper: [https://arxiv.org/abs/1810.04805](https://arxiv.org/abs/1810.04805)

Also try Jay Alammar’s blogs, for example this one: [alammar.github.io/illustrated-bert/](http://alammar.github.io/illustrated-bert/)

When you do sentiment analysis, you are using the basic BERT to code your text as numbers, and then you tune a final layer to your task. The Huggingface models such as BertForSequenceClassification already include a “final” layer for you, which is randomly initialized. See this page: [https://huggingface.co/transformers/model\_doc/bert.html](https://huggingface.co/transformers/model_doc/bert.html)

You will probably want to Freeze most of the BERT layers while you fine-tune the last layer, at least initially. See this post and the reply by sgugger: [How to freeze some layers of BertModel](https://discuss.huggingface.co/t/how-to-freeze-some-layers-of-bertmodel/917)

Good luck with it all.