# Can the T5 model classify codes such as codebert-small-v1? **URL:** \ **Category:** Beginners\ **Created:** [April 27, 2025, 10:03am UTC](https://discuss.huggingface.co/t/can-the-t5-model-classify-codes-such-as-codebert-small-v1/152496 "2025-04-27T10:03:32Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![kirilinko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kirilinko/32/46423_2.png) [@kirilinko](https://discuss.huggingface.co/u/kirilinko)\ **Post date:** [April 27, 2025, 10:03am UTC](https://discuss.huggingface.co/t/can-the-t5-model-classify-codes-such-as-codebert-small-v1/152496/1 "2025-04-27T10:03:33Z")
Hello. I’m doing code classification with codebert-small-v1, but as the maximum sequence is 512 tokens, this may limit me when faced with a certain amount of code (because of the size). On the other hand, I’ve noticed that T5 has a greater margin as regards the maximum sequence. Is it possible to use the T5 model for sort code classification to have the same output as codebert-small-v1? In the sense that I have the probability of appearance of each class of vulnerability in the code? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 27, 2025, 10:27am UTC](https://discuss.huggingface.co/t/can-the-t5-model-classify-codes-such-as-codebert-small-v1/152496/2 "2025-04-27T10:27:35Z")
I’m not familiar with it, but it seems possible. > **[Salesforce/codet5-base · Hugging Face](https://huggingface.co/Salesforce/codet5-base)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. > **[VulCatch: Enhancing Binary Vulnerability Detection through CodeT5...](https://arxiv.org/abs/2408.07181)** > > Binary program vulnerability detection is critical for software security, yet existing deep learning approaches often rely on source code analysis, limiting their ability to detect unknown vulnerabilities. To address this, we propose VulCatch, a... > **[huggingface/CodeBERTa-small-v1 · Hugging Face](https://huggingface.co/huggingface/CodeBERTa-small-v1)** > > We’re on a journey to advance and democratize artificial intelligence through open source and open science. ---
**Author:** ![kirilinko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kirilinko/32/46423_2.png) [@kirilinko](https://discuss.huggingface.co/u/kirilinko)\ **Post date:** [April 28, 2025, 9:12am UTC](https://discuss.huggingface.co/t/can-the-t5-model-classify-codes-such-as-codebert-small-v1/152496/3 "2025-04-28T09:12:37Z")
But I’m a bit surprised, when I try to classify with “TFAutoModelForSequenceClassification”, I get an error telling me that model T5 is not compatible. However, with codeBert small, no problem. I want to try another model because, I lack performance in predictions. My current model manages to classify the code well according to the CWE around 8 classes, but not when the code is vulnerable (only two classes) Do you have any idea what to do? ---
**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\ **Post date:** [April 28, 2025, 12:50pm UTC](https://discuss.huggingface.co/t/can-the-t5-model-classify-codes-such-as-codebert-small-v1/152496/4 "2025-04-28T12:50:13Z")
Hmm… > > > \## Environment info > > > \- \`transformers\` version: 4.3.2 > \- Platform: Linux-4.18….0-193.el8.x86\_64-x86\_64-with-glibc2.10 > \- Python version: 3.8.3 > \- PyTorch version (GPU?): 1.5.1+cu101 (True) > \- Tensorflow version (GPU?): not installed (NA) > \- Using GPU in script?: yes > \- Using distributed or parallel set-up in script?: single gpu > > \### Who can help > > Perhaps @patrickvonplaten, @patil-suraj could help? > > \## Information > > Model I am using (Bert, XLNet ...): T5 > > The problem arises when using: > \* \[\] the official example scripts: (give details below) > \* \[x\] my own modified scripts: (give details below) > > The tasks I am working on is: > \* \[\] an official GLUE/SQUaD task: (give the name) > \* \[x\] my own task or dataset: (give details below) > > \## To reproduce > > I'm trying to run the T5 base model. It seems that I use the correct model path (i.e., t5-base) and it finds and downloads the model, but crashes when it tries to instantiate it. The problem seems to be around the configuration class not being found. This is what I get: > > \`\`\` > File "../../../models/tr-4.3.2/run\_puppets.py", line 279, in main > model = AutoModelForSequenceClassification.from\_pretrained( > File "/dccstor/redrug\_ier/envs/last-tr/lib/python3.8/site-packages/transformers/models/auto/modeling\_auto.py", line 1362, in from\_pretrained > raise ValueError( > ValueError: Unrecognized configuration class \ for this kind of AutoModel: AutoModelForSequenceClassification. > Model type should be one of ConvBertConfig, LEDConfig, DistilBertConfig, AlbertConfig, CamembertConfig, XLMRobertaConfig, MBartConfig, BartConfig, LongformerConfig, RobertaConfig, SqueezeBertConfig, LayoutLMConfig, BertConfig, XLNetConfig, MobileBertConfig, FlaubertConfig, XLMConfig, ElectraConfig, FunnelConfig, DebertaConfig, GPT2Config, OpenAIGPTConfig, ReformerConfig, CTRLConfig, TransfoXLConfig, MPNetConfig, TapasConfig. > \`\`\` > I dig a bit and I may have a hunch why this happens. The config file is there: https://github.com/huggingface/transformers/blob/master/src/transformers/models/t5/configuration\_t5.py#L32 > but it's not recorded here: https://github.com/huggingface/transformers/blob/master/src/transformers/models/auto/modeling\_auto.py#L514 > > So the check here fails: https://github.com/huggingface/transformers/blob/master/src/transformers/models/auto/modeling\_auto.py#L1389 > > And the ValueError is raised. > > I hope this is it. It looks like an easy fix :) Thanks! > > PS: I'm running the same scripts/files with other models without problems. This seems to be something specific to T5. > even though T5 can be used very well for text-classification it remains a text-to-text only model. So you can only load the model via > from transformers import AutoModelForConditionalGeneration > model = AutoModelForConditionalGeneration.from\_pretrained(“t5-small”) ---
**Author:** ![kirilinko](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/kirilinko/32/46423_2.png) [@kirilinko](https://discuss.huggingface.co/u/kirilinko)\ **Post date:** [April 30, 2025, 11:23am UTC](https://discuss.huggingface.co/t/can-the-t5-model-classify-codes-such-as-codebert-small-v1/152496/5 "2025-04-30T11:23:13Z")
thank you ! ---
**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\ **Post date:** [April 30, 2025, 11:24pm UTC](https://discuss.huggingface.co/t/can-the-t5-model-classify-codes-such-as-codebert-small-v1/152496/6 "2025-04-30T23:24:02Z")
This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.