My first ai model

Beginners
1

Hey, i am learning ai model training so i want you to suggest me usecase for which i should create my first ai model.

Requirements

  1. I have a100 with 80gb ram so something that fit in it easily
2

hey man, thats really cool you are getting into ML. but look, we have several types of models (like: CNNs, RNNs, transformers)

if you are a student like me, I reccomend getting started with transformers.
also one more thing. if you ask help from a chatbot, they will say you can train big models like 30B+ models, but please stick to 230M-4B LoRA training for FT, and under 250M for PT.

also there are two main types of attention: bi directional and casual. I recommend finetuning a 350M LFM model with r32 a64 LoRA (not QLoRA. you have enough compute). then you should ry more models like qwen3.5-800M and maybe the new gemma4 models as well!

then once you are comfortable with things like loss masking, you should do a small PT run with around 50B tokens. try training a 120M model (it has just enough parameters to not dissapoint you if u didnt plan ur data quite right, so its just neough. maybe bumping it to 150M will make a considerable change as well. although some papers i read said that 141M is efficient, so go from somewhere around 120-150M)

for the PT architecture, use an LLM for the math and code. try to learn the high level things like how you calculate loss for PT, context length and position encoding.

for the architecture, use something like qwen3-600M and scale it down (smollm2’s 135M model’s arch works as well!)

since you have an A100, you can train this really quickly so you can comfortably learn things yourself. something with this field is that no matter how good you are with the math or science, if you havnt don anything practical, knowing just math and science wont help you.

then you should slowly write your own code and math. I also like this books (I am in HS yr10, but if u are above HS yr8 ish) you should read this book (the ebook is free, but the physical on is a bit expensive). wait, i actually have two books if u like reading.

if you know some math skim this a bit: (book – second link)
but I reccomend you to read both these together!

and for the love of god, please dont get advice from LLMs (I learnt it the hard way) :sob:

BTW, you have a lot of compute, but dont waste it on big models, try to work on smaller models and then once you know what you are doing, scale up to bigger models.

here is the list of models I like FT:

  • qwen3.5-800m and 2b
  • gemma4 2b (the projectors kinda mess things up, so avoid this for now)
  • LFM2.5 models (tbh for reason these models take very little VRAM. I am using unsloth zoo. the only problem is the license, but you are using them to learn, so… dont really care about the license)
  • gemma3-270M

also here is a fun challeng before doing PT, try doing CPT (continuid pre training) on smolLM2 135M (to expand its context length) you got the VRAM for it so dont worry about it. Then do a chat FT to chat with it. and also you can try doing a reasoning FT as well, but try doing the normal user/assistant one, then you can later redo the FT on the CPTed model for the reasoning dataset (which has <think> [thought process] </think> XML tags, and the model can think like a human)

eitherways, I will give you some advice: be sensible on how you are using your compute!

3

Thanks, I’ll keep that in mind. I’ll work with 150m models first to understand all the basics and advance concepts only then I’ll move to bigger models. Help appreciated :+1: