Large Language Model

• It is a type of model that excels at understanding and generating human language
• trained on vast amount of text data
• consists of million of parameters
• mostly based on Transformers
• Typically decoder-based transformers with billions of params
• Examples
  • DeepSeek-R1 by DeepSeek
  • GPT4 by OpenAI
  • Llama3 by Meta
  • SmolLM2 by Hugging Face
  • Gemma by Google
  • Mistral by Mistral
• Ref: https://huggingface.co/learn/agents-course/unit1/what-are-llms

Limitations of LLM

• Hallucinations: They can generate incorrect information confidently
• Lack of true understanding: operate purely on statistical patterns
• Bias: They may reproduce biases present in their training data or inputs.
• Context windows: They have limited context windows
• Computational resources: They require significant computational resources

LLM capabilities

• Base model is trained on raw text data
  • It can be fine tuned and allowed capabilities like instruction following and tool calling
• Instruct: To follow instructions and engage in conversations
• Tool calling: To perform function calls
• Multilingual
• Vision
• Audio
• Reasoning
• Multimodal: Process Text, Image, Audio simultaneously
• Mixture of Experts (MoE)
• Embedding
• Deep Research
• Structured Output

NLP

• NLP is a field of linguistics and machine learning focused on understanding everything related to human language
• The aim of NLP tasks is not only to understand single words individually, but to be able to understand the context of those words
• LLMs are modern technique within NLP
• AI -> ML -> NLP -> LLM
• NLP common tasks
  • Classify sentence (sentiment, spam or not etc.)
  • Classify part of speech of a word in sentence
  • Speech recognition (speech-to-text)
  • Text-to-speech
• https://huggingface.co/learn/llm-course/chapter1/2

Prompt

• The input sequence you provide an LLM is called a prompt
• When we pass information to LLMs, we structure our input in a way that guides the generation of the LLM toward the desired output. This is called prompting
• See Prompt_Engineering

Training LLM

• They are trained on large datasets of text
• It is unsupervised learning and model learns structure and pattern of language
• After pre-training, they are fine-tuned for specific tasks
• See __Training

Model Compression

• https://en.wikipedia.org/wiki/Model_compression
• Process of reducing the size of models itself
• Techniques
  • Pruning
  • Low Rank Factorization
  • Quantization

Pruning

• Pruning sparsifies a large model by setting some parameters to exactly zero
• This effectively reduces the number of parameters

Low Rank Factorization

• Weight matrices can be approximated by low-rank matrices
• Single Value Decomposition (SVD) can be utilized

Quantization

• When AI Model is created its weights are stored as massive 16-bit floating-point (FP16) numbers
• Quantization reduces the precision to 8-bit (FP8) or 4-bit (FP4) which speeds up generation times with minimum loss in reasoning capabilities
• It introduces quantization error causing a drop in model accuracy, reasoning, or response quality
• Perplexity measures how quantization degrades model accuracy
• This reduces the size of the model
• DP64 (Double Precision 64-bit floating-point) are rarely used by Models because of their memory footprint

	Use case
DP64	Used in Physics Simulations
FP16	Used in LLM Training and high precision inference
FP8	Enterprise Deployment for LLM inference
FP4	Ultra-low quantization for local consumer grade device

GGUF Quantization formats

• https://medium.com/@paul.ilvez/demystifying-llm-quantization-suffixes-what-q4-k-m-q8-0-and-q6-k-really-mean-0ec2770f17d3
• Q{bits}_{quantization type}_{precision size}
• bits
  • number of bits to save each weight
• quantization type
  • K: grouped
  • 0: ungrouped or global (legacy)
  • 1: (legacy)
• precision size
  • S: Small
  • M: Medium
  • L: Large

Model	Perplexity (ppl)	Notes	Recommended
Q4_0	+0.2499	small, high quality loss
Q4_K_M (aka Q4_K)	+0.0535	medium, balanced quality	Yes
Q5_K_S	+0.0353	large, low quality loss	Yes
Q5_K_M	+0.0142	large, very low quality loss	Yes
Q8_0	+0.0004	very large, extremely low quality loss