Inference is the process of providing input to a trained large language model and receiving generated output, such as text, images, code, or other files.
Based on ChatML (Chat Markup Language) template format
ChatML was developed by OpenAI
Types of Roles
system
user
assistant
tool
Chat template is jinja2 based
Original Message
[ {"role": "system", "content": "You are a helpful assistant focused on technical topics."}, {"role": "user", "content": "Can you explain what a chat template is?"}, {"role": "assistant", "content": "A chat template structures conversations between users and AI models..."}, {"role": "user", "content": "How do I use it ?"},]
Chat Template for SmolLM2
{% for message in messages %}{% if loop.first and messages[0]['role'] != 'system' %}<|im_start|>systemYou are a helpful AI assistant named SmolLM, trained by Hugging Face<|im_end|>{% endif %}<|im_start|>{{ message['role'] }}{{ message['content'] }}<|im_end|>{% endfor %}
Final Result
<|im_start|>systemYou are a helpful assistant focused on technical topics.<|im_end|><|im_start|>userCan you explain what a chat template is?<|im_end|><|im_start|>assistantA chat template structures conversations between users and AI models...<|im_end|><|im_start|>userHow do I use it ?<|im_end|>
Example of Llama3.2 model
User: How are you?
AI: Hello, How is your day going so far?
-------- Converted using special tokens -------
<|begin_of_text|>
<|start_header_id|>user<|end_header_id|>
How are you?
<|eot_id|>
<|start_header_id|>assistant<|end_header_id|>
Hello, How is your day going so far?
<|eot_id|>
Inference Process
---
title: LLM Inference Process
---
flowchart TD
A["Input Text"] --> B["Tokenizer"]
B --> C["Input Token IDs"]
subgraph LLM["Raw LLM (Learned Weights)"]
direction TD
D["Embedding Layer"]
E["Transformer Blocks"]
F["Final Normalization"]
G["Output Projection"]
H["Vocabulary Logits"]
D --> E --> F --> G --> H
end
C --> D
H --> I["Decoding<br/>(Greedy, Temperature, Top-p, etc.)<br/>Select Next Token ID"]
I --> J{"Stop token or<br/>length limit?"}
J -- No --> C
J -- Yes --> K["Detokenizer"]
K --> L["Generated Text"]
classDef input fill:#e8f2ff,stroke:#3973ac,color:#142b45
classDef model fill:#eaf7ed,stroke:#36834b,color:#173d22
classDef inference fill:#fff1db,stroke:#bd7c21,color:#54340b
class A,B,C,K,L input
class D,E,F,G,H model
class I,J inference