GGUF Inferencer in R

Published

June 2026

Introduction

This one’s been a long time in the making. It’s a bit of a personal endeavor— AI can be scary, but being able to “make one” yourself really demystifies it! I’d say the technical content here is old news, but I haven’t actually seen any inferencer-from-scratch writeups recently.1

Anyway, this article is about a little R package2 I wrote to parse GGUF files and run an interactive chat using a specific LLM. It all runs natively in base R3— pretty slowly, of course, but this makes it easy to track, visualize, and understand how the math works. All code was made by hand <3. (As well as the writeup, as always.)

The package inferences off of SmolLM3— a tiny, pretty decent LLM from 2025 that uses relatively simple, modern architecture and is small enough to run at a passable speed in R. It was created by the team behind Huggingface— the platform for sharing locally-run LLMs— and has a nice writeup about how it was created. Sidenote: if you weren’t aware, local models have improved significantly in the past year (2025-2026). They’re comparable to last year’s online models in chat quality, and the most powerful can even code effectively! Many will easily run at a decent speed on consumer computers (e.g. my PC with a GPU from 2016), and some can even run well on a mobile phone. SmolLM3 isn’t great at facts or coding (it’s the second-smallest model I could find!), but it certainly passes the Turing test.

This writeup is somewhat casual— there’s plenty of formal resources out there on these topics. The first five sections go over how inferencing works (and how I do it in the package), Section 7 explains how to use the package to run inference yourself in R, and the last section talks about the interesting process of how I learned all this.

If you want to try the package out for yourself, skip to Section 7 and start the setup; it takes 30 minutes or so for everything to load, which will give you plenty of time to read this article :)

This article contains quite a few foldable infoboxes.

1. Big Picture

In order to perform inference with an LLM, you need a prompt, the model’s weights, and information about the model’s architecture— the math you’re supposed to use for that particular model. Locally-run models these days (2026) come in a single .gguf file (Section 2), which contains the weights and some metadata telling you which architecture to use.

Generative inference works in three key steps. First, a word*4 is tokenized (Section 3)— turned into an ID corresponding to a word in the model’s vocabulary, which is stored in the GGUF— and then embedded— turned into a large vector corresponding to that word. Each word’s vector is different; and but they all have the same length— 2048, in the case of SmolLM3 and many other models.

Then, this vector is passed through a bunch of math— the transformer (a specialized neural network; Section 5)— based on the model architecture. This is usually a bunch of simple matrix operations, involving the model weights and a set of saved vectors from all previous words in the prompt and predictions this session. These operations are applied over and over in series with different weights— SmolLM3 applies them 36 times per prediction.

Finally, the resulting vector after all this math is compared with the original set of embeddings (the vectors for every word in the vocabulary) to see which words’ vectors it most closely resembles. This produces a set of probabilities, which are used to pick which word to predict (Section 4). Then, the picked word gets fed right back into the first step, tokenized and passed through the transformer! This repeats ad nauseum.

Before text generation starts, the model runs this process with the entire prompt (chat template + user message) as one big matrix instead of a vector— but the math is the same.

2. GGUF Files

2.1 Structure

GGUFs are LLMs’ all-in-one filetype for running inference, designed to replace the mess of files we had to use a few years ago. A GGUF contains info on how a the LLM should be run— important metadata, rules for tokenizing, and a chat template the chatbot was trained to use — as well as the model weights themselves (compressed to reduce filesize).

GGUFs are stored as raw bytes, which means you’ll have to manually write a script to interpret them as the numbers and letters they intend to represent. Most coding languages have tools for reading user-defined raw filestructures, but R is unfortunately lacking in this regard; I had to manually write a number of functions to interpret the raw bytes correctly.

GGUFs follow a relatively simple structure: first, a short, fixed-length set of header variables; then, some number of metadata entries; next, information on the size and location of the model weights; and finally, after a small amount of padding to make the index a multiple of 32, all the weights, one after another. The header defines how many metadata and tensor entries there are.

GGUFs include a few non-standard raw data types in addition to the usual uint32 et al.. gguf_string is used for all text, and simply consists of a uint32 (32-bit unsigned integer, a binary number from 0 to 4,294,967,295) indicating the length of the text string, and then that many bytes of raw UTF-8 characters. So “hello” shows up as 05 00 00 00 68 65 6c 6c 6f; the number 5 (taking up four bytes as a uint32) followed by 5 characters corresponding to each letter in “hello”. GGUFs also use a custom datatype for arrays, and store tensors in a special number format discussed in the next section.

Here’s a blueprint I made5 to show the exact byte format of a GGUF, with examples from the SmolLM3 GGUF we’ll be using:

#Header
(47 47 55 46) Magic Number ("GGUF" in UTF-8)
(uint32) GGUF Version
(uint64) # of Tensors (n_tensors)
(uint64) # of Metadata Entries (n_metadata)

#Metadata: Repeat for # of Metadata Entries
(gguf_metadata x n_metadata):
    (gguf_string) Name
    (uint32) Data Type
    [type] Data
    
    #Example Metadata Entry:
    (gguf_string):
        (uint32) String Length = 18
        (18-byte UTF-8) String = smollm3.vocab_size
    (uint32) Data Type = 4 (int32)
    (int32) Value = 128256
    
    #Template Array Entry
    (gguf_string) Name
    (uint32) Data Type = 9 (array)
    (gguf_array):
        (uint32) Data Type
        (uint64) # of Array Entries (n_entries)
        [type x n_entries] Data
        
    #Example Array Entry
    (gguf_string) Name = general.languages
    (uint32) Data Type = 9 (array)
    (gguf_array):
        (uint32) Data Type = 8 (string)
        (uint64) # of Array Entries = 8
        (gguf_string x 8):
            (gguf_string) String = "en"
            (gguf_string) String = "fr"
            (gguf_string) String = "es"
            (gguf_string) String = "it"
            (gguf_string) String = "pt"
            (gguf_string) String = "zh"
            (gguf_string) String = "ar"
            (gguf_string) String = "ru"

#Tensor Info: Repeat for # of Tensors
(gguf_tensor x n_tensors): 
    (gguf_string) Name
    (uint32) Dimensions (1 or 2)
    (uint64) # of Columns
    [if Dimensions > 1]:
      (uint64) # of Rows
    (uint32) Tensor Data Type (different from metadata)
    (uint64) Offset in bytes from Blob
    
    #Example Tensor Entry
    (gguf_string) Name = token_embd.weight
    (uint32) Dimensions = 2
    (uint64) Columns = 2048
    (uint64) Rows = 128256
    (uint32) Tensor Data Type = 8 (Q8_0)
    (uint64) Offset = 8192
    
[padding until position is a multiple of 32]
#Blob is here: start of tensor values

#Example F32 Tensor: output_norm.weight (1 row, 2048 columns)
(float32 x 2048) Values

#Example Q8_0 Tensor: blk.0.ffn_gate.weight (11008 rows, 2048 columns)
(Q8_0 x 2048/32 x 11008): #64 Q8_0 chunks per row, for 11008 rows 
#2048 values per row @ 32 values per chunk = 2048/32 = 64 chunks per row
    (Q8_0 x 64): #row 1
      (Q8_0):
        (float16) Chunk Multiplier
        (int8 x 32) Chunk Values
      (Q8_0)
      (Q8_0)
      ... #64 times
    (Q8_0 x 64) #row 2
    (Q8_0 x 64) #row 3
    ... #11008 times

And here’s the parsed header, metadata, and tensor data from the SmolLM3 GGUF. This is a neatened view of what you’ll get by running view(model), since read_gguf stores all this info in a single R object. The Metadata section is particularly interesting to look through to get a feel for what’s included in a GGUF.

value
magic GGUF
gguf_version 3
tensor_count 326
metadata_count 38
type value
general.architecture string smollm3
general.type string model
general.name string Smollm3-3B
general.basename string Smollm3-3B
general.quantized_by string Unsloth
general.size_label string 3B
general.license string apache-2.0
general.repo_url string https://huggingface.co/unsloth
general.base_model.count int32 1
general.base_model.0.name string SmolLM3 3B
general.base_model.0.organization string HuggingFaceTB
general.base_model.0.repo_url string https://huggingface.co/HuggingFaceTB/SmolLM3-3B
general.tags array (1) unsloth
general.languages array (8) en, fr, es, it, pt, zh, ar, ru
smollm3.block_count int32 36
smollm3.context_length int32 65536
smollm3.embedding_length int32 2048
smollm3.feed_forward_length int32 11008
smollm3.attention.head_count int32 16
smollm3.attention.head_count_kv int32 4
smollm3.rope.freq_base float32 5e+06
smollm3.attention.layer_norm_rms_epsilon float32 9.99999997475243e-07
smollm3.vocab_size int32 128256
smollm3.rope.dimension_count int32 128
tokenizer.ggml.model string gpt2
tokenizer.ggml.pre string smaug-bpe
tokenizer.ggml.tokens array (128256) !, “, #, $, %, &, ’, (, ), *, +, ,, -, ., /, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9
tokenizer.ggml.token_type array (128256) 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1
tokenizer.ggml.merges array (280147) Ġ Ġ, Ġ ĠĠĠ, ĠĠ ĠĠ, ĠĠĠ Ġ, i n, Ġ t, Ġ ĠĠĠĠĠĠĠ, ĠĠ ĠĠĠĠĠĠ, ĠĠĠĠ ĠĠĠĠ, ĠĠĠ ĠĠĠĠĠ, ĠĠĠĠĠĠĠ Ġ, ĠĠĠĠĠ ĠĠĠ, ĠĠĠĠĠĠ ĠĠ, e r, Ġ ĠĠ, ĠĠ Ġ, o n, Ġ a, r e, a t, s t, e n, o r, Ġ th, Ġt h
tokenizer.ggml.eos_token_id int32 128012
tokenizer.ggml.padding_token_id int32 128004
tokenizer.chat_template string {#- Copyright 2025-present the Unsloth team…
general.quantization_version int32 2
general.file_type int32 7
quantize.imatrix.file string SmolLM3-3B-GGUF/imatrix_unsloth.dat
quantize.imatrix.dataset string unsloth_calibration_SmolLM3-3B.txt
quantize.imatrix.entries_count int32 252
quantize.imatrix.chunks_count int32 505
cols rows type offset
output_norm.weight 2048 1 F32 0
token_embd.weight 2048 128256 Q8_0 8192
blk.0.attn_k.weight 2048 512 Q8_0 279093248
blk.0.attn_norm.weight 2048 1 F32 280207360
blk.0.attn_output.weight 2048 2048 Q8_0 280215552
blk.0.attn_q.weight 2048 2048 Q8_0 284672000
blk.0.attn_v.weight 2048 512 Q8_0 289128448
blk.0.ffn_down.weight 11008 2048 Q8_0 290242560
blk.0.ffn_gate.weight 2048 11008 Q8_0 314195968
blk.0.ffn_norm.weight 2048 1 F32 338149376
blk.0.ffn_up.weight 2048 11008 Q8_0 338157568
blk.1.attn_k.weight 2048 512 Q8_0 362110976
blk.1.attn_norm.weight 2048 1 F32 363225088
blk.1.attn_output.weight 2048 2048 Q8_0 363233280
blk.1.attn_q.weight 2048 2048 Q8_0 367689728
blk.1.attn_v.weight 2048 512 Q8_0 372146176
blk.1.ffn_down.weight 11008 2048 Q8_0 373260288
blk.1.ffn_gate.weight 2048 11008 Q8_0 397213696
blk.1.ffn_norm.weight 2048 1 F32 421167104
blk.1.ffn_up.weight 2048 11008 Q8_0 421175296
blk.2.attn_k.weight 2048 512 Q8_0 445128704
blk.2.attn_norm.weight 2048 1 F32 446242816
blk.2.attn_output.weight 2048 2048 Q8_0 446251008
blk.2.attn_q.weight 2048 2048 Q8_0 450707456
blk.2.attn_v.weight 2048 512 Q8_0 455163904
blk.2.ffn_down.weight 11008 2048 Q8_0 456278016
blk.2.ffn_gate.weight 2048 11008 Q8_0 480231424
blk.2.ffn_norm.weight 2048 1 F32 504184832
blk.2.ffn_up.weight 2048 11008 Q8_0 504193024
blk.3.attn_k.weight 2048 512 Q8_0 528146432
blk.3.attn_norm.weight 2048 1 F32 529260544
blk.3.attn_output.weight 2048 2048 Q8_0 529268736
blk.3.attn_q.weight 2048 2048 Q8_0 533725184
blk.3.attn_v.weight 2048 512 Q8_0 538181632
blk.3.ffn_down.weight 11008 2048 Q8_0 539295744
blk.3.ffn_gate.weight 2048 11008 Q8_0 563249152
blk.3.ffn_norm.weight 2048 1 F32 587202560
blk.3.ffn_up.weight 2048 11008 Q8_0 587210752
blk.4.attn_k.weight 2048 512 Q8_0 611164160
blk.4.attn_norm.weight 2048 1 F32 612278272
blk.4.attn_output.weight 2048 2048 Q8_0 612286464
blk.4.attn_q.weight 2048 2048 Q8_0 616742912
blk.4.attn_v.weight 2048 512 Q8_0 621199360
blk.4.ffn_down.weight 11008 2048 Q8_0 622313472
blk.4.ffn_gate.weight 2048 11008 Q8_0 646266880
blk.4.ffn_norm.weight 2048 1 F32 670220288
blk.4.ffn_up.weight 2048 11008 Q8_0 670228480
blk.5.attn_k.weight 2048 512 Q8_0 694181888
blk.5.attn_norm.weight 2048 1 F32 695296000
blk.5.attn_output.weight 2048 2048 Q8_0 695304192
blk.5.attn_q.weight 2048 2048 Q8_0 699760640
blk.5.attn_v.weight 2048 512 Q8_0 704217088
blk.5.ffn_down.weight 11008 2048 Q8_0 705331200
blk.5.ffn_gate.weight 2048 11008 Q8_0 729284608
blk.5.ffn_norm.weight 2048 1 F32 753238016
blk.5.ffn_up.weight 2048 11008 Q8_0 753246208
blk.6.attn_k.weight 2048 512 Q8_0 777199616
blk.6.attn_norm.weight 2048 1 F32 778313728
blk.6.attn_output.weight 2048 2048 Q8_0 778321920
blk.6.attn_q.weight 2048 2048 Q8_0 782778368
blk.6.attn_v.weight 2048 512 Q8_0 787234816
blk.6.ffn_down.weight 11008 2048 Q8_0 788348928
blk.6.ffn_gate.weight 2048 11008 Q8_0 812302336
blk.6.ffn_norm.weight 2048 1 F32 836255744
blk.6.ffn_up.weight 2048 11008 Q8_0 836263936
blk.7.attn_k.weight 2048 512 Q8_0 860217344
blk.7.attn_norm.weight 2048 1 F32 861331456
blk.7.attn_output.weight 2048 2048 Q8_0 861339648
blk.7.attn_q.weight 2048 2048 Q8_0 865796096
blk.7.attn_v.weight 2048 512 Q8_0 870252544
blk.7.ffn_down.weight 11008 2048 Q8_0 871366656
blk.7.ffn_gate.weight 2048 11008 Q8_0 895320064
blk.7.ffn_norm.weight 2048 1 F32 919273472
blk.7.ffn_up.weight 2048 11008 Q8_0 919281664
blk.8.attn_k.weight 2048 512 Q8_0 943235072
blk.8.attn_norm.weight 2048 1 F32 944349184
blk.8.attn_output.weight 2048 2048 Q8_0 944357376
blk.8.attn_q.weight 2048 2048 Q8_0 948813824
blk.8.attn_v.weight 2048 512 Q8_0 953270272
blk.8.ffn_down.weight 11008 2048 Q8_0 954384384
blk.8.ffn_gate.weight 2048 11008 Q8_0 978337792
blk.8.ffn_norm.weight 2048 1 F32 1002291200
blk.8.ffn_up.weight 2048 11008 Q8_0 1002299392
blk.9.attn_k.weight 2048 512 Q8_0 1026252800
blk.9.attn_norm.weight 2048 1 F32 1027366912
blk.9.attn_output.weight 2048 2048 Q8_0 1027375104
blk.9.attn_q.weight 2048 2048 Q8_0 1031831552
blk.9.attn_v.weight 2048 512 Q8_0 1036288000
blk.9.ffn_down.weight 11008 2048 Q8_0 1037402112
blk.9.ffn_gate.weight 2048 11008 Q8_0 1061355520
blk.9.ffn_norm.weight 2048 1 F32 1085308928
blk.9.ffn_up.weight 2048 11008 Q8_0 1085317120
blk.10.attn_k.weight 2048 512 Q8_0 1109270528
blk.10.attn_norm.weight 2048 1 F32 1110384640
blk.10.attn_output.weight 2048 2048 Q8_0 1110392832
blk.10.attn_q.weight 2048 2048 Q8_0 1114849280
blk.10.attn_v.weight 2048 512 Q8_0 1119305728
blk.10.ffn_down.weight 11008 2048 Q8_0 1120419840
blk.10.ffn_gate.weight 2048 11008 Q8_0 1144373248
blk.10.ffn_norm.weight 2048 1 F32 1168326656
blk.10.ffn_up.weight 2048 11008 Q8_0 1168334848
blk.11.attn_k.weight 2048 512 Q8_0 1192288256
blk.11.attn_norm.weight 2048 1 F32 1193402368
blk.11.attn_output.weight 2048 2048 Q8_0 1193410560
blk.11.attn_q.weight 2048 2048 Q8_0 1197867008
blk.11.attn_v.weight 2048 512 Q8_0 1202323456
blk.11.ffn_down.weight 11008 2048 Q8_0 1203437568
blk.11.ffn_gate.weight 2048 11008 Q8_0 1227390976
blk.11.ffn_norm.weight 2048 1 F32 1251344384
blk.11.ffn_up.weight 2048 11008 Q8_0 1251352576
blk.12.attn_k.weight 2048 512 Q8_0 1275305984
blk.12.attn_norm.weight 2048 1 F32 1276420096
blk.12.attn_output.weight 2048 2048 Q8_0 1276428288
blk.12.attn_q.weight 2048 2048 Q8_0 1280884736
blk.12.attn_v.weight 2048 512 Q8_0 1285341184
blk.12.ffn_down.weight 11008 2048 Q8_0 1286455296
blk.12.ffn_gate.weight 2048 11008 Q8_0 1310408704
blk.12.ffn_norm.weight 2048 1 F32 1334362112
blk.12.ffn_up.weight 2048 11008 Q8_0 1334370304
blk.13.attn_k.weight 2048 512 Q8_0 1358323712
blk.13.attn_norm.weight 2048 1 F32 1359437824
blk.13.attn_output.weight 2048 2048 Q8_0 1359446016
blk.13.attn_q.weight 2048 2048 Q8_0 1363902464
blk.13.attn_v.weight 2048 512 Q8_0 1368358912
blk.13.ffn_down.weight 11008 2048 Q8_0 1369473024
blk.13.ffn_gate.weight 2048 11008 Q8_0 1393426432
blk.13.ffn_norm.weight 2048 1 F32 1417379840
blk.13.ffn_up.weight 2048 11008 Q8_0 1417388032
blk.14.attn_k.weight 2048 512 Q8_0 1441341440
blk.14.attn_norm.weight 2048 1 F32 1442455552
blk.14.attn_output.weight 2048 2048 Q8_0 1442463744
blk.14.attn_q.weight 2048 2048 Q8_0 1446920192
blk.14.attn_v.weight 2048 512 Q8_0 1451376640
blk.14.ffn_down.weight 11008 2048 Q8_0 1452490752
blk.14.ffn_gate.weight 2048 11008 Q8_0 1476444160
blk.14.ffn_norm.weight 2048 1 F32 1500397568
blk.14.ffn_up.weight 2048 11008 Q8_0 1500405760
blk.15.attn_k.weight 2048 512 Q8_0 1524359168
blk.15.attn_norm.weight 2048 1 F32 1525473280
blk.15.attn_output.weight 2048 2048 Q8_0 1525481472
blk.15.attn_q.weight 2048 2048 Q8_0 1529937920
blk.15.attn_v.weight 2048 512 Q8_0 1534394368
blk.15.ffn_down.weight 11008 2048 Q8_0 1535508480
blk.15.ffn_gate.weight 2048 11008 Q8_0 1559461888
blk.15.ffn_norm.weight 2048 1 F32 1583415296
blk.15.ffn_up.weight 2048 11008 Q8_0 1583423488
blk.16.attn_k.weight 2048 512 Q8_0 1607376896
blk.16.attn_norm.weight 2048 1 F32 1608491008
blk.16.attn_output.weight 2048 2048 Q8_0 1608499200
blk.16.attn_q.weight 2048 2048 Q8_0 1612955648
blk.16.attn_v.weight 2048 512 Q8_0 1617412096
blk.16.ffn_down.weight 11008 2048 Q8_0 1618526208
blk.16.ffn_gate.weight 2048 11008 Q8_0 1642479616
blk.16.ffn_norm.weight 2048 1 F32 1666433024
blk.16.ffn_up.weight 2048 11008 Q8_0 1666441216
blk.17.attn_k.weight 2048 512 Q8_0 1690394624
blk.17.attn_norm.weight 2048 1 F32 1691508736
blk.17.attn_output.weight 2048 2048 Q8_0 1691516928
blk.17.attn_q.weight 2048 2048 Q8_0 1695973376
blk.17.attn_v.weight 2048 512 Q8_0 1700429824
blk.17.ffn_down.weight 11008 2048 Q8_0 1701543936
blk.17.ffn_gate.weight 2048 11008 Q8_0 1725497344
blk.17.ffn_norm.weight 2048 1 F32 1749450752
blk.17.ffn_up.weight 2048 11008 Q8_0 1749458944
blk.18.attn_k.weight 2048 512 Q8_0 1773412352
blk.18.attn_norm.weight 2048 1 F32 1774526464
blk.18.attn_output.weight 2048 2048 Q8_0 1774534656
blk.18.attn_q.weight 2048 2048 Q8_0 1778991104
blk.18.attn_v.weight 2048 512 Q8_0 1783447552
blk.18.ffn_down.weight 11008 2048 Q8_0 1784561664
blk.18.ffn_gate.weight 2048 11008 Q8_0 1808515072
blk.18.ffn_norm.weight 2048 1 F32 1832468480
blk.18.ffn_up.weight 2048 11008 Q8_0 1832476672
blk.19.attn_k.weight 2048 512 Q8_0 1856430080
blk.19.attn_norm.weight 2048 1 F32 1857544192
blk.19.attn_output.weight 2048 2048 Q8_0 1857552384
blk.19.attn_q.weight 2048 2048 Q8_0 1862008832
blk.19.attn_v.weight 2048 512 Q8_0 1866465280
blk.19.ffn_down.weight 11008 2048 Q8_0 1867579392
blk.19.ffn_gate.weight 2048 11008 Q8_0 1891532800
blk.19.ffn_norm.weight 2048 1 F32 1915486208
blk.19.ffn_up.weight 2048 11008 Q8_0 1915494400
blk.20.attn_k.weight 2048 512 Q8_0 1939447808
blk.20.attn_norm.weight 2048 1 F32 1940561920
blk.20.attn_output.weight 2048 2048 Q8_0 1940570112
blk.20.attn_q.weight 2048 2048 Q8_0 1945026560
blk.20.attn_v.weight 2048 512 Q8_0 1949483008
blk.20.ffn_down.weight 11008 2048 Q8_0 1950597120
blk.20.ffn_gate.weight 2048 11008 Q8_0 1974550528
blk.20.ffn_norm.weight 2048 1 F32 1998503936
blk.20.ffn_up.weight 2048 11008 Q8_0 1998512128
blk.21.attn_k.weight 2048 512 Q8_0 2022465536
blk.21.attn_norm.weight 2048 1 F32 2023579648
blk.21.attn_output.weight 2048 2048 Q8_0 2023587840
blk.21.attn_q.weight 2048 2048 Q8_0 2028044288
blk.21.attn_v.weight 2048 512 Q8_0 2032500736
blk.21.ffn_down.weight 11008 2048 Q8_0 2033614848
blk.21.ffn_gate.weight 2048 11008 Q8_0 2057568256
blk.21.ffn_norm.weight 2048 1 F32 2081521664
blk.21.ffn_up.weight 2048 11008 Q8_0 2081529856
blk.22.attn_k.weight 2048 512 Q8_0 2105483264
blk.22.attn_norm.weight 2048 1 F32 2106597376
blk.22.attn_output.weight 2048 2048 Q8_0 2106605568
blk.22.attn_q.weight 2048 2048 Q8_0 2111062016
blk.22.attn_v.weight 2048 512 Q8_0 2115518464
blk.22.ffn_down.weight 11008 2048 Q8_0 2116632576
blk.22.ffn_gate.weight 2048 11008 Q8_0 2140585984
blk.22.ffn_norm.weight 2048 1 F32 2164539392
blk.22.ffn_up.weight 2048 11008 Q8_0 2164547584
blk.23.attn_k.weight 2048 512 Q8_0 2188500992
blk.23.attn_norm.weight 2048 1 F32 2189615104
blk.23.attn_output.weight 2048 2048 Q8_0 2189623296
blk.23.attn_q.weight 2048 2048 Q8_0 2194079744
blk.23.attn_v.weight 2048 512 Q8_0 2198536192
blk.23.ffn_down.weight 11008 2048 Q8_0 2199650304
blk.23.ffn_gate.weight 2048 11008 Q8_0 2223603712
blk.23.ffn_norm.weight 2048 1 F32 2247557120
blk.23.ffn_up.weight 2048 11008 Q8_0 2247565312
blk.24.attn_k.weight 2048 512 Q8_0 2271518720
blk.24.attn_norm.weight 2048 1 F32 2272632832
blk.24.attn_output.weight 2048 2048 Q8_0 2272641024
blk.24.attn_q.weight 2048 2048 Q8_0 2277097472
blk.24.attn_v.weight 2048 512 Q8_0 2281553920
blk.24.ffn_down.weight 11008 2048 Q8_0 2282668032
blk.24.ffn_gate.weight 2048 11008 Q8_0 2306621440
blk.24.ffn_norm.weight 2048 1 F32 2330574848
blk.24.ffn_up.weight 2048 11008 Q8_0 2330583040
blk.25.attn_k.weight 2048 512 Q8_0 2354536448
blk.25.attn_norm.weight 2048 1 F32 2355650560
blk.25.attn_output.weight 2048 2048 Q8_0 2355658752
blk.25.attn_q.weight 2048 2048 Q8_0 2360115200
blk.25.attn_v.weight 2048 512 Q8_0 2364571648
blk.25.ffn_down.weight 11008 2048 Q8_0 2365685760
blk.25.ffn_gate.weight 2048 11008 Q8_0 2389639168
blk.25.ffn_norm.weight 2048 1 F32 2413592576
blk.25.ffn_up.weight 2048 11008 Q8_0 2413600768
blk.26.attn_k.weight 2048 512 Q8_0 2437554176
blk.26.attn_norm.weight 2048 1 F32 2438668288
blk.26.attn_output.weight 2048 2048 Q8_0 2438676480
blk.26.attn_q.weight 2048 2048 Q8_0 2443132928
blk.26.attn_v.weight 2048 512 Q8_0 2447589376
blk.26.ffn_down.weight 11008 2048 Q8_0 2448703488
blk.26.ffn_gate.weight 2048 11008 Q8_0 2472656896
blk.26.ffn_norm.weight 2048 1 F32 2496610304
blk.26.ffn_up.weight 2048 11008 Q8_0 2496618496
blk.27.attn_k.weight 2048 512 Q8_0 2520571904
blk.27.attn_norm.weight 2048 1 F32 2521686016
blk.27.attn_output.weight 2048 2048 Q8_0 2521694208
blk.27.attn_q.weight 2048 2048 Q8_0 2526150656
blk.27.attn_v.weight 2048 512 Q8_0 2530607104
blk.27.ffn_down.weight 11008 2048 Q8_0 2531721216
blk.27.ffn_gate.weight 2048 11008 Q8_0 2555674624
blk.27.ffn_norm.weight 2048 1 F32 2579628032
blk.27.ffn_up.weight 2048 11008 Q8_0 2579636224
blk.28.attn_k.weight 2048 512 Q8_0 2603589632
blk.28.attn_norm.weight 2048 1 F32 2604703744
blk.28.attn_output.weight 2048 2048 Q8_0 2604711936
blk.28.attn_q.weight 2048 2048 Q8_0 2609168384
blk.28.attn_v.weight 2048 512 Q8_0 2613624832
blk.28.ffn_down.weight 11008 2048 Q8_0 2614738944
blk.28.ffn_gate.weight 2048 11008 Q8_0 2638692352
blk.28.ffn_norm.weight 2048 1 F32 2662645760
blk.28.ffn_up.weight 2048 11008 Q8_0 2662653952
blk.29.attn_k.weight 2048 512 Q8_0 2686607360
blk.29.attn_norm.weight 2048 1 F32 2687721472
blk.29.attn_output.weight 2048 2048 Q8_0 2687729664
blk.29.attn_q.weight 2048 2048 Q8_0 2692186112
blk.29.attn_v.weight 2048 512 Q8_0 2696642560
blk.29.ffn_down.weight 11008 2048 Q8_0 2697756672
blk.29.ffn_gate.weight 2048 11008 Q8_0 2721710080
blk.29.ffn_norm.weight 2048 1 F32 2745663488
blk.29.ffn_up.weight 2048 11008 Q8_0 2745671680
blk.30.attn_k.weight 2048 512 Q8_0 2769625088
blk.30.attn_norm.weight 2048 1 F32 2770739200
blk.30.attn_output.weight 2048 2048 Q8_0 2770747392
blk.30.attn_q.weight 2048 2048 Q8_0 2775203840
blk.30.attn_v.weight 2048 512 Q8_0 2779660288
blk.30.ffn_down.weight 11008 2048 Q8_0 2780774400
blk.30.ffn_gate.weight 2048 11008 Q8_0 2804727808
blk.30.ffn_norm.weight 2048 1 F32 2828681216
blk.30.ffn_up.weight 2048 11008 Q8_0 2828689408
blk.31.attn_k.weight 2048 512 Q8_0 2852642816
blk.31.attn_norm.weight 2048 1 F32 2853756928
blk.31.attn_output.weight 2048 2048 Q8_0 2853765120
blk.31.attn_q.weight 2048 2048 Q8_0 2858221568
blk.31.attn_v.weight 2048 512 Q8_0 2862678016
blk.31.ffn_down.weight 11008 2048 Q8_0 2863792128
blk.31.ffn_gate.weight 2048 11008 Q8_0 2887745536
blk.31.ffn_norm.weight 2048 1 F32 2911698944
blk.31.ffn_up.weight 2048 11008 Q8_0 2911707136
blk.32.attn_k.weight 2048 512 Q8_0 2935660544
blk.32.attn_norm.weight 2048 1 F32 2936774656
blk.32.attn_output.weight 2048 2048 Q8_0 2936782848
blk.32.attn_q.weight 2048 2048 Q8_0 2941239296
blk.32.attn_v.weight 2048 512 Q8_0 2945695744
blk.32.ffn_down.weight 11008 2048 Q8_0 2946809856
blk.32.ffn_gate.weight 2048 11008 Q8_0 2970763264
blk.32.ffn_norm.weight 2048 1 F32 2994716672
blk.32.ffn_up.weight 2048 11008 Q8_0 2994724864
blk.33.attn_k.weight 2048 512 Q8_0 3018678272
blk.33.attn_norm.weight 2048 1 F32 3019792384
blk.33.attn_output.weight 2048 2048 Q8_0 3019800576
blk.33.attn_q.weight 2048 2048 Q8_0 3024257024
blk.33.attn_v.weight 2048 512 Q8_0 3028713472
blk.33.ffn_down.weight 11008 2048 Q8_0 3029827584
blk.33.ffn_gate.weight 2048 11008 Q8_0 3053780992
blk.33.ffn_norm.weight 2048 1 F32 3077734400
blk.33.ffn_up.weight 2048 11008 Q8_0 3077742592
blk.34.attn_k.weight 2048 512 Q8_0 3101696000
blk.34.attn_norm.weight 2048 1 F32 3102810112
blk.34.attn_output.weight 2048 2048 Q8_0 3102818304
blk.34.attn_q.weight 2048 2048 Q8_0 3107274752
blk.34.attn_v.weight 2048 512 Q8_0 3111731200
blk.34.ffn_down.weight 11008 2048 Q8_0 3112845312
blk.34.ffn_gate.weight 2048 11008 Q8_0 3136798720
blk.34.ffn_norm.weight 2048 1 F32 3160752128
blk.34.ffn_up.weight 2048 11008 Q8_0 3160760320
blk.35.attn_k.weight 2048 512 Q8_0 3184713728
blk.35.attn_norm.weight 2048 1 F32 3185827840
blk.35.attn_output.weight 2048 2048 Q8_0 3185836032
blk.35.attn_q.weight 2048 2048 Q8_0 3190292480
blk.35.attn_v.weight 2048 512 Q8_0 3194748928
blk.35.ffn_down.weight 11008 2048 Q8_0 3195863040
blk.35.ffn_gate.weight 2048 11008 Q8_0 3219816448
blk.35.ffn_norm.weight 2048 1 F32 3243769856
blk.35.ffn_up.weight 2048 11008 Q8_0 3243778048

2.2 Tensors

“Tensors” are a term thrown around everywhere in AI, which frustrates me because… a tensor is just an n-dimensional array, and LLM inference only involves vectors and matrices! Sure, you could call vectors and matrices 1D and 2D tensors, but it just makes the whole thing feel less familiar and approachable. Higher-dimension tensors are used in training LLMs, but even there they’re not anything special.

LLMs using a simple transformer architecture (i.e. the structure of the inference math) will have some amount of layers (SmolLM3 has 36), where each layer contains the same set of tensors used in the math; to run inference, we’ll pass a matrix through the layers one-by-one and have the exact same math applied to it 36 times in a row— but the weights will be different on each layer.

Anyway, a GGUF contains a compressed version of its LLM’s weights, which compose the aforementioned tensors (matrices and vectors) needed to run inference. Models are trained using floats of a specific precision (often 32-bit), but it turns out that if you approximate these 32-bit “pure” weights with reasonably close numbers, the LLM still performs quite well. This is the idea behind quantization— if you come up with a format for approximating those weights that makes the math faster or (most importantly) lets you store all the weights using less memory, you can run the model significantly faster with minimal intelligence loss.

There’s something like 30 common ways of quantizing model weights, and they aren’t too special— just different ways of lossily compressing the matrix weights to varying degrees. However, in our case, decompressing complex quants just adds unnecessary overhead, because we’re just going to do the math with R’s native 64-bit floats (unlike some clever programs, which use custom instructions to run the math directly on the compressed weights). So I opted to use Q8_0 quantification, which is one of the simplest formats to unpack.

The blueprint in the previous section outlines the structure of Q8_0 weights. As you may have seen in the Tensor Info block, Q8_0 GGUFs only store their tensors as Q8_0 or 32-bit floats (F32), making them quite convenient to work with. In comparison, something like Q4_K_M often uses 4 or 5 different types of quantization for its weights! (That lets you give impactful tensors a higher precision, while highly compressing tensors that don’t have as signficant an impact on the math.)

But the bottom line is that we want to unpack these weights into regular numeric data to use in R. But… it’s a lot of data! Uncompressed and cast into R’s numeric type, the weights take up a whopping 25 gigabytes of memory. In order to efficiently load this data when we do the math, I opted to save each layer of tensors as its own R object, in an .rds file in tempdir(). These files will be discussed further in section 4. Inferencing, where they get used.

2.3 read_gguf.R

Everything I discussed in this section is performed by one function: read_gguf(). Here it is, if you’re curious: (all code is also in the ggufr repo)

3. Tokenizing

3.1 Purpose

In order to predict text based on a prompt, we need to turn the prompt into a set of vectors that we can perform the math on. The first step in that is to tokenize it— turn it into a bunch of IDs corresponding to the vectors to use.

Ideally, we’d want every word to be a token, but because we want this model to work for multiple language and not crash when it receives a string like “fovriegfivnvoihuev”, we need another approach— allowing word fragments and letters to be their own token. Modern models have a fixed vocabulary of tokens, which includes which includes every common unicode character along with a bunch of words. This means that, even if a word isn’t in the vocabulary, it can still be broken up into smaller tokens. SmolLM3 has a vocabulary of 128k tokens— quite a lot!

So how do we break things up? First, we use a regex pattern to split the prompt into individual words and symbols, keeping spaces at the start of each word: “How”, ” are”, ” you”, “?”. Then, each word is further split into characters, and we attempt to reassemble those characters into the largest token we can— often a single token for the whole word. Applying this merging process to each word and symbol separated by the regex leaves us with a list of tokens that we can then use to run inference.

3.2 Merging Details

Modern LLMs have simplified the merging process significantly— in addition to the vocabulary, the GGUF contains a list of merges, and characters are simply combined based on if they appear in the merges. For example, the merges include “o w” and “w ow”— so in the tokenizing process, “wow” will be iteratively merged from individual characters into the whole word: (w,o,w) -> (w,ow) -> (wow). Importantly, this process is deterministic— so input text gets tokenized and interpreted identically whether the model is being trained or generating. My package implements merging in the BPE() function, which does this with the raw bytes— no need to convert them to characters.

There’s one extra bit of nuance. You may have seen in the GGUF metadata that there’s a lot of the Ġ character floating around— this is from an annoying convention from earlier language processing research that maps all invisible characters to visible, less-common latin ones, in order to make debugging easier. This isn’t just visual— the actual model vocab uses the bytes for Ġ instead of space. This is especially relevant since most words in the vocab start with a space (e.g. “Ġforgotten”)— this helps differentiate between words and suffixes, and if you think about it, almost every word in a piece of writing is preceded by a space.

Anyway, tokenizer.R contains BPE() and an implementation of this invisible character mapping based on the original by Andrej Karpathy. I recommend reading that instead of the R code, since it’s more fully commented.

3.3 The Chat Template

GGUFs come with a recommended chat template to turn the LLM into the sort of chatbot we expect. By default, the LLM is just a word-predictor— if you feed it part of a phrase, it’ll predict what it thinks the next words will be. It turns out that it’s quite easy to turn a word predictor into something more…

Modern chatbots achieve this by inserting the prompt into a template like this:

You are a helpful AI assistant.
User says: [prompt].
You say: [

Naturally, the next predictions are predictions of what a helpful AI assistant would say. This is (surprisingly) tantamount to actually being an AI assistant!

Left: An example of word prediction: the LLM makes up the details itself, as it’s just a word predictor.
infer("Yesterday I went to my friend's barbecue and", model, use_template=FALSE)
Right: The same prompt with the chat template; a reasonable exchange with an assistant.
infer("Yesterday I went to my friend's barbecue and...", model, use_template=TRUE, show_template=TRUE)

The GGUF chat template uses another weird, esoteric format (jinja2) to actually create this prompt template. In the package I went through SmolLM3’s template and manually trimmed it down into a short text template to make it easier to feed into the model; you can see this in the template() function at the bottom of tokenizer.R. Chat templates use some special tokens— located at the end of the model’s vocabulary— to help the LLM know the structure of the template; for example, the string <|im_start|> indicates the start of a new message.

4. Inferencing

Now that we have a list of token IDs to feed into the model, it’s time to run inference! This involves two steps: first, an initial prefill, and then the actual word generation.

The purpose of prefilling is to process the prompt tokens and predict the first new token. This actually uses the same math and functions as regular generation, but it does it for the whole prompt at once with big matrix multiplications. Prefill starts adding to the KV cache— the data the model uses to remember what tokens have come appeared so far, and at the end of prefill we cut off the last row of the matrix to use to predict the first word.

Then, regular text generation: we loop a process of embedding the previous token ID, feeding it into the transformer, and then using the output of that to predict the next token. This process is detailed in the next section.

The ggufr package does this all in inferencer.R. The transformer uses 36 layers applied one-by-one to the input; we saved the GGUF tensors for these layers as .rds files back in read_gguf(), and during the inference we manually load them, apply the math, and unload them again, one by one. This is a funny higher-level approximation of the RAM shuffling that efficient inferencers have to perform— moving the weights in and out of the GPU/CPU is the primary bottleneck in running an LLM in almost every situation, and it’s no different here. This is why having a lot of RAM is so desirable at the moment— if your RAM is big enough, you never have to move the weights around and can run the LLM as fast as the math allows.

Here’s inferencer.R: the function to focus on in infer().

5. Transforming

5.1 Big Idea

Attention!

This is the most important section in this article, in my opinion. It covers how a vector of 2048 numbers can represent a word, and how this lets an LLM “guess” the next word.

So how does the transformer work?

The purpose of the overall transformer is to produce an output— the predicted next token— based on several inputs— every word in the conversation so far and their absolute positions (e.g. what was the 5th word?). And with just those inputs, it’s very good at predicting the next word!

The way it does this sorta makes sense (if you look into the math), though it feels somewhat magical— why does this one work for LLMs when older machine learning techniques don’t? There’s a good answer to this: to quote Edison6, we’ve come up with 1,000 similar models that don’t work, and this one happened to be the one that succeeded. And it took 5 years7 for us to realize it!

We start by getting the embedding for the last token by taking a single row out of the Embedding Matrix corresponding to the token’s ID.

Tensor: Embedding Matrix

token_embd.weight
128256 rows x 2048 columns
range: (-9.5, 5.4)
mean: -0.006 ± 0.16

This matrix contains the initial embeddings for each token. Each row corresponds to a token in the model’s vocabulary.

This embedding is a vector that represents a “fuzzy idea” of what the token is supposed to be, with 2048 values usually between -1 and 1. The purpose of the transformer is to take the current token and add and subtract from it to get a fuzzy idea of the next token, so that we can then decide which token it should be.

Here’s an example of this “fuzzy idea” representation: below are the embeddings for the tokens “breakfast”, “lunch”, and “baseball”,8 visually represented as colored lines on a graph.

Code 
words = c(' breakfast',' lunch',' baseball')
E = readRDS(paste0(model$dir,'embeddings.rds'))$token_embd.weight
embeddings = embedding$token_embd.weight[tokenize(' breakfast lunch baseball',model),]

image(t(embeddings))
axis(side=2, at=c(1,0.5,0), labels = c("breakfast", "lunch", "baseball"),las=2,tick=F)

Looks noisy. Let’s zoom into the first 100 values of each, and draw positive values in red and negative in blue. Lighter values are closer to zero.

In theory, the embeddings for “breakfast” and “lunch” should be more similar than with “baseball”, but this isn’t apparent from plotting them. But if we take the dot product between them, the difference emerges: (values rounded for convenience)

This is actually how we decide which token to predict after transformation; the transformer spits out an unknown fuzzy representation vector for the next token, and we dot it with the embeddings for the entire model vocabulary. The tokens with the highest results from the dot product are the most closely related tokens to that fuzzy representation, so we pick one of them9, throw out the old vector, and replace it with the actual embedding for that token to feed back into the transformer.

5.2 Layer Components

Let’s look at an embedded token vector as it goes through a single transformer layer. This section will describe the math we perform; I’m not confident in describing why we do each of these steps, but I can certainly say what to do. Code here is paraphrased from transformer.R (see section 5.3), with some SmolLM3-specific values inserted.

# ---- Input Embedding ----
input = c(...)
# input: [1, 2048] <- dimension of vector

A transformer layer (for this model) takes in an embedding vector (1x2048) and adds10 two new 1x2048 vectors to it, which it calculates using the input and the tensors for that layer. One vector is for attention— which takes into account all the previous tokens— and one is for the FFN (feed-forward network), which is a simple, traditional neural network component based on only the current state.

First comes the attention section.

5.2.1 GQA (Grouped-Query Attention)

Immediately, we normalize the input and multiply it element-wise with a set of weights for this layer. This normalization includes an epsilon term to avoid dividing by zero, which is interestingly defined in the GGUF.

# ---- Normalize Input ----
RMSN = function(v, norm, epsilon) norm * v / sqrt(mean(v^2) + epsilon)
NormIn = RMSN(input, layer$attn_norm.weight, metadata$smollm3.attention.layer_norm_rms_epsilon)
# NormIn: [1, 2048]
# attn_norm.weight: [1, 2048]

Next, we matrix-multiply this with three sets of weights to create three vectors: Query, Key, and Value. I won’t go into exactly what these represent11, but they’re critical for letting the model “understand” the previous words and relationships in the chat.

# ---- Get New Query/Key/Value Vectors ----
Q = NormIn %*% t(layer$attn_q.weight)
K = NormIn %*% t(layer$attn_k.weight)
V = NormIn %*% t(layer$attn_v.weight)
# Q: [1, 2048]
# K, V: [1, 512]
# attn_q.weight: [2048, 2048]
# attn_k.weight, attn_v.weight: [512, 2048]

These vectors are now split up into multiple “heads”. I believe the primary purpose of this is to make it easier to parallelize the math, since the heads do not affect each other12.

Importantly, in this model, there are fewer KV heads than Q, and we’ll reuse each K and V head for 4 Q heads. This makes this a GQA transformer, and has the advantage of requiring significantly less KV storage than an MHA model (where no heads are reused), at presumably minimal intelligence cost.

# ---- Break QKV into Heads ----
q = vector('list',16)
k = vector('list',4)
v = vector('list',4)
for (i in 1:length(q)) {
  q[[i]] = Q[,128 * (i - 1) + 1:128,drop=FALSE]
}
for (i in 1:length(k)) {
  k[[i]] = K[,128 * (i - 1) + 1:128,drop=FALSE]
  v[[i]] = V[,128 * (i - 1) + 1:128,drop=FALSE]
}
# q: list of 16 heads, 16 x [1, 128]
# k, v: lists of 4 heads, 4 x [1, 128]
#note that while there are more Q heads than KV, all heads have the same dimensions.

Next, we’ll apply RoPE (rotary positional embedding) to the Q and K heads (but NOT V). This is the part that lets the model “understand” the position of words relative to each other. It’s really neat— in each head, half the values are treated as X coordinates, the other as Y, and the coordinates are rotated based on the absolute position of the current token; i.e., rotate 4 times if this is the fourth token so far in the chat. Each of the 64 coordinate pairs is rotated at a different speed— and because rotation is cyclic (e.g. a certain amount of rotations will return to the original value), RoPE lets the model recognize pretty much any spatial relationship between words[^oo]!

The speed of rotation (as well as a lot of other info, like head size and count) are in the GGUF. SmolLM3 actually uses a system it called NoPE— RopE is not applied on every fourth layer, for reasons discussed in their writeup.

# ---- Apply RoPE ----
RoPE = function(v, pos, base, dims){
  angles = pos * base ^ -(2*(1:(dims/2) - 1)/dims)
  odds = seq(1, dims, by = 2) #XY pairs are interleaved
  evens = seq(2, dims, by = 2)
  X = v[odds]
  Y = v[evens]

  v[odds] = X * cos(angles) - Y * sin(angles) #rotate X
  v[evens] = Y * cos(angles) + X * sin(angles) #rotate Y
  v
}

#apply rope to q and k heads
token_pos = #absolute position of token in chat
if (layer$id %% 4 != 3) {
  for (i in 1:16) q[[i]] = RoPE(q[[i]], token_pos, metadata$smollm3.rope.freq_base, 128)
  for (i in 1:4) k[[i]] = RoPE(k[[i]], token_pos, metadata$smollm3.rope.freq_base, 128)
}

Now, the K and V heads are added to the cache of all previous K and V heads. This is where the model stores info on all previous tokens— and because we cache after applying RoPE, this already includes info about the tokens’ positions. After adding to the cache, we retrieve the entire KV cache to use in the next step.

# ---- Retrieve, Add to, and Cache KV ----
  kv = retrieve_cache(layer$id)
  for (i in 1:heads_kv) {
    k[[i]] = rbind(kv$k[[i]], k[[i]]) #append new values as the bottom (newest) row
    v[[i]] = rbind(kv$v[[i]], v[[i]])
    kv$k[[i]] = k[[i]]
    kv$v[[i]] = v[[i]]
  }
  save_cache(kv, layer$id)
  # k, v: list of 4 heads, 4 x [total tokens so far, 128]

Then it’s finally time to put the QKV together. For each q head, a new head is created (I call these “ctx” heads). The q head is matrix-multiplied with its corresponding k head, normalized a bit, and then matrix-multiplied again with the corresponding v head. Theoretically, this is the step where the model actually combines and processes the information about previous tokens with its current fuzzy guess (embedding) of the next token.

# ---- Apply Attention ----
softmax = function(v){x = exp(v - max(v)); x / sum(x)}
attend = function(q,k,v,head_dim) {#qkv heads, not the full list
  qk = softmax(q %*% t(k))/sqrt(head_dim)) 
  # qk: [1, 1] <- just a single value per head during generation!
  qk %*% v
}
#attend() is a bit more complex when prefilling, not shown here

ctx = vector('list',16)
for (i in 1:heads_q) { #get ctx for each Q head
  kv_head = ceiling(i/4) #which kv head? (each kv head is used for 4 q heads, hence GQA architecture)
  ctx[[i]] = attend(q[[i]], k[[kv_head]], v[[kv_head]], 128)
}
# ctx: list of 16 heads, 16 x [1, 128]

Finally, all of these new heads are put back together to make one 1x2048 vector.

# ---- Reassemble Full Vector ----
Context = matrix(0, 1, 2048)
for (i in 1:16) Context[,128 * (i - 1) + 1:128] = ctx[[i]] #recombine ctx heads, the inverse of how we broke up Q
# Context: [1, 2048]

Then this vector is matrix-multiplied with a set of weights for the layer, and added to the original input vector.

# ---- Weight Output ----
Out = Context %*% t(layer$attn_output.weight)
# Out: [1, 2048]
# attn_output.weight: [1, 2048]

# ---- Add to Input ----
Attn_Output = input + Out
# Attn_Out: [1, 2048]

Then this output is fed into the FFN section.

5.2.2 FFN (Feed-Forward Network)

Just like in the Attention section, we take the previous output and normalize+weight it before anything else.

# ---- Normalize Input ----
NormIn = RMSN(Attn_Out, layer$ffn_norm.weight, meteadata$smollm3.attention.layer_norm_rms_epsilon)
# NormIn: [1, 2048]
# ffn_norm.weight: [1, 2048]

Then, we matrix-multiply with two tensors to increase our dimension and set up a traditional neural network gate. Gate = G, Values = U.

# ---- Project Up ----
G = NormIn %*% t(layer$ffn_gate.weight) #Gate
U = NormIn %*% t(layer$ffn_up.weight) #Up Weights
# G, U: [1, 11008]
# ffn_gate.weight, ffn_up.weight: [11008, 2048]

Apply the gate with element-wise multiplication. Modern LLMs use SwiGLU rather than the traditional ReLU13,14, but it serves the same purpose: negative values in G are snapped to 0, which also zeroes the corresponding value in U when they’re multiplied.

SiLU = function(x) x/(1 + exp(-x))
# ---- Activate Weights ----
Activation = SiLU(G) * U
# Activation: [1, 11008]

Now the activated weights are projected back down to 1x2048, and added again to the input.

# ---- Project Down ----
Out = Activation %*% t(layer$ffn_down.weight)
# Out: [1, 2048]
# ffn_down.weight: [2048, 11008]

# ---- Add to Input ----
FFN_Out = Attn_Out + Out
# FFN_Out: [1, 2048]

Then we send the final result (FFN_Out) off to the next layer!

5.3 transformer.R

This file contains all the math for an embedding through a transformer layer. (all code is also in the ggufr repo)

The code is a bit more robust than what was shown in the previous sections; it works for both prefill and single-token generation, and uses slightly different variable names. It’s still commented and designed to be highly readable— it was made as a blueprint for myself if I ever want to implement this transformer elsewhere!

6. Unembedding

As discussed in the previous section, we dot-product the fuzzy output embedding with the embeddings for the entire model vocabulary to get a list of similarity values. There’s a lot of ways to choose a token from this; the typical approach is to turn the values into a list of probabilities for each word (such that the sum of all probabilities is 1), incorporating some parameters like temperature (how “stretched” the probabilities area; lower temperature makes the highest-similarity tokens significantly more likely to be chosen), top-K (only the top K most similar tokens are considered for choosing), min-P (all tokens below a certain likelihood are ignored), and quite a few more to reduce the likelihood of getting a wacky prediction. Once a token is chosen, it’s added to the generated text, re-embedded, and fed back into the model for the next prediction.

Interestingly, these parameters are somewhat naive, and I haven’t seen any public resources quantifying how to best use them. We’re very clearly still in the early stages of using LLMs, where there’s a million options, everyone is trying to figure out what works, and a lot of the process is unoptimized15 or yet to be practically quantified.

7. The ggufr Package

7.1 Setup

The package has just one dependency and is relatively easy to try out and play around with. Running the model is slow but shouldn’t require a particularly powerful computer.

To get started, run this to install the package and download16 the 3GB GGUF17 to run it with. This will take 5 or 10 minutes depending on your internet speed.

devtools::install_github("gbkorr/ggufr")
library(ggufr)
path = paste0(tools::R_user_dir('ggufr'),"/SmolLM3-3B-Q8_0.gguf")
download_model(path) #if you want to undo this, run delete_model(path)

Then to load the model, run this. During this process, the tensors are unpacked and saved in tempdir(); the unpacked tensors take up 25GB of space, so make sure your computer has some free space. The tensors are deleted when you close R/RStudio, so you’ll get the space back.

#this will take around 30 minutes
model = read_gguf(path)

Once that’s finished, the model is ready! You’ll be able to run it as much as you like until you close R.

7.2 Usage

The main function to run inference is infer(). This will automatically insert your prompt into the chat template and run the model on it. Token generation speed is mainly tied to how fast R can load the .rds tensor layers; I get around a token a minute on my computer.

infer(
  prompt, #the text to feed into the model
  model, #pass the model object in
  instructions=NULL, 
    #set this to a string if you want to add a system prompt, 
    #e.g. "You are an AI assistant that always rhymes."
  use_template=TRUE, 
    #set this to FALSE to feed ONLY the prompt into the model; 
    #it will behave as a word predictor as discussed in section 3.3
  show_template=FALSE, 
    #prints the full input and output text, instead of a cleaned "prompt | response" view.
  temperature=1 #prediction temperature
)

Here’s what I got from this: "Who is Hadley Wickham?" |> infer(model)

Text generation in progress. In another prompt, it told me Hadley was Canadian… factual details are rather hit-or-miss for small LLMs.

That’s about it. Mainly, this package provides a (relatively) simple codebase that should make it easy to tweak and play around with the inferencer.

8. Learning Process

I took a pretty new approach (for me) for learning things for this project.

Resources for understanding LLM architecture on an implementation level are quite scattered due to the fast-paced nature of AI development in the past few years. Many of the best guides are from 202318 and 2024, where it’s hard to tell what is and isn’t outdated.

I usually prefer finding articles myself or reading a textbook to learn how to do something, but because of these reasons, I decided this would be a good situation to learn from an LLM. For such a frequently-discussed, cutting-edge topic, I figured one of the big models could do a good job finding and synthesizing recent, relevant sources on the topic of LLM implementation. I started by asking for an overview of the modern GGUF and transformer structure, and how this differs from early models, and received a detailed and accurate response— for broad, bigger-picture questions like these, I’ve found LLMs to answer quite well.19

I ended up structuring this project almost like a class: I would ask Gemini to describe a part of the LLM implementation, and then code it entirely on my own, occasionally asking for clarification and advice— but not code!— when needed.

This structure worked tremendously well for me— because I was doing the implementation myself, I was building the important experience and understanding myself, and my own experimentation ironed out any (rare) inaccuracies in the “teacher’s” teaching.

Now, I’m sure an AI could have written the code for me as well, but that would miss the point— the whole goal was to give myself a deeper (and practical) understanding of how LLM inferencing worked, and I certainly got it!

Footnotes

  1. I can’t even find many from-scratch projects, let alone writeups! And the few I have found have been heavily vibe-coded and vibe-written…↩︎

  2. I’m not sure it really counts as a package— just a set of mostly undocumented functions you can use.↩︎

  3. Except for tokenizing, which needs the re package to perform one extended regex.↩︎

  4. Token, but the nuance isn’t relevant until later.↩︎

  5. By hand; no text or code on this page was AI-generated! That’s what makes it fun, you see.↩︎

  6. Who wasn’t the nicest fellow, unfortunately.↩︎

  7. From its publication in 2017 to the rise of LLMs in 2023. Of course, lots of research involving it led up to the events of 2023, but the general idea stands.↩︎

  8. Actually, it’s the embeddings for ” breakfast”, ” lunch”, and ” baseball”, since words are usually encountered after a space in natural language. There are a lot more words like this in the model dictionary than without a space.↩︎

  9. By turning the dot product results into probabilities, so more closely related tokens are more likely to be selected.↩︎

  10. Some of the values in the vectors will be negative, so the average value doesn’t go up.↩︎

  11. There’s plenty of explanations online— though I’m a little suspicious of how accurately the QKV’s actual behavior matches how they’re intended to work.↩︎

  12. i.e. more precisely: the q heads don’t interact with each other, the k heads don’t interact with each other, but each q head interacts with a k head. This means the math for each q head can all be performed in parallel.↩︎

  13. Unlike the traditional ReLU gates, SwiGLU preserves small negative values, which makes the deep learning work better. Why? From the official paper introducing SwiGLU: “We offer no explanation as to why these architectures seem to work; we attribute their success, as all else, to divine benevolence.” -Noam Shazeer↩︎

  14. Actually, here’s some good discussion of why SwiGLU might work better. I’m not sure what Noam was on about.↩︎

  15. Like the fact that we use jinja and Ġ, yeesh. All the focus at the moment is making things run faster (which is the best thing to focus on, of course).↩︎

  16. From the official Huggingface endpoint; the group that both created this model and generally hosts downloads for all local LLMs.↩︎

  17. Which will be saved in tools::R_user_dir('ggufr'). R provides a dedicated folder for each package to store files like this in.↩︎

  18. E.g. https://www.youtube.com/watch?v=kCc8FmEb1nY↩︎

  19. Including when I ask them a broad question about something I’m quite experienced with.↩︎