Kagamino's Blog

In 2019 I first got into tinkering with machine learning. I kind of dropped out of it at the end of the year, and now that it’s all over the news, I feel I can’t draw a direct connection to what I used to know. Today I’d like to try and bridge some of that gap.

The framework era

The classic work at the time was MNIST, hand written character recognition. This was cool because I think you could actually write the whole neural network in pure python, and still have it achieve reasonable performance within tens of minutes of training.

With today’s Torch framework, a simple solution looks like this simple module. It’s a lot of jargon, but there is a single flow for data, and only two layers are stateful (conv1 and fc) and actually learn. Those states are “the weights”.

class SimpleMNIST(nn.Module):
    def __init__(self):
        super()
        # Input: single-channel image (28x28)
        # 1. 2D convolution: 1 channel -> 8 channels, 3x3 filters
        self.conv1 = nn.Conv2d(in_channels=1, out_channels=8, kernel_size=3, padding=1)
        # 2. Linear layer: 8 channels * 14x14 pixels -> 10 classes (digits 0 to 9)
        self.fc = nn.Linear(8 * 14 * 14, 10)

    def forward(self, x):
        x = self.conv1(x)  # Shape: [batch, 8, 28, 28]
        x = F.relu(x)  # ReLU activation
        x = F.max_pool2d(x, 2)  # Shape: [batch, 8, 14, 14]
        x = torch.flatten(x, 1)  # Flatten: [batch, 1568]
        x = self.fc(x)  # Shape: [batch, 10]
        return x

When I was doing research for my master’s degree, each framework had a way to store and load weights for a specific module, or architecture. If you changed a step, you could not use the weights. This meant you basically had to share your script alongside your weights so someone could use it.

Runtimes and shared formats

In 2019, the ONNX format became widely used. This format combines the architecture with the weights, so anyone can run a shared model. It comes with an official backend from Microsoft, ORT - ONNX Runtime.

To me, a simple definition of a runtime would be Runtime: model, input → output. If you remove the variable model, you end up with a more classic algorithm. In the same idea, Python is a runtime Python: code, input → output. The runtime itself can contain deterministic algorithms, and also provide resource management (memory allocation, thread scheduling, …). Traditionally, in machine learning, pattern recognition benefits the most from using this trainable model slot.

So you can basically use any training framework, export your trained model in ONNX, and a device with ORT can run the model in production. I can try this for myself.

torch.onnx.export(
    model,
    torch.randn(1, 1, 28, 28),
    "mnist_simple.onnx",
    input_names=["input_image"],
    output_names=["predictions"],
    opset_version=18,
)

Then, with the help of onnx lib, I can see the architecture is encoded.

import onnx

onnx_model = onnx.load("mnist_simple.onnx")
onnx.checker.check_model(onnx_model)
for i, node in enumerate(onnx_model.graph.node):
    print(f"Node {i} : {node.op_type} (Input: {node.input} → Output: {node.output})")

Node 0 : Conv (Input: ['input_image', 'conv1.weight', 'conv1.bias'] → Output: ['conv2d'])
Node 1 : Relu (Input: ['conv2d'] → Output: ['relu'])
Node 2 : MaxPool (Input: ['relu'] → Output: ['max_pool2d'])
Node 3 : Reshape (Input: ['max_pool2d', 'val_4'] → Output: ['view'])
Node 4 : Gemm (Input: ['view', 'fc.weight', 'fc.bias'] → Output: ['predictions'])

I can see the ONNX format defines ops (Relu, Reshape, …), and ORT implements kernels that the model can use. If you export a model with a recent ONNX but run it on an old runtime, the architecture would not be supported because not implemented. This allows programmatic architectures alongside highly performant, hardware specific kernels.

This is not hypothetical: in my recent tests around local LLMs, llama.cpp refused to run Ling 3.0 Tiny, because of an unsupported architecture. The GGUF file declares bailingmoe3, a hybrid design that was brand new. The individual bricks already existed in llama.cpp, but each architecture needs its own explicit graph implementation - this one landed on August 17, 2026 (PR #26608), right after my tests. Same story as ONNX: the format carries the declaration, the runtime implements it.

Any app can now use the model, but it needs to provide the pre and post processing - transforming an image into a vector, and output probabilities into characters.

def load_image(path: str) -> np.ndarray:
    # Preprocessing must mirror the training pipeline: grayscale, 28x28, [0, 1]
    image = Image.open(path).convert("L").resize((28, 28))
    pixels = np.asarray(image, dtype=np.float32) / 255.0
    if pixels.mean() > 0.5:
        # MNIST digits are white on black; invert dark-background images
        pixels = 1.0 - pixels
    return pixels.reshape(1, 1, 28, 28)


def detect_char(session: ort.InferenceSession, image_path: str) -> int:
    input_name = session.get_inputs()[0].name
    logits = np.asarray(session.run(None, {input_name: load_image(image_path)})[0])
    return int(logits[0].argmax())


session = ort.InferenceSession(
    "mnist_simple.onnx", providers=["CPUExecutionProvider"]
)

Specialized runtimes

As you can see, pre and post processing can be tedious. ORT is a very general runtime, but for specific use cases, runtimes can provide a more complete harness.

In Computer Vision - what I’m doing here, I need image encoding, maybe resizing and normalization. Actually the few lines I wrote could become a Computer Vision runtime. The CLI could take any image in, do the transformation, and use the model of my choice as the core intelligence.

In Speech Recognition, voice activity detection, token decoding, FFT are steps that benefit little from learning, and instead benefit from highly optimized code - FFT is done best without matrix multiplication. This is why sherpa-onnx is, as its name suggests, a speech specific, hardcoded harness around an ORT pass.

With LLMs, you hear way more often about GGUF than ONNX, and llama.cpp than ORT. LLMs need tokenization - the pre processing step, caching, quantization, parallelization, and more that I don’t know.

Example: SqueezeNet, 1.3 MB for Computer Vision

Let’s take a small vision model and use Netron to display its architecture. SqueezeNet was released in 2016.

Here is a small compilation of formats & runtimes couples, in case you hear about them:

• ONNX + ORT - General purpose
• ONNX + sherpa-onnx - Real time audio
• GGUF + llama.cpp / ggml - LLM, memory optimized, compression
• Safetensors + vLLM - FP16, mmap, datacenter LLM inference
• TensorRT Engine (.engine / .plan) + NVIDIA Runtime - CUDA kernels, audio / vision
• OpenVINO IR (.xml + .bin) + Runtime - Intel optimized
• Core ML (.mlpackage) + Framework - Apple Silicon
• RKNN + rknn-toolkit-lite - Rockchip / ARM Edge AI

I hope you get the picture. Each use case has specific pre and post processing, and also processing steps that are hardcoded - and often not adapted to GPUs. In the middle, it loads matrix weights for large multiplication - that’s the part that learns. sherpa-onnx for speech, fastdeploy for Vision all use ORT for this middle ONNX compute. Others have recoded a new runtime to get even better performance - llama.cpp and the GGUF format support very aggressive compression on weights for LLMs.