Langur Monkey - Toni Sagristà Sellés

GGUF quantization guide

Toni Sagrista Selles — Wed, 04 Mar 2026 00:00:00 +0000

I like running my own LLMs locally. Open models are becoming more and more powerful, with exciting releases like the latest Qwen 3.5 family scoring highly in benchmarks even in their smaller variants. This makes managing and running your own models more viable, as it becomes increasingly easy to repurpose old hardware for local inference with progressively better results. For local users and modest purposes, the GGUF format introduced by llama.cpp is the de-facto default.

Since local inference is typically heavily restricted by the available hardware, several optimization techniques have been implemented to make the models leaner and faster. Perhaps the most important of these is quantization, which trims down the bit count per parameter to achieve lower memory usage and (sometimes) faster inference. The challenge is that there are many different formats and strategies for quantization. In this post, I summarize them, providing a bird’s-eye view on the available techniques, their strengths, and their weaknesses.

Naming conventions

Most GGUF quantization names follow this pattern: Q{bits}{method}{size}

Component	Meaning
`Q`	Quantized format
`2/3/4/5/6/8`	Bits per weight (lower = smaller file, more compression)
`K`	“K-quant”: grouped/blockwise quantization with per-group scales
`I` / `IQ`	“I-quant”: importance-matrix/non-linear quantization for aggressive compression
`0` / `1`	Legacy ungrouped formats (symmetric/asymmetric)
`_S` / `_M` / `_L`	Small/Medium/Large: precision mix across tensor types
`_XS` / `_XXS` / `_NL`	Extra-small / Non-linear variants for I-quants

Unquantized formats

These are base formats with no compression. Usually, but not always, models are trained in these formats.

Format	Bits	Description	Notes
FP32	32-bit float	Full precision, ~26GB for 7B model	Research, debugging
FP16	16-bit float	Half precision, ~13GB for 7B model	GPU training/inference baseline
BF16	16-bit brain float	Same size as FP16, better dynamic range for training	Training on modern GPUs

These preserve maximum accuracy but require significant VRAM. Most local users quantize to reduce size by 75-90%.

Legacy quantization formats (`Q_0`, `Q_1`)

Simple per-block linear quantization. Fast but less accurate at low bits.

Following is a table that contains the format name, number of bits, size (for a 7B model), perplexity, and some notes/recommendations. The perplexity represents the difference between the quantized model and the base model, with lower scores indicating better accuracy and less uncertainty.

Format	Bits	Size (7B)	Perplexity $\Delta$	Notes
Q8_0	~8-bit	~6.7 GB	+0.0004	Near-lossless; safe INT8 baseline
Q5_1	~5-bit	~4.7 GB	+0.0415	Legacy; superseded by Q5_K_M
Q5_0	~5-bit	~4.3 GB	+0.0796	Legacy; balanced but outdated
Q4_1	~4-bit	~3.9 GB	+0.1846	Legacy; substantial quality loss
Q4_0	~4-bit	~3.5 GB	+0.2499	Legacy; high quality loss

K-quants or I-quants are generally preferred over legacy formats at $\leq 5$ bits. Q8_0 remains useful for compatibility.

K-Quant formats (modern default)

Use two-level block quantization (small blocks $\rightarrow$ super-blocks) with double-quantized scales for better quality-per-bit.

Format	Effective Bits	Size (7B)	Perplexity $\Delta$	Notes
Q6_K	~6.0	~5.15 GB	+0.0044	“Almost lossless” with savings
Q5_K_L	~5.3	~4.6 GB	+0.010	High-quality 5-bit
Q5_K_M	~5.1	~4.45 GB	+0.0142	Recommended high-quality 5-bit
Q5_K_S	~4.9	~4.33 GB	+0.0353	Recommended balanced 5-bit
Q4_K_L	~4.7	~4.0 GB	+0.040	Relaxed 4-bit mix
Q4_K_M	~4.5	~3.80 GB	+0.0535	Most popular default
Q4_K_S	~4.3	~3.56 GB	+0.1149	Smaller 4-bit, more loss
Q3_K_L	~3.5	~3.35 GB	+0.1803	Aggressive 3-bit
Q3_K_M	~3.3	~3.06 GB	+0.2437	Balanced 3-bit
Q3_K_S	~3.1	~2.75 GB	+0.5505	Very small, high loss
Q2_K	~2.5	~2.67 GB	+0.8698	Extreme compression, not recommended

Q4_K_M seems to be the community sweet spot, as it offers ~75% size reduction with minimal noticeable quality loss for most tasks.

I-Quant formats (aggressive compression)

IQs use non-linear reconstruction, lookup tables, and importance-matrix calibration for maximum quality at very low bit counts. They trade decoding speed for size.

Format	Effective Bits	Size (7B)	Notes
IQ4_NL	~4.5	~3.9 GB	Non-linear 4-bit; CPU-friendly speed
IQ4_XS	~4.25	~3.7 GB	Best quality/size at 4-bit; slightly slower decode
IQ3_M	~3.6	~3.2 GB	High-quality 3-bit I-quant
IQ3_S	~3.4	~3.0 GB	Balanced 3-bit aggressive
IQ3_XS	~3.2	~2.8 GB	Very small 3-bit
IQ3_XXS	~3.0	~2.6 GB	Extreme 3-bit compression
IQ2_M	~2.7	~2.4 GB	High-quality 2-bit (rare)
IQ2_S	~2.5	~2.2 GB	Aggressive 2-bit
IQ2_XS	~2.3	~2.0 GB	Very aggressive 2-bit
IQ2_XXS	~2.1	~1.9 GB	Extreme; significant quality loss

I-quants require importance matrix (imatrix) calibration during quantization for best results. Without it, quality can degrade noticeably.

Special formats

Format	Bits	Description
TQ1_0	~1.6	Ternary quantization $\{-1, 0, +1\}$, for massive models like DeepSeek where fitting in VRAM is critical

Decision guide

Best balance $\rightarrow$ Q4_K_M (default recommendation)
Maximum quality, still compressed $\rightarrow$ Q5_K_M or Q6_K
Tight VRAM, acceptable quality loss $\rightarrow$ Q4_K_S or IQ4_XS
Absolute smallest file $\rightarrow$ IQ3_XS / Q3_K_S (test quality first)
Near-original accuracy $\rightarrow$ Q8_0 or keep FP16/BF16
Running on CPU $\rightarrow$ Q4_K_M or IQ4_NL for better decode speed
Fitting a 70B model on 24GB VRAM $\rightarrow$ IQ3_S or Q3_K_M + CPU offload

Conclusions

As you see, this looks like the wild west at first glance, but this mess is not without order. Most importantly, there are good reasons behind every quantization type. Here are some takeaways.

Bits $\neq$ quality alone: A well-designed 4-bit format (Q4_K_M, IQ4_XS) can outperform a naive 5-bit legacy format.
Suffixes matter: _M variants selectively keep sensitive layers at higher precision, improving quality with minimal size cost.
Hardware matters: I-quants compress more but decode slower on CPUs; K-quants often give better tokens/sec on consumer hardware.
Calibration helps: Models quantized with an importance matrix (imatrix) retain more accuracy, especially at $\leq 3$ bits.
Test your use case: Perplexity benchmarks are only guides. Always validate outputs for your specific tasks.

For the latest format support and benchmarks, check the llama.cpp repository or community hubs like Hugging Face, where curators like bartowski and Unsloth publish tested GGUF variants.

LM Studio on systemd linger

Toni Sagristà Sellés — Fri, 27 Feb 2026 00:00:00 +0000

The release of LM Studio 0.4.5 has introduced a much needed feature in this local LLM suite that has it much more attractive with respect to other similar projects. LM Link allows you to connect multiple LM Studio instances across your network to share models and perform inference seamlessly.

By sheer chance, I was just playing around with setting up an LM Studio server in an old laptop that I planned to use for inference. I would connect AnythingLLM clients to it to make API requests. The timing of 0.4.5 was perfect for me, as I could now use LM Studio for inference directly, and forget about using up my own Tailscale network. But some setup was needed in the laptop. To make this work effectively, the LM Studio server needs to run in the background, start automatically on boot, and persist even when I’m not logged in.

The LM Studio website provides the source of a service file. It suggests creating it as a system-wide service, which is weird, as the default installation method (at least on Linux) sets everything up in the user home directory. I modified it a bit to make things clean, as I want this to be a user systemd service. It keeps the process tied to your user environment but, with a little tweak called lingering, allows it to run without an active SSH or GUI session. Here is the setup.

By default, user services stop the moment the user logs out. To prevent this and allow the LM Studio daemon to start at boot and stay alive, run:

loginctl enable-linger $USERNAME

Then, create a directory for your user services if it doesn’t exist:

mkdir -p ~/.config/systemd/user/

After that, create a file named lms.service in that directory (~/.config/systemd/user/lms.service), with the following contents:

[Unit]
Description=LM Studio Server
After=network.target

[Service]
Type=forking
# This line kills any existing lms processes to prevent the "left-over" error
ExecStartPre=-/home/$USERNAME/.lmstudio/bin/lms daemon down
ExecStartPre=/home/$USERNAME/.lmstudio/bin/lms daemon up
ExecStart=/home/$USERNAME/.lmstudio/bin/lms server start
# Give it a specific PID file if lms supports it, otherwise systemd guesses
ExecStop=/home/$USERNAME/.lmstudio/bin/lms server stop
ExecStopPost=/home/$USERNAME/.lmstudio/bin/lms daemon down

# Ensure systemd doesn't get confused by the CLI exiting
RemainAfterExit=yes

[Install]
WantedBy=default.target

Once the file is saved, tell systemd to reload its configuration and enable the service:

# Reload the user daemon
systemctl --user daemon-reload

# Enable the service to start on boot
systemctl --user enable lms.service

# Start the service now
systemctl --user start lms.service

If you want to load a specific model by default, add an additional ExecStartPre line, like this:

ExecStartPre=/home/YOUR_USERNAME/.lmstudio/bin/lms load openai/gpt-oss-20b --yes

You can check the service status with systemctl --user lms.service. And that is it. You can now use your old hardware for inference with small local LLMs from any of your other machines.

Yet another architectural update for Play Kid

Toni Sagrista Selles — Wed, 28 Jan 2026 00:00:00 +0000

I finished my previous post on Play Kid, only two days ago, with the following words:

Next, I’ll probably think about adding Game Boy Color support, but not before taking some time off from this project.

Yeah, this was a lie.

I have previously written about Play Kid, my Game Boy emulator. Here, I introduced it and talked about the base implementation and the quirks of the Game Boy CPU, PPU, and hardware in general. Here, I explained the tech stack update from SDL2 to Rust-native libraries. In that last post, I mentioned the dependency version hell I unwillingly descended into as a result of adopting pixels as my rendering library. This forced me to stay on very old versions of wgpu and egui.

I want my Game Boy emulator to use the latest crate versions for various reasons, so this could not be. I saw a simple and direct path of update, which consisted on adopting eframe to manage the application life cycle, directly drawing the LCD to a texture, and dropping pixels altogether.

One of the things that nagged me about the pixels crate is that its integration with egui was kind of lackluster. There was no easy way to render the pixels frame buffer to an egui panel, so I had to render it centered inside the window. The immediate mode GUI lived in mostly in Window widgets on top of it. Unfortunately, these windows occluded the Game Boy LCD. In a proper debugger, you must be able to see the entire LCD plus the debug interface.

So I dropped pixels and adopted a render-to-texture approach. In it, you create the LCD texture at the beginning from the egui context, and then copy the LCD contents to it in the update() method.

 fn new(...) -> Self {
 [...]
 // Create texture from egui context and Game Boy LCD width and height.
 let texture = _cc.egui_ctx.load_texture(
 "lcd_screen",
 egui::ColorImage::new(
 [DISPLAY_WIDTH, DISPLAY_HEIGHT],
 vec![egui::Color32::BLACK; DISPLAY_WIDTH * DISPLAY_HEIGHT],
 ),
 egui::TextureOptions::NEAREST,
 );
 // Create app struct.
 Self {...}
 }

 [...]

 fn update(...) {
 [...]

 // Render LCD to texture.
 if frame_ready {
 let size = [DISPLAY_WIDTH, DISPLAY_HEIGHT];
 let color_image =
 egui::ColorImage::from_rgba_unmultiplied(size, &machine.memory.ppu.fb_front);
 self.screen_texture
 .set(color_image, egui::TextureOptions::NEAREST);
 }
 }

With this, we can easily render the texture in egui’s CentralPanel, and the debug interface in a SidePanel to the right. This is the result:

The debug panel, showing the machine state and a code disassembly.

Some additional tweaks here and there, and the UI looks much more polished and professional in version 0.3.0.

Version 0.4.0 enables loading ROM files from the UI. Initially I thought about making the cartridge struct optional with Option<Cartridge>, but this spiraled out of control fast. I found that making the full Machine (which contains the Memory, PPU, APU, Joystick, etc.) optional worked much better, as there was only one reference to it in the top-level struct, PlayKid. And, like so, you can dynamically load ROM files from the UI:

Play Kid with the top menu bar and the ‘Open ROM’ menu entry.

So, what’s in the future for Play Kid? Well, there are a couple of features that I’d really like to add at some point:

Save states—Currently, Play Kid emulates the SRAM by saving and restoring it from files for supported games. I would like to add saving and restoring the full state of the emulator in what is known as save states. Possibly, the serde crate can help with this.
GBC—Of course, I would like adding Game Boy Color support. It is not trivial, but also not exceedingly complicated. I never owned a GBC, so I’d see this as a good opportunity to explore its game catalog.

Game Boy emulator tech stack update

Toni Sagrista Selles — Mon, 26 Jan 2026 00:00:00 +0000

Notice

This post is about Play Kid version 0.2.0, which uses pixels to create and manage the pixel frame buffer. Since then, versions 0.3.0 and 0.4.0 have been released, which dropped pixels in favor of rendering directly to a texture. See this write-up for more information.

In my previous post, I shared the journey of building Play Kid, my Game Boy emulator. At the time, I was using SDL2 to handle the “heavy lifting” of graphics, audio, and input. This was released as v0.1.0. It worked, and it worked well, but it always felt a bit like a “guest” in the Rust ecosystem. SDL2 is a C library at heart, and while the Rust wrappers are good, they bring along some baggage like shared library dependencies and difficult integration with Rust-native UI frameworks.

So I decided to perform a heart transplant on Play Kid. For version v0.2.0 I’ve moved away from SDL2 entirely, replacing it with a stack of modern, native Rust libraries: wgpu, pixels, egui, winit, rodio, and gilrs:

winit & pixels: These handle the windowing and the actual Game Boy frame buffer. pixels allows me to treat the 160x144 LCD as a simple pixel buffer while wgpu handles the hardware-accelerated scaling and aspect ratio correction behind the scenes.
egui: This was a big step-up. Instead of my minimal homegrown UI library from the SDL2 version, I now have access to a full-featured, immediate-mode GUI. This allowed me to build the debugger I had in mind from the beginning.
rodio & gilrs: These replaced SDL2’s audio and controller handling with pure-Rust alternatives that feel much more ergonomic to use alongside the rest of the machine.

Debug panel

The most visible change is the new Debug Panel.

The new integrated debugger features a real-time disassembly view and breakpoint management.

One of the coolest additions is the Code disassembly panel. It decodes the ROM instructions in real-time, highlighting the current PC and allowing me to toggle breakpoints just by clicking on a line. The breakpoints themselves are now managed in a dedicated list, shown in red at the bottom.

The rest of the debug panel shows what we already had: the state of the CPU, the PPU, and the joypad.

Your browser does not support the video tag.

Playing around with the new Play Kid UI based on egui.

Dependency hell

Of course, no modern Rust migration is complete without a descent into dependency hell. This new stack comes with a major catch: pixels is a bit of a picky gatekeeper. Its latest version is 0.15 (January 2025). It is pinned to an older version of wgpu (0.19 vs the current 28.0), and it essentially freezes the rest of the project in a time capsule.

To keep the types compatible, I’m forced to stay on egui 0.26 (current is 0.33) and winit 0.29 (current is 0.30), even though the rest of the ecosystem has moved on to much newer, shinier versions. It’s kind of frustrating. You get the convenience of the pixels buffer, but you pay for it by being locked out of the latest API improvements and features. Navigating these version constraints felt like solving a hostage negotiation between crate maintainers. Not very fun.

Conclusion

Despite the dependency issues, I think the project is now in a much better place. The code is cleaner, the debugger is much better, and it’s easier to ship binaries for Linux, Windows, and macOS via GitHub Actions.

If you’re interested in seeing the new architecture or trying out the new debugger, the code is updated on Codeberg and GitHub.

Next, I’ll probably think about adding Game Boy Color support, but not before taking some time off from this project.

Play Kid

Tue, 20 Jan 2026 00:00:00 +0000

Play Kid is yet another Game Boy emulator, written in Rust. But hey, it is MY Game Boy emulator, and I’m proud of it. I wrote some words about its development here. Find all posts about Play Kid here.

Play Kid running different games with different color palettes.

Here are the main features of Play Kid:

All CPU instructions implemented.
Full memory map implemented.
Modes: ROM, MBC1, MBC2, MBC3.
Audio is implemented, with 4 channels, envelopes, sweep, and stereo.
Supports game controllers.
Multiple color palettes.
Save screenshot of current frame buffer.
FPS counter.
Respects 160:144 aspect ratio by letter-boxing.
Debug panel:
- Step instruction.
- Step scanline.
- Pause/continue current execution.
- Displays internal state of CPU, PPU, and Joypad.
- Full program disassembly, with breakpoints.
Save RAM to .sav files to emulate the battery-backed SRAM. Those are saved every minute.
Working games/roms:
- Passes dmg-acid2
- Tetris
- Pokémon
- Super Mario Land
- Super Mario Land 2: 6 Golden Coins
- Wario Land (Super Mario Land 3)
- Wario Land II
- Bugs Bunny Crazy Castle
- The Amazing Spider-Man
- Dr. Mario
- Probably many, many more
For Linux, macOS, and Windows.

Downloads

You can grab packages for Linux, macOS, and Windows here:

Build

Build the project with cargo build.

Run

The usual Rust stuff.

 cargo run

You can also pass in a ROM file with cargo run -- your-rom.gb.

Make the binary with:

 cargo build --release

Operation

If you don’t pass in any ROM file as an argument, you need to select using the top menu bar, File▶Open ROM….

Here are the Joypad keyboard mappings:

enter - Start button
space - Select button
a - A button
b - B button

The keyboard is clumsy for playing Game Boy games, so you can use any game controller. Controllers are detected when hot-plugged.

Additionally, there are some more actions available:

p - change the palette colors
w - trigger the SRAM save operation to .sav file.
f - toggle FPS monitor
s - save a screenshot, with name screenshot_[time].jpg
d - enter debug mode
Esc - exit the emulator

You can also use the provided UI.

Debug panel

You can open the debug panel any time by pressing d, by clicking on Machine▶Debug panel…, or activate it at launch with the -d/--debug flag. The debug panel shows up in a translucent window. It provides a view of the internal state of the emulator, with:

Current address, instruction, operands, and opcode, to the top.
Internal state of CPU, PPU, and JOYP, to the left.
Disassembly of the program, to the right.
Breakpoints.

The debug panel, showing the machine state and a code disassembly.

You can use the provided UI controls to work with debug mode. You can also use the keyboard. These are the key bindings:

F6 - step a single instruction
F7 - step a scanline
F9 - continue execution until breakpoint (if paused), or pause execution (if running)
r - reset the CPU
d - exit debug mode and go back to normal full-speed emulation
Esc - exit the emulator

You can also use breakpoints. A list with the current breakpoint addresses is provided at the bottom. To create a breakpoint, either click on the address in the disassembly panel, or enter it (in $abcd format) into the text field and click +. Remove a breakpoint by clicking the × in the breakpoints list. Clear all current breakpoints with Clear all.

Your browser does not support the video tag.

Playing around with the Play Kid debug UI.

CLI args

There are some CLI arguments that you can use:

Play Kid 0.4.0
Toni Sagristà - tonisagrista.com
Minimalist Game Boy emulator for the cool kids.
Usage: playkid [OPTIONS] [INPUT]
Arguments:
[INPUT] Path to the input ROM file to load
Options:
-s, --scale <SCALE> Initial window scale. It can also be resized manually [default: 4]
-d, --debug Activate debug mode. Use `d` to stop program at any point
-f, --fps Show FPS counter. Use `f` to toggle on and off
--skipcheck Skip global checksum, header checksum, and logo sequence check
-h, --help Print help
-V, --version Print version

SDL2 version

Play Kid started as an SDL2 application, but it was moved to a pure Rust tech stack using pixels, winit, egui, and rodio. This makes it much easier to build for different targets (including WASM!). Additionally, the SDL2 version contains a minimalist homegrown UI library that I’m particularly proud about, but it can’t hold a candle to egui. It looks like this:

The debug mode.

The SDL2 version is forever tagged playkid-sdl2 (playkid-sdl2@codeberg, playkid-sdl2@github).

Useful links

Pandocs: https://gbdev.io/pandocs/
Complete tech reference: https://gekkio.fi/files/gb-docs/gbctr.pdf
Game Boy CPU manual: http://marc.rawer.de/Gameboy/Docs/GBCPUman.pdf
Game Boy CPU instructions: https://meganesu.github.io/generate-gb-opcodes/

Implementing a Game Boy emulator

Toni Sagristà Sellés — Tue, 20 Jan 2026 00:00:00 +0000

Notice

This post is about Play Kid version 0.1.0, which works with SDL2. Since then, version 0.2.0 has been released. This new version uses Rust native crates like winit, egui, rodio, and pixels. See this write-up for more information on the new version.

When I was a kid my parents got me and my brothers a brand new Game Boy. I used to play it a lot, and I was mesmerized and amazed at how such a tiny brick could give life to so many wonderful worlds. Some of my favorite games were Super Mario Land (1, 2, and 3, Wario Land), The Amazing Spider-Man, and Kirby’s Dream Land. We also had some other games, like Bugs Bunny Crazy Castle or Tetris, that I also played quite a lot. I remember fondly too.

The first Super Mario Land in particular was magical to me. I would complete it, and then do it again in one sitting. There were no save games, as it used MBC1 with no RAM (more on that later). You really had to git gud. The second play through had the same levels but more enemies, in what must have been one of the first implementations of New Game+. After this harder mode was completed, you could select and play any level. Too bad all the progress went away as soon as you powered off the device. Super Mario Land 2: 6 Golden Coins already introduced save games (used MBC1+RAM+Battery). It was a huge step-up in QoL for Game Boy games. We also had the original NES, and later the SNES, but the Game Boy always remained that cozy little fun device that you could just pick up and play for a quick session. It was a significant part of my childhood.

Your browser does not support the video tag.

Me playing Super Mario Land, levels 2 to 4, with my own emulator.

Some time ago, I wrote a CHIP-8 emulator in Rust, and found the experience very rewarding. The consensus online seems to be that the recommended next step, as far as emulator development goes, is the original Game Boy. I know that the internet is flooded with “Implementing a Game Boy emulator” kind of articles, and (judging by the GitHub counts) this is a very unoriginal project, but I still wanted to try my hand at it. So I took the bait and started working on it. Today, I’m excited to share my very own Game Boy emulator.

The original Game Boy, also referred to as DMG in nerdy spaces and released in 1989, is an excellent target for emulation. Its hardware is rather well-documented, its architecture is relatively simple compared to modern systems, yet it’s complex enough to present real challenges, which I for sure faced.

I called my implementation Play Kid. It is written in Rust and uses SDL2 for graphics, sound, and input. Here are the main features:

All CPU instructions implemented.
Full memory map implemented.
Modes: ROM, MBC1, MBC2, MBC3.
Audio is implemented, with 4 channels, envelopes, sweep, and stereo.
Supports game controllers via SDL (also hot-plugging).
Multiple color palettes.
Save screenshot of current frame buffer.
Respects 160:144 aspect ratio by letter-boxing.
Debug mode:
- Step instruction.
- Step scanline.
- Pause/continue current execution.
- FPS counter.
- Displays internal state.
- Breakpoints.
- Uses own minimal UI library with horizontal/vertical layouts, labels, buttons, and text fields.
Save RAM to .sav files to emulate the battery-backed SRAM.
Automatically adapts to multi-DPI setups by scaling the UI.
Working games/roms:
- Passes dmg-acid2
- Tetris
- Pokémon Red
- Super Mario Land
- Super Mario Land 2: 6 Golden Coins
- Super Mario Land 3: Wario Land
- Wario Land II
- Bugs Bunny Crazy Castle
- The Amazing Spider-Man
- Kirby’s Dream Land
- Dr. Mario
- Probably many more
Works on Linux, macOS, and Windows.

In this post, I navigate the general architecture of the Game Boy, emphasize interesting hardware quirks that made it a challenge to emulate properly, and present my implementation. I won’t go into too much detail, as I took care to make the code understandable and well-commented. In any case, for more details on a specific subject, you can refer to the code repository itself.

Foreword

I have based my implementation, mainly, on three resources:

Pan Docs – possibly the best documentation on the Game Boy system out there.
Game Boy CPU Manual – documents the Game Boy architecture in great detail.
Instruction table – all 8-bit opcodes into a single table, complete with descriptions for each one of them. Very handy.

On certain occasions, I found some of the Opcode descriptions to be unclear or a bit ambiguous. In these cases, I referred to actual pre-existing implementations. By far, the implementation I liked the most is RBoy. It is a very compact and mostly uncommented Game Boy Color emulator, written in Rust. Most of the instructions are the same as in the Game Boy DMG, so it worked very well for me.

Game Boy essentials

Before we get into the details, let’s talk a bit about how the Game Boy actually works. Emulation is basically about mimicking hardware in software. We’re not just running Game Boy code, we’re simulating an entire machine cycle by cycle.

The original Nintendo Game Boy. Original by Evan-Amos.

The Game Boy’s execution model is pretty simple. The CPU has a 16-bit PC register that points to the next instruction in memory. Each cycle, it fetches the byte at that address, decodes it into an instruction, executes it, and increments the PC. Instructions are composed of opcode+operand, and can be one, two, or three bytes long. Each instruction takes a specific number of machine cycles (M-cycles) to complete, usually between 1 and 6. Since the CPU runs at around 4.19 MHz (master clock) and one M-cycle equals four clock cycles (T-cycles), timing is important.

The system uses memory-mapped I/O. This means hardware components like the display, audio, and input are accessed by reading and writing to specific memory addresses. Want to check if the A button is pressed? Read from address 0xFF00. Need to start a DMA transfer? Write to 0xFF46. This design makes the architecture simpler, but it also means that memory access can trigger all sorts of side effects, and you need to manage the mapping in your code.

Interrupts let hardware grab the CPU’s attention. When something happens (the display finishing a frame, or a timer overflow), the corresponding interrupt flag is set. If interrupts are enabled (via the IME flag), the CPU pauses what it’s doing, pushes the current PC onto the stack, and jumps to a predefined interrupt handler address. This lets games sync with hardware events instead of constantly checking for them.

The hardware components all move forward together based on the CPU’s cycle count. When the CPU executes a 4 M-cycle instruction, the PPU advances 16 T-cycles worth of scanline rendering, the APU moves its sound generation forward, and timers tick along. Everything runs in lockstep, which means accurate emulation needs cycle-level precision across all the different parts.

Understanding this basic execution model (fetch, decode, execute, repeat, with interrupts and memory-mapped hardware mixed in) is key before getting into the specifics. Each part of the system we’ll look at next works within this framework.

The CPU

The heart of the Game Boy is a modified Intel 8080/Zilog Z80 processor running at approximately 4.19 MHz. Nintendo’s custom chip, sometimes called the SM83 or LR35902, includes most Z80 instructions but removes some and adds a few Game Boy-specific ones.

The CPU features eight 8-bit registers (A, B, C, D, E, F, H, L) that can be paired into four 16-bit registers (AF, BC, DE, HL), plus a 16-bit SP and the aforementioned PC. The F register is special, as its four most significant bits hold flags that track the results of operations:

Zero (Z)
Subtraction (N)
Half-Carry (H)
Carry (C)

In my implementation, the CPU is contained in machine.rs, which contains the main components of the hardware (memory, registers, PPU, APU, etc.), controls the execution state (with debug), and implements the instructions.

/// # Machine
/// The machine contains the registers, the memory, and the display, and
/// controls the execution and CPU state. It also implements the CPU, which
/// decodes and executes instructions.
pub struct Machine<'a, 'b> {
 /// The registers.
 pub registers: Registers,
 /// The main memory.
 pub memory: Memory<'a>,
 /// The display.
 display: Display<'b>,
 /// The UI state.
 ui_state: Rc<RefCell<UIState>>,
 /// Interrupt master enable flag.
 ime: bool,
 /// EI operation is delayed by one instruction, so we use this counter.
 ei: u8,
 /// DI operation is delayed by one instruction, so we use this counter.
 di: u8,
 /// Flag that holds the running status.
 running: bool,
 /// CPU halted.
 halted: bool,
 /// T-states, basic unit of time, and 1:1 with the clock.
 t_cycles: u32,
 /// M-cycles, base unit for CPU instructions, and 1:4 with the clock.
 m_cycles: u32,
 /// The debug manager.
 debug: DebugManager,
 /// Print FPS every second.
 fps: bool,
 /// The event pump.
 event_pump: EventPump,
}

The registers are implemented in a struct, which is trivial:

/// # Registers
/// We have 7 1-bit registers (`a`, `b`, `c`, `d`, `e`, `h`, `l`),
/// which can be accessed individually, or together as 16 bits,
/// in the combinations `af`, `bc`, `de` and `hl`.
/// We also have the flags register, `f`, whose 4 most significant bits are the flags
/// zero `z`, subtraction `n`, half-carry `h` and carry `c`.
/// Additionally, we have two 16-bit special registers, the stack pointer `sp`, and
/// the program counter `pc`.
pub struct Registers {
 pub a: u8,
 pub b: u8,
 pub c: u8,
 pub d: u8,
 pub e: u8,
 /// Flags register.
 /// - Bit 7: `z`, zero flag
 /// - Bit 6: `n`, subtraction flag (BCD)
 /// - Bit 5: `h`, half-carry flag (BCD)
 /// - Bit 4: `c`, carry flag
 pub f: u8,
 pub h: u8,
 pub l: u8,
 /// Stack pointer register.
 pub sp: u16,
 /// Program counter register.
 pub pc: u16,
}

This struct is complented with handy methods to read and write the registers, either individually or as 16-bit combinations. As you can see, the code is very well documented.

The instructions live in an enum, as do their operands. The instructions enum contains the 256 8-bit instructions, and the 256 16-bit instructions prefixed by 0xCB.

/// # Instructions
/// This enum contains all supported instructions.
/// Instructions that can act with both r8 and r16 registers are first named
/// by default in their r8 version. The r16 versions add a `16` at the end of the name.
#[derive(Debug)]
pub enum Instruction {
 // NoOp.
 NOP(),
 // Stop.
 STOP(),
 // Halt.
 HALT(),

 // Jump HL.
 JPHL(),
 // Jump.
 JP(CC),
 // Relative jumps.
 JR(CC),

 // Push.
 PUSH(R16EXT),
 // Pop.
 POP(R16EXT),

 // Load.
 LD(R8),
 LDcp(R8, R8),
 LD16(R16),
 [...]
}

/// Enumerates the r8 registers.
#[derive(Debug)]
pub enum R8 {
 A,
 B,
 C,
 D,
 E,
 H,
 L,
 HL,
}

/// Enumerates the r16 registers.
#[derive(Debug)]
pub enum R16 {
 BC,
 DE,
 HL,
 SP,
}

/// Enumerates r16 registers for POP and PUSH, which contain AF.
#[derive(Debug)]
pub enum R16EXT {
 BC,
 DE,
 HL,
 AF,
}

/// Enumerates the R16 registers to be used in (some) load operations.
#[derive(Debug)]
pub enum R16LD {
 BC,
 DE,
 HLp,
 HLm,
 A8,
 C,
 A16,
}

/// Enumerates jump conditions, mostly flags.
#[derive(Debug)]
pub enum CC {
 NONE,
 NZ,
 Z,
 NC,
 C,
}

/// RST's target address, divided by 8.
#[derive(Debug)]
pub enum TGT3 {
 T0,
 T1,
 T2,
 T3,
 T4,
 T5,
 T6,
 T7,
}

And then, I have a from_byte(byte: u8) method that matches opcodes to their actual enum values:

impl Instruction {
 /// Construct an instruction from its byte representation.
 pub fn from_byte(byte: u8) -> Option<Instruction> {
 match byte {
 // NOP
 0x00 => Some(Instruction::NOP()),
 // STOP
 0x10 => Some(Instruction::STOP()),

 // LD (r16)
 0x01 => Some(Instruction::LD16(R16::BC)),
 0x11 => Some(Instruction::LD16(R16::DE)),
 0x21 => Some(Instruction::LD16(R16::HL)),
 0x31 => Some(Instruction::LD16(R16::SP)),

 [...]

 // Undocumented OPCODE
 _ => {
 println!(
 "{}: Unimplemented instruction {:#02x}, treating as NOP",
 "WARN".yellow(),
 byte
 );
 Some(Instruction::NOP())
 }
 }
 }

I also have a from_byte_0xcb(byte: u8) method used in the actual implementation to decode the 16-bit opcodes.

Actually implementing the CPU meant building a match with cases for ever single one of the 512 instructions. Each instruction updates the PC, manipulates registers or memory, and sets flags according to specific rules. Some instructions were very straightforward to implement; others, like DAA, required careful consideration of edge cases and bit manipulation. The implementation structure is quite straightforward:

 /// Execute a single instruction, and returns the number of cycles it takes.
 fn execute(&mut self, run_instr: RunInstr, opcode: u8) -> u8 {
 // Actually execute the instruction.
 match run_instr.instr {
 // NOP: no operation.
 Instruction::NOP() => 1,
 // STOP
 Instruction::STOP() => {
 self.stop();
 1
 }
 // HALT
 Instruction::HALT() => {
 // HALT only halts if IME is enabled.
 if self.ime {
 self.halt();
 }
 // Otherwise, HALT acts like NOP.
 1
 }

 // LD r16
 Instruction::LD16(r16) => match r16 {
 R16::BC => {
 let nw = self.read16();
 self.registers.set_bc(nw);
 3
 }
 R16::DE => {
 let nw = self.read16();
 self.registers.set_de(nw);
 3
 }
 R16::HL => {
 let nw = self.read16();
 self.registers.set_hl(nw);
 3
 }
 R16::SP => {
 self.registers.sp = self.read16();
 3
 }
 },

 [...]

 // OPCODE 16-bit (0xCB).
 Instruction::OPCODE16() => {
 // Read next byte, construct instruction, execute 0xCB instruction.
 let opcode0xcb = self.read8();
 let instr0xcb = Instruction::from_byte_0xcb(opcode0xcb);
 let msg = format!("Incorrect 0xCB opcode: {:#04X}", opcode0xcb);
 self.execute_0xcb(instr0xcb.expect(&msg), opcode0xcb)
 }

 // Never should happen.
 _ => panic!("Instruction is not implemented: {:#04X}", opcode),
 }
 }

And, of course, there is an analogous execute_0xcb(...) method that contains the implementation for all the 16-bit opcodes.

The CPU doesn’t run in isolation and needs to be cycle-accurate. Each instruction takes a specific number of machine cycles (M-cycles), typically 1-6, and the rest of the system needs to advance accordingly. Getting timing right is quite important, because the graphics hardware, audio, and even some games depend on precise cycle counts. This is why execute(...) -> u8 returns an unsigned integer. This is the number of cycles that the instruction takes. The main loop handles the input events, and then runs as many cycles per frame as necessary.

 while self.running {
 let frame_start_time = std::time::Instant::now();

 // Handle input events.
 self.handle_events();

 let mut cycles_this_frame: usize = 0;
 while cycles_this_frame < constants::CYCLES_PER_FRAME {
 let (t, m) = self.machine_cycle();
 self.m_cycles += m;
 self.t_cycles += t;
 cycles_this_frame += t as usize;

 // Fill the LCD buffer if needed.
 self.display.render_lcd(&mut self.memory);
 }

 // Flush and present the frame.
 self.display.canvas.flush();
 self.display.present();

 frame_count += 1;

 // Sleep for the remaining time in the frame.
 self.sleep_next_frame(frame_start_time);
 }
 }

The machine cycle runs a single CPU cycle. A CPU cycle does the following:

Read next opcode from the memory address pointed by PC
Increment PC
Decode the instruction
Run the instruction – execute(...)
Run a memory cycle
Update the IME and handle interrupts

The code looks like this:

 /// Main loop of the machine.
 fn cycle(&mut self) -> u8 {
 // Fetch next instruction, and parse it.
 let opcode = self.read8();
 let run_instr = RunInstr::new(opcode, &self.memory, &self.registers);

 // Execute the instruction.
 self.execute(run_instr, opcode)
 }

 /// Run a machine cycle.
 fn machine_cycle(&mut self) -> (u32, u32, bool) {
 // CPU instruction.
 // One machine cycle (M-cycle) is 4 clock cycles.
 let mut m_cycles = if self.running {
 if self.halted {
 // CPU is halted, don't execute instructions but still consume 1 M-cycle.
 1
 } else {
 // Run next CPU instruction.
 self.cycle() as u32
 }
 } else {
 // NOOP instruction.
 1
 };
 let mut t_cycles = m_cycles * 4;

 // Memory cycle.
 if m_cycles > 0 {
 // Memory cycle.
 let (_, r) = self.memory.cycle(t_cycles);
 if !r {
 // Close down.
 self.running = false;
 return (0, 0, false);
 }
 } else {
 t_cycles = 0;
 m_cycles = 0;
 }

 // Update IME.
 self.ime_update();

 // Handle interrupts if necessary.
 let interrupt_m_cycles = self.interrupt_handling();

 // Return.
 if interrupt_m_cycles > 0 {
 (interrupt_m_cycles * 4, interrupt_m_cycles, true)
 } else {
 (t_cycles, m_cycles, true)
 }
 }

We’ll talk about the memory cycle when we reach the memory section. For now, let’s look at interrupts, as it was an interesting challenge. The Game Boy supports five interrupt types: V-Blank, LCD STAT, Timer, Serial, and Joypad. The CPU checks for pending interrupts after each instruction, and if interrupts are enabled and one is pending, it pushes the PC onto the stack and jumps to the corresponding interrupt vector. Implementing the IME flag and its one-instruction delay after the EI instruction was a subtle detail that broke several games until I got it right. This is handled in ime_update() and interrupt_handling():

 /// Updates the IME (Interrupt Master Enable) flag.
 /// This is necessary because the effect of the EI and DI instructions
 /// is delayed by one instruction.
 fn ime_update(&mut self) {
 self.di = match self.di {
 2 => 1,
 1 => {
 self.ime = false;
 0
 }
 _ => 0,
 };
 self.ei = match self.ei {
 2 => 1,
 1 => {
 self.ime = true;
 0
 }
 _ => 0,
 };
 }

 /// Interrupt handling. The IF bit corresponding to this interrupt, and the IME flag
 /// are reset by the CUP. IF acknowledges the interrupt, and IME prevents any other
 /// interrupts from being handled until re-enabled (with RETI).
 /// In this case, the corresponding interrupt handler is called by pushing the PC
 /// to the stack, and then setting it to the address of the interrupt handler.
 fn interrupt_handling(&mut self) -> u32 {
 let pending = self.memory.ie & self.memory.iff;

 // Wake from HALT if any interrupt is pending (even if IME is disabled).
 if pending != 0 {
 if self.halted {
 self.halted = false;
 }
 }

 // If no interrupts are pending OR IME is disabled, return.
 if !self.ime || pending == 0 {
 return 0;
 }

 // Reset IME.
 self.ime = false;

 // IE and IF have the following format:
 //
 // | 7 6 5 | 4 | 3 | 2 | 1 | 0 |
 // | 1 | Joypad | Serial | Timer | LCD | VBlank |
 //
 for i in 0..5 {
 let bit = 1 << i;
 if pending & bit != 0 {
 self.memory.iff &= !bit;
 let pc = self.registers.pc;
 self.push_stack(pc);
 self.registers.pc = match bit {
 0x01 => 0x0040,
 0x02 => 0x0048,
 0x04 => 0x0050,
 0x08 => 0x0058,
 0x10 => 0x0060,
 _ => panic!("Unknown interrupt bit {}", bit),
 };
 break;
 }
 }

 5 // 5 M-cycles = 20 T-cycles
 }

Memory

The Game Boy’s 16-bit address space is divided into distinct regions, each serving a specific purpose:

0x0000-0x3FFF: ROM bank 0 (16 KB, fixed)
0x4000-0x7FFF: ROM bank 1-N (16 KB, switchable via MBCs, more on that later)
0x8000-0x9FFF: Video RAM (8 KB)
0xA000-0xBFFF: External RAM (8 KB, cartridge)
0xC000-0xDFFF: Work RAM (8 KB)
0xE000-0xFDFF: Echo RAM (mirror of 0xC000-0xDDFF)
0xFE00-0xFE9F: OAM (sprite data)
0xFF00-0xFF7F: I/O registers
0xFF80-0xFFFE: High RAM (127 bytes)
0xFFFF: IE register

Implementing memory access required a dispatcher that routes reads and writes to the appropriate hardware component. Reading from VRAM accesses the PPU’s memory, writing to 0xFF00-0xFF7F triggers side effects in various subsystems, and accessing cartridge ROM or RAM goes through the mapper.

The memory struct contains the Work RAM, the High RAM, I/O registers, the IF and IE, a reference to the PPU, a reference to the APU, a reference to the cartridge, the timer, and the joypad.

/// # Memory
/// The Game Boy uses a 2-byte address space (0x0000 to 0xFFFF) to map the different
/// types of memory (RAM, VRAM, Cartridge memory, etc.)
pub struct Memory<'a> {
 /// Work RAM.
 pub wram: [u8; constants::WRAM_SIZE],
 // High RAM.
 pub hram: [u8; constants::HRAM_SIZE],
 // I/O registers.
 pub io: [u8; constants::IO_SIZE],
 // IF: interrupt flag.
 pub iff: u8,
 // IE flag: interrupt enable.
 pub ie: u8,
 // Cartridge reference.
 pub cart: &'a mut Cartridge,
 /// The PPU, Picture Processing Unit.
 pub ppu: PPU,
 /// The timer.
 pub timer: Timer,
 /// The joypad.
 pub joypad: Joypad,
 /// The APU, Audio Processing Unit.
 pub apu: APU,
}

The actual mapping is implemented in the read() and write() operations. The address is matched to the specific range in the address space and forwarded to the appropriate component. For instance, the read method:

 /// Read a byte of memory at the given `address`.
 pub fn read8(&self, address: u16) -> u8 {
 match address {
 0x0000..=0x3FFF => {
 // 16kB bank #0 (cartridge).
 self.cart.read(address)
 }
 0x4000..=0x7FFF => {
 // 16kB switchable ROM bank (cartridge).
 self.cart.read(address)
 }
 0x8000..=0x9FFF => {
 // VRAM.
 self.ppu.read(address)
 }
 0xA000..=0xBFFF => {
 // 8kB switchable RAM bank (cartridge).
 self.cart.read_ram(address)
 }
 [...]
 }
 }

Memory-mapped I/O registers control everything from LCD modes to audio channels. Some registers are read-only, some write-only, and some have bits that behave differently on read versus write. The DMA register at 0xFF46 is particularly interesting. Writing to it triggers a transfer of 160 bytes from ROM or RAM to OAM, during which the CPU should only access High RAM. Getting DMA timing correct was essential for sprite-heavy games.

The PPU

The Picture Processing Unit is what puts pixels on the screen. It generates a 160×144 display running at around 60 Hz. Getting it working meant understanding scanlines, timing, and how backgrounds, windows, and sprites all come together.

The PPU works one scanline at a time. Each scanline takes 456 T-cycles (dots), and a full frame has 154 scanlines: 144 visible ones and 10 during V-Blank. While rendering a scanline, the PPU goes through different modes:

Mode 2 (OAM Search): Scanning which sprites will be visible on this scanline (80 dots)
Mode 3 (Pixel Transfer): Actually drawing the pixels (variable, ~172 dots)
Mode 0 (H-Blank): Horizontal blank, waiting for the scanline to finish (~204 dots)
Mode 1 (V-Blank): Vertical blank, screen done rendering (10 scanlines × 456 dots)

During V-Blank, games usually update their state and prepare for the next frame. My implementation transitions between modes based on the current scanline and dot count:

let new_mode = if self.ly >= 144 {
 1 // VBlank.
} else {
 match self.fdot {
 0..=80 => 2, // OAM Scan.
 81..=252 => 3, // Drawing.
 _ => 0, // HBlank.
 }
};

The background is made up of 8x8 pixel tiles stored in VRAM. Each tile uses 2 bits per pixel, giving you four shades of gray (or colors if you apply a palette). There are two tile data areas (at 0x8000 and 0x8800) and two tile maps that tell you which tiles go where. The LCDC register controls which areas to use, and you can think of it as configuring how the PPU interprets the data in VRAM.

Fetching tile data means reading two bytes per tile row and combining them to get the color IDs:

let tile_addr = tile_addr_base + (line * 2);
let low_byte = self.read(tile_addr);
let high_byte = self.read(tile_addr + 1);

let mut pixels = [0u8; 8];
for i in 0..8 {
 let color_id = ((high_byte >> (7 - i)) & 0x1) << 1 | ((low_byte >> (7 - i)) & 0x1);
 pixels[i] = color_id;
}

Windows work similarly to backgrounds but ignore scrolling. They’re positioned absolutely on screen using WX and WY registers. My implementation tracks an internal window line counter (WLY) that only increments when the window actually renders on a scanline, which is important for games that turn the window on and off mid-frame.

Sprites (or objects) are 8x8 or 8x16 pixels that sit on top of the background. The OAM holds up to 40 sprite definitions, each with position, tile index, and attributes like palette selection, flip flags, and priority. The hardware can only show 10 sprites per scanline, and when more than 10 overlap, priority goes to sprites with lower X coordinates first, then lower OAM index:

// Game Boy priority: X asc, then OAM index asc.
sprites.sort_by(|a, b| {
 if a.x != b.x {
 a.x.cmp(&b.x)
 } else {
 a.oam_index.cmp(&b.oam_index)
 }
});

One of the trickier parts was getting sprite priority right. The priority bit in the sprite attributes controls whether a sprite appears above all background pixels or only above background color 0. I store the background color IDs in a separate buffer as I render the background, then check against those values when drawing sprites:

let bg_color_id = self.priorities[self.ly as usize * constants::DISPLAY_WIDTH + x_pos];

let obj_behind_bg = (sprite.attributes & 0x80) != 0;
let sprite_has_priority = if obj_behind_bg {
 // Priority bit set: only draw sprite over BG color 0.
 bg_color_id == 0
} else {
 // Priority bit clear: always draw sprite (over any BG color).
 true
};

This lets me respect the priority rules properly without having to re-read background data.

I cache tile data while rendering each scanline to avoid repeatedly decoding the same tiles. For backgrounds and windows, I cache up to 32 tiles (one row). For sprites, I use a HashMap keyed by tile ID, line number, and flip flags.

The PPU’s timing had to be cycle-accurate. Each frame needs exactly 70224 T-cycles (154 scanlines × 456 dots). The main loop advances the PPU by the number of cycles the CPU just executed, and the PPU updates its mode and LY register (current scanline) accordingly:

pub fn cycle(&mut self, t_cycles: u32) {
 if !self.is_ppu_enabled() {
 return;
 }

 self.fdot += t_cycles;

 // Update mode if it changed.
 if new_mode != self.mode {
 self.update_mode(new_mode);
 }

 // Handle transition to next scanline.
 if self.fdot >= 456 {
 self.fdot -= 456;
 self.ly = (self.ly + 1) % 154;
 }
}

Getting the timing right was kind of important because some games rely on precise V-Blank and H-Blank timing. Otherwise, you get visual glitches, and good luck debugging them.

In order to test my PPU implementation, I turned to the DMG Acid2 test. This is a rather exhaustive PPU test in the form of a ROM. If you see a happy face with HELLO WORLD! above, and dmg-acid2 by Matt Currie below, you’re good. In my case, the right chin was missing. I spent an ginormous amount of time trying to get it to render properly. This is what I was getting:

Oh no, the right chin is missing!

I was so close. The README has an entry that looks exactly like it. The right chin is rendered using the window, and the window tile map, so I kept looking at the window code. It was a pain, and for the life of me, I could not find the issue even though I looked incessantly. In the end, I fed my code to an LLM and it found the error right away: In my read and write VRAM operations, I had WX and WY backwards!

 // This is NOT correct!
 // WY.
 0xFF4A => self.wx,
 // WX.
 0xFF4B => self.wy,

This was frustrating, but it really goes to show what random side effects such a stupid mistake can have.

After countless hours, the stupid bug was mended and Acid2 was good!

Now that my PPU was more or less feature-complete, I tested it with other games. Seeing the first sprite render correctly, or watching the background scroll smoothly in Mario for the first time made it worth it. The PPU is obviously the most (only?) visible part of the emulator, so getting it to a state where it was able to run all the games I wanted to run felt good.

At the very end, when the graphics were already in an excellent shape, I implemented multiple color palettes. The original Game Boy was monochrome, but you can map those four gray shades to any colors you want. IIRC, the Super Game Boy did something similar, and it allowed you to create your own too! I included a few classic palettes (DMG Classic, Pocket, etc.) and some more creative ones. You can cycle through them while playing with p, which makes the games look fresh.

Some of the palettes in Play Kid.

At this point, I could play lots of games, but I had no audio. This had to be mended.

The APU

I consider the Audio Processing Unit (APU) to be one of the most boring parts of this project. It synthesizes sound through four channels: two pulse wave channels (square waves), one wave channel (custom waveform), and one noise channel (pseudo-random). Each channel has its own registers for frequency, volume, envelope, and duty cycle.

The APU is fully synchronized with the CPU and the PPU, and tus runs at 4.19 MHz. This is the master clock that we’ve seen previously. However, the APU doesn’t do everything at 4.19 MHz. Internally, it uses a frame sequencer that divides this master clock down to 512 Hz to manage high-level audio events:

Length Counters: Clocked at 256 Hz (every 2nd step).
Frequency Sweep: Clocked at 128 Hz (every 4th step).
Volume Envelopes: Clocked at 64 Hz (every 8th step).

It steps through eight different phases, some updating length counters, some update the sweep function, and some updating volume envelopes. My implementation tracks this with a frame timer that increments with each T-cycle and runs at 512 Hz (4194304 Hz / 8192 = 512 Hz):

self.frame_timer += t_cycles;
if self.frame_timer >= 8192 {
 // 512Hz is 4.19 Mhz / 8192.
 self.frame_timer -= 8192;

 // Phases from 0 to 7.
 match self.frame_sequencer {
 0 | 2 | 4 | 6 => {
 self.step_length();
 if self.frame_sequencer == 2 || self.frame_sequencer == 6 {
 self.step_sweep();
 }
 }
 7 => {
 self.step_envelope_ch1();
 self.step_envelope_ch2();
 self.step_envelope_ch4();
 }
 _ => {}
 }
 self.frame_sequencer = (self.frame_sequencer + 1) % 8;
}

Each pulse channel (1 and 2) generates square waves with four different duty cycles. The duty cycle controls the waveform shape and is stored as patterns:

let duty_idx = (self.read(0xFF11) >> 6) as usize;
let patterns = [
 [0, 0, 0, 0, 0, 0, 0, 1], // 12.5%
 [1, 0, 0, 0, 0, 0, 0, 1], // 25%
 [1, 0, 0, 0, 0, 1, 1, 1], // 50%
 [0, 1, 1, 1, 1, 1, 1, 0], // 75%
];

let signal = patterns[duty_idx][self.ch1_duty_step];
let volume_multiplier = (self.ch1_volume as f32 / 15.0) * 0.05;

The wave channel (channel 3) is different. It plays back samples from a 32-nibble waveform stored in Wave RAM. Games can write custom waveforms here, which makes this channel super flexible:

// Get 4-bit sample from Wave RAM (32 samples total, 2 per byte).
let byte = self.wave_ram[self.ch3_sample_idx / 2];
let mut sample = if self.ch3_sample_idx % 2 == 0 {
 byte >> 4 // High nibble.
} else {
 byte & 0x0F // Low nibble.
};

// Volume shift based on NR32 register.
sample >>= volume_shift;

The noise channel (channel 4) uses a LFSR to generate pseudo-random noise. The LFSR is a 15-bit shift register that XORs bits 0 and 1, then shifts the result back in:

let bit0 = self.ch4_lfsr & 0x01;
let bit1 = (self.ch4_lfsr >> 1) & 0x01;
let result = bit0 ^ bit1;

self.ch4_lfsr = (self.ch4_lfsr >> 1) | (result << 14);
if (nr43 & 0x08) != 0 {
 // Short mode (7-bit)
 self.ch4_lfsr = (self.ch4_lfsr & !(1 << 6)) | (result << 6);
}

This creates the characteristic noise used for drums, explosions, and other effects.

The sweep function on channel 1 was particularly tricky. It modulates the frequency over time by shifting the current frequency right and either adding or subtracting the result. The overflow check happens both when sweep is triggered and when calculating the new frequency:

fn calculate_sweep_freq(&mut self) -> u16 {
 let nr10 = self.read(0xFF10);
 let sweep_step = nr10 & 0x07;
 let descending = (nr10 & 0x08) != 0;

 let delta = self.ch1_sweep_shadow_freq >> sweep_step;
 let new_freq = if descending {
 self.ch1_sweep_shadow_freq.saturating_sub(delta)
 } else {
 self.ch1_sweep_shadow_freq + delta
 };

 // Overflow check.
 if new_freq > 2047 {
 self.ch1_enabled = false;
 }

 new_freq
}

Getting the timing and overflow behavior right required reading Pan Docs extensively, and I also looked at RBoy’s implementation.

Sound samples are generated at 44100 Hz and mixed into stereo output based on the NR51 register, which controls left/right panning for each channel. SDL2’s audio queue system fit naturally with the APU design. I generate samples as the emulator runs and queue them for SDL to play back:

let (l_sample, r_sample) = self.generate_sample();
self.buffer.push(l_sample);
self.buffer.push(r_sample);

if self.buffer.len() >= 2048 {
 // Throttle if queue is getting too full
 while self.device.size() > 8192 {
 std::thread::sleep(std::time::Duration::from_millis(1));
 }
 self.device.queue_audio(&self.buffer).unwrap();
 self.buffer.clear();
}

Keeping audio and video synchronized without drift required careful management of cycle counts and buffer sizes. The throttling mechanism prevents the emulator from running too fast when the audio queue backs up.

I implemented channel 1 first, and then ran Super Mario Land. I could hear some sounds (jumps, parts of the music), but the experience was underwhelming. Since all channels follow a similar structure, I had channel 2 working in no time. Now I could hear the iconic melody of the first stage, all the jumps, coins, and enemies. I implemented the other channels soon after.

Interestingly, in Super Mario Land, the music is mostly in channels 2 and 3. Channel 1 is for things like jumps, coins, and some high-pitched parts of melodies. Channel 4 is used in the random noise that some enemies produce when they die (like the bomb turtles), and some percussion beats in the background.

I still have no idea whether my APU contains any bugs, or if it sounds exactly as it’s supposed to. I have no way to tell. To my ears, it sounds amazing.

Cartridges

Most Game Boy cartridges aren’t just ROM chips. They contain MBCs that expand the addressable memory beyond the CPU’s 64 KB limit. Different games use different MBCs, each with its own banking scheme.

Every cartridge has a header header that contains essential information about the game, stored in the first few bytes of the cartridge’s ROM. This information helps the system identify and load the game properly. The header includes:

Logo – the first 48 bytes (0x00-0x2F) contain the Nintendo logo, ensuring the authenticity of the cartridge.
Title – the next 16 bytes (0x34-0x43) store the 8-character game title, which is displayed when the game is loaded.
Game Boy/GBC Flag – the byte at 0x143 indicates whether the game is for the original Game Boy (0x00), or a Game Boy Color-compatible game (0x80).
Cartridge Type – the byte at 0x147 specifies the cartridge type, such as ROM-only or with memory banks. Here is a non-exhaustive list of common values:
- 0x00 – ROM only
- 0x01 – MBC1
- 0x02 – MBC1+RAM
- 0x03 – MBC1+RAM+BATTERY
- 0x05 – MBC2
- 0x06 – MBC2+BATTERY
- 0x11 – MBC3
- 0x12 – MBC3+RAM
- 0x13 – MBC3+RAM+BATTERY
ROM Size – the byte at 0x148 specifies the ROM size in powers of 2. For example, 0x00 is 32KB, 0x01 is 64KB, 0x02 is 128KB, etc.
RAM Size – the byte at 0x149 specifies the RAM size in kilobytes. For example, 0x00 means no RAM, 0x03 means 32KB, 0x04 means 128KB, etc.
Destination Code – located at 0x14A, this byte indicates the region the game is intended for. 0x00 is for Japan, 0x01 for Overseas.
Licensee Code – the byte at 0x14B contains a code identifying the licensed company that produced the game.
Mask ROM Version – the byte at 0x14C is the version number of the game’s mask ROM (usually set to 0x00).
Complement Checksum – the byte at 0x14D is used for the checksum calculation to verify the integrity of the ROM data.
Global Checksum – the last 2 bytes (0x14E-0x14F) store a checksum value that ensures the data integrity of the entire ROM.

This header allows the Game Boy to load and display the game properly by checking for compatibility and loading the correct resources for the specified hardware.

When it comes to MBCs, I implemented ROM-only and MBC1/2/3 into Play Kid.

ROM-only

ROM-only cartridges are the simplest, with no banking at all. These are limited to 32 KB and include simple games like Tetris. No mapper logic needed, just direct reads from the ROM data.

Tetris and Dr. Mario use ROM-only cartridges.

MBC1

MBC1 is the most common mapper, supporting up to 2 MB of ROM and 32 KB of RAM. It uses banking to switch between ROM and RAM banks by writing to specific address ranges. Understanding that writes to ROM addresses don’t actually write to ROM but instead configure the mapper took some mental adjustment.

MBC1 has two banking modes controlled by a mode register. In ROM banking mode (mode 0), you get access to more ROM banks but only one RAM bank. In RAM banking mode (mode 1), you can switch between multiple RAM banks but lose access to the upper ROM banks:

pub fn write(&mut self, address: u16, value: u8) {
 match address {
 // RAM enable.
 0x0000..=0x1FFF => {
 self.ram_enabled = (value & 0x0F) == 0x0A;
 }
 // ROM bank number (lower 5 bits).
 0x2000..=0x3FFF => {
 let mut bank = value as usize & 0x1F;
 if bank == 0 {
 bank = 1; // Bank 0 becomes bank 1.
 }
 self.rom_bank = (self.rom_bank & 0x60) | bank;
 }
 // RAM bank or upper ROM bank bits.
 0x4000..=0x5FFF => {
 let bits = value as usize & 0x03;
 if self.banking_mode == 0 {
 self.rom_bank = (bits << 5) | (self.rom_bank & 0x1F);
 } else {
 self.ram_bank = bits;
 }
 }
 // Banking mode select.
 0x6000..=0x7FFF => {
 self.banking_mode = value & 0x01;
 }
 _ => {}
 }
}

Super Mario Land (1 and 2), Wario Land, Kirby’s Dream Land, and The Amazing Spider-Man all use MBC1.

MBC2

MBC2 has a quirky design. It includes 512×4 bits of built-in RAM, and bit 8 of the write address determines whether you’re configuring ROM banking or RAM enable. Both operations share the same address range (0x0000-0x1FFF):

match addr {
 0x0000..=0x1FFF => {
 if (addr >> 8) & 0x01 == 0 {
 // Bit 8 is 0: RAM enable.
 self.ram_enabled = (val & 0x0F) == 0x0A;
 } else {
 // Bit 8 is 1: ROM bank select.
 self.rom_bank = val & 0x0F;
 if self.rom_bank == 0 {
 self.rom_bank = 1;
 }
 }
 }
 // ...
}

Since MBC2’s RAM is only 4 bits per byte, reads return the value OR’d with 0xF0, and writes mask to the lower 4 bits.

I think I have not tested any game that uses MBC2.

MBC3

MBC3 adds a real-time clock (RTC) alongside standard ROM/RAM banking. The RTC includes seconds, minutes, hours, and days counters that can be accessed by setting the RAM bank register to values 0x08-0x0C. I understand that some games use the RTC to track time-based events, but I have not tested this:

pub fn read_ram(&self, address: u16) -> u8 {
 if !self.ram_enabled {
 return 0xFF;
 }

 match self.ram_bank {
 0x00..=0x03 => {
 // Standard RAM banks.
 let offset = (self.ram_bank * 0x2000) + (address as usize - 0xA000);
 self.ram[offset % self.ram.len()]
 }
 0x08 => self.rtc_seconds,
 0x09 => self.rtc_minutes,
 0x0A => self.rtc_hours,
 0x0B => self.rtc_days_low,
 0x0C => self.rtc_days_high,
 _ => 0xFF,
 }
}

The RTC is latched by writing 0 then 1 to address range 0x6000-0x7FFF. This copies the current time into the registers so the game can read a consistent value. My implementation updates the RTC with system time:

fn update_rtc(&mut self) {
 let now = SystemTime::now()
 .duration_since(UNIX_EPOCH)
 .unwrap()
 .as_secs();

 self.rtc_seconds = (now % 60) as u8;
 self.rtc_minutes = ((now / 60) % 60) as u8;
 self.rtc_hours = ((now / 3600) % 24) as u8;

 let days = (now / 86400) as u16;
 self.rtc_days_low = (days & 0xFF) as u8;

 // Day MSB and carry flag.
 let day_msb = ((days >> 8) & 0x01) as u8;
 self.rtc_days_high = (self.rtc_days_high & 0xFE) | day_msb;

 if days > 511 {
 self.rtc_days_high |= 0x80; // Set overflow flag.
 }
}

Pokémon and Wario Land II use MBC3.

Finally, I implemented battery-backed SRAM. It meant saving the contents of cartridge RAM to .sav files when the emulator exits and loading them on startup, if they exist. You can also trigger the save operation at any time with w. I get the RAM from the cartridge, and then persist it (if not empty):

 /// Save SRAM of current cartridge to `.sav` file.
 pub fn save_sram(&self, rom_path: &str) {
 let save_path = Path::new(rom_path).with_extension("sav");

 // Only save if the mapper actually has RAM.
 let ram_data = match &self.cart_type {
 CartridgeType::MBC1(mbc) => &mbc.get_ram(),
 CartridgeType::MBC2(mbc) => &mbc.get_ram(),
 CartridgeType::MBC3(mbc) => &mbc.get_ram(),
 _ => return,
 };

 if !ram_data.is_empty() || matches!(self.cart_type, CartridgeType::MBC2(_)) {
 if let Ok(mut file) = File::create(&save_path) {
 let _ = file.write_all(ram_data);
 println!(
 "{}: Save data written to disk: {}",
 "WR".magenta(),
 save_path.display()
 );
 }
 }
 }

The load operation checks for the file, loads it if it exists, and passes it to each MBC’s set_ram() method.

This feature is essential for saving progress in many games. Currently, the .sav files are written to the same directory where the ROM file is, with the same name and sav extension.

UI and debug mode

In order to help me debug the emulator as development progressed, I implemented a terminal-based debug interface that let me look under the hood and inspect the internal state of the machine. This is implemented in the src/debugcli.rs file in the repository, but it is now unused.

When the emulator was already working quite well, I implemented a GUI debug mode with a custom immediate-mode UI library built on top of SDL2 rendering. You can enable it at any time with d.

The UI library supports horizontal and vertical layouts, labels, buttons, and text fields. Enough to build a useful debug interface without pulling in a fat dependency. It’s minimal but effective. Writing it was not trivial. I knew that UI libraries are complex and take a lot of work to get right, and this little spin-off sub-project only confirmed this knowledge.

Super Mario Land 2 running in Play Kid with the debug UI.

The debug mode features:

Current instruction, operand, and PC
Opcode
CPU status (either RUNNING or HALTED)
M- and T-cycles
CPU registers and flags
PPU state (LCDC, STAT, LYC, LY, LX)
Joypad state for all inputs (U, D, L, R, A, B, START, SELECT)
Breakpoints

You can step through execution one instruction (F6) or one scanline (F7) at a time, pause and continue (F9), and set breakpoints at specific addresses. You can also reset the CPU with r. I also included an FPS counter (f). You can enable and disable the debug mode at any time with d.

One quality-of-life feature I’m proud of is automatic DPI scaling. On high-DPI displays, the UI scales appropriately so everything remains readable without manual configuration. This was a pain to handle manually. The emulator also respects the Game Boy’s 160:144 aspect ratio by letterboxing the display. It’s a small detail, but it ensures games look correct rather than stretched or squashed.

Future work

Here are some items I would like to do at some point:

Move away from SDL2 in favor of Rust-native libraries like winit+pixels+rodio for easy cross-compilation, and web. I actually have a branch where I implemented the APU using rodio
Implement more MBCs if I stumble upon games that need them.
Add Game Boy Color support. As I understand it, it is not too complicated.

Conclusions

Building Play Kid taught me more about computer architecture, bit manipulation, and systems programming than any tutorial could. Every component of the Game Boy (CPU, memory, graphics, audio) works in concert, and understanding their interactions required patience, testing, and a lot of debugging.

Rust was an excellent choice for this project. My CHIP-8 emulator was already written in Rust, so this was a safe choice for me. I think the language has been praised enough at this time, so I’ll spare this part. SDL2 provided a simple, cross-platform foundation for rendering, input, and audio without unnecessary complexity.

If you’re interested in emulation, I think the Game Boy is a solid choice as first project if you have previous programming experience. The documentation is excellent and the hardware is approachable.

Play Kid’s source code is on Codeberg and GitHub.

Google unkills JPEG XL?

Toni Sagrista Selles — Fri, 28 Nov 2025 00:00:00 +0000

I’ve written about JPEG XL in the past. First, I noted Google’s move to kill the format in Chromium in favor of the homegrown and inferior AVIF.¹² Then, I had a deeper look at the format, and visually compared JPEG XL with AVIF on a handful of images.

The latter post started with a quick support test:

“If you are browsing this page around 2023, chances are that your browser supports AVIF but does not support JPEG XL.”

Well, here we are at the end of 2025, and this very sentence still holds true. Unless you are one of the 17% of users using Safari³, or are adventurous enough to use a niche browser like Thorium, LibreWolf or the newer Zen Browser, chances are you see the AVIF banner in green and the JPEG XL image in black/red.

The good news is, this will change soon. In a dramatic turn of events, the Chromium team has reversed its Obsolete tag, and has decided to support the format in Blink (the engine behind Chrome/Chromium/Edge). Given Chrome’s position in the browser market share, I predict the format will become a de factor standard for images in the near future.

Let’s recap

I’ve been following JPEG XL since its experimental support in Blink. What started as a promising feature was quickly axed by the team in a bizarre and ridiculous manner. First, they asked the community for feedback on the format. Then, the community responded very positively. And I don’t only mean a couple of guys in their basement. Meta, Intel, Cloudinary, Adobe, ffmpeg, libvips, Krita, and many more. After that came the infamous comment:

da...@chromium.org da...@chromium.org

#85 Oct 31, 2022 12:34AM

Thank you everyone for your comments and feedback regarding JPEG XL. We will be removing the JPEG XL code and flag from Chromium for the following reasons:

Experimental flags and code should not remain indefinitely

There is not enough interest from the entire ecosystem to continue experimenting with JPEG XL

The new image format does not bring sufficient incremental benefits over existing formats to warrant enabling it by default

By removing the flag and the code in M110, it reduces the maintenance burden and allows us to focus on improving existing formats in Chrome

Yes, right, “not enough interest from the entire ecosystem”. Sure.

Anyway, following this comment, a steady stream of messages pointed out how wrong that was, from all the organizations mentioned above and many more. People were noticing in blog posts, videos, and social media interactions.

Strangely, the following few years have been pretty calm for JPEG XL. However, a few notable events did take place. First, the Firefox team showed interest in a JPEG XL Rust decoder, after describing their stance on the matter as “neutral”. They were concerned about the increased attack surface resulting from including the current 100K+ lines C++ libjxl reference decoder, even though most of those lines are testing code. In any case, they kind of requested a “memory-safe” decoder. This seems to have kick-started the Rust implementation, jxl-rs, from Google Research.

To top it off, a couple of weeks ago, the PDF Association announced their intent to adopt JPEG XL as a preferred image format in their PDF specification. The CTO of the PDF Association, Peter Wyatt, expressed their desire to include JPEG XL as the preferred format for HDR content in PDF files.⁴

Chromium’s new stance

All of this pressure exerted steadily over time made the Chromium team reconsider the format. They tried to kill it in favor of AVIF, but that hasn’t worked out. Rick Byers, on behalf of Chromium, made a comment in the Blink developers Google group about the team welcoming a performant and memory-safe JPEG XL decoder in Chromium. He stated that the change of stance was in light of the positive signs from the community we have exposed above (Safari support, Firefox updating their position, PDF, etc.). Quickly after that, the Chromium issue state was changed from Obsolete to Assigned.

About JPEG XL

This is great news for the format, and I believe it will give it the final push for mass adoption. The format is excellent for all kinds of purposes, and I’ll be adopting it pretty much instantly for this and the Gaia Sky website when support is shipped. Some of the features that make it superior to the competition are:

Lossless re-compression of JPEG images. This means you can re-compress your current JPEG library without losing information and benefit from a ~30% reduction in file size for free. This is a killer feature that no other format has.
Support for wide gamut and HDR.
Support for image sizes of up to 1,073,741,823x1,073,741,824. You won’t run out of image space anytime soon. AVIF is ridiculous in this aspect, capping at 8,193x4,320. WebP goes up to 16K², while the original 1992 JPEG supports 64K².
Maximum of 32 bits per channel. No other format (except for the defunct JPEG 2000) offers this.
Maximum of 4,099 channels. Most other formats support 4 or 5, with the exception of JPEG 2000, which supports 16,384.
JXL is super resilient to generation loss.⁵
JXL supports progressive decoding, which is essential for web delivery, IMO. WebP or HEIC have no such feature. Progressive decoding in AVIF was added a few years back.
Support for animation.
Support for alpha transparency.
Depth map support.

For a full codec feature breakdown, see Battle of the Codecs.

Conclusion

JPEG XL is the future of image formats. It checks all the right boxes, and it checks them well. Support in the overwhelmingly most popular browser engine is probably going to be a crucial stepping stone in the format’s path to stardom. I’m happy that the Chromium team reconsidered their inclusion, but I am sad that it took so long and so much pressure from the community to achieve it.

Motion trails in Gaia Sky

Toni Sagrista Selles — Sat, 14 Jun 2025 00:00:00 +0000

Astronomical scenes, especially those rendered interactively, often feature supraluminal camera motion over immense distances. Sometimes, these motions are rendered by applying trail effects to light-emitting objects to enhance the faster-than-light velocity sensation. Gaia Sky will get an implementation of motion trails in the next version (3.6.9). Motion trails are a visual effect that stretches stars, galaxies, and other light-emitting particles in the direction of the velocity vector of the camera, giving a sense of speed and enhancing the perception of motion through space. This technique is inspired by relativistic visualizations and classic star streak effects, but it is grounded in angular motion rather than raw velocity.

Vertex-based stretching of stars in supraluminal travel in Gaia Sky.

In this post, I describe the technical details that made implementing a performant, vertex-based solution into Gaia Sky possible.

Traditional approach

The typical way to render such an effect is through a post-processing motion blur pass, especially common in game engines and cinematic visualizations. The process is roughly the following:

Separate Render Targets — stars, particles, or other emissive elements are rendered to dedicated framebuffers (with color and velocity).
Velocity Buffers — each rendered object encodes its screen-space motion vector based on differences between current and previous frame transforms.
Post-Processing Blur Shader — a fullscreen pixel shader smears pixels along their velocity vector, creating blur or streaks. This can be implemented with linear sampling or more advanced temporal accumulation techniques.

This method applies uniformly to all rendered elements, works in screen space, typically using a pixel shader, and reuses the pre-existing motion blur infrastructure in engines. However, it requires additional render targets and history buffers, it is almost impossible to control and isolate per-object, and it does not handle transparencies or additive blending correctly. What can we do about it?

Our approach

In Gaia Sky, the effect is instead implemented directly in the vertex shader. Each object is a quad billboard oriented toward the camera. Trails are computed procedurally based on the screen-space velocity of the objects (difference in projected position over time steps), the stretch direction based on camera motion, and a distance-based fade-out.

This technique is simpler and has some nice bonuses:

It avoids history buffers or post-processing passes.
It is fully compatible with additive blending and transparency.
It scales well with hundreds of thousands of objects.
It is tunable based on distance, screen velocity, and apparent motion.

In short, this vertex-based approach integrates the effect tightly into Gaia Sky’s real-time astronomical renderer, improving performance, visual clarity, and configurability.

The shader

The code lives in the snippet that computes the billboard rotation, i.e., the part of the code that orients the quad to the camera to make it a billboard. This snippet uses the following inputs:

s_vert_pos: vec4 — vertex position.
s_obj_pos: vec3 — object position w.r.t. camera.
s_proj_view: mat4 — camera view-projection matrix.
s_size: float — quad size.
u_camUp: vec3 — world-space camera up vector.
u_camVel: vec3 — world-space camera velocity.
u_dt: float — delta time in seconds between frames.
u_uToMpc: float — conversion factor from internal units to Mpc.

The shader chunk computes the final vertex position and puts it into gpos: vec4. In the next subsections I break down the shader parts and explain each one separately.

0. Initial orientation

Here we just compute the orientation base of the quad/billboard, taking as reference the vector from the camera to the center of the object (s_obj_pos).

 s_obj = normalize(s_obj_pos);
 s_right = normalize(cross(u_camUp, s_obj));
 s_up = normalize(cross(s_obj, s_right));

1. Rotation and velocity projection

The quad is oriented toward the camera using a rotation quaternion. We then rotate the camera velocity into the quad’s local space to determine stretch direction.

vec4 s_quat = q_look_at(s_obj, s_up);
vec4 s_quat_inv = q_conj(s_quat);
vec3 local_pos = s_vert_pos.xyz;
vec3 centered_pos = local_pos;
vec3 local_cam_vel = qrot(s_quat_inv, u_camVel);

2. Estimate screen-space movement

We simulate where the object would be next frame by offsetting it by the camera velocity u_camVel using the frame time u_dt. This gives a screen-space motion estimate in NDC (Normalized Device Coordinates). Note the division by the $w$ component of clip coordinates, needed to convert to NDC.

vec3 obj_world = s_obj_pos;
vec3 obj_next = obj_world - u_camVel * u_dt;

vec4 clip_now = s_proj_view * vec4(obj_world, 1.0);
vec4 clip_next = s_proj_view * vec4(obj_next, 1.0);

vec2 ndc_now = clip_now.xy / clip_now.w;
vec2 ndc_next = clip_next.xy / clip_next.w;
float screenVel = length(ndc_now - ndc_next);

3. Calculate trail stretch amount

The stretch magnitude is driven by the screen-space velocity screenVel, shaped using a nonlinear power function and clamped. The actual parameters need to be tuned to every scene.

vec3 stretch_dir = normalize(vec2(local_cam_vel.x, local_cam_vel.y).xyx);
float stretch = pow(screenVel * 450.0, 2.0);
stretch = clamp(stretch, 0.0, 6.0);

4. Distance-based fadeout

To avoid unnecessary GPU load from distant galaxies (which barely move visually), the effect fades out smoothly between 30 and 50 Mpc. This only ever affects very distant, high-z galaxies due to the distances involved in the thresholds.

// Object distance in Mpc
float distMpc = length(s_obj_pos) * u_uToMpc;
float fade = smoothstep(50.0, 30.0, distMpc);
stretch *= fade;

5. Apply trail effect

Directionally stretch the quad along the motion vector. Brightness is adjusted to maintain consistent visual intensity. Otherwise, the stars appear brighter as they are stretched, which is bad.

if (screenVel < 0.0001 || fade < 0.01) {
 stretch = 0.0;
} else {
 // Stretch around center position
 float taper = dot(stretch_dir, centered_pos);
 centered_pos += stretch * taper * stretch_dir;

 float brightnessScale = 2.0 / (1.0 + stretch);
 v_col.rgb *= clamp(brightnessScale, 0.0, 1.0);
}

6. Final assembly

We recenter, scale, rotate back to world space, and project to clip space.

local_pos = centered_pos * s_size;
vec3 world_pos = s_obj_pos + qrot(s_quat, local_pos);
// Final vertex position
gpos = s_proj_view * vec4(world_pos, 1.0);

Results

You have already seen the image at the top of this post. But an image without motion is frankly unimpressive. You can see a video of the effect in this Bluesky post:

#GaiaSky will get a star streaks shader in the next release. This effect elongates stars in the direction of motion to create a sort of Star Trek travel effect. Of course, this can be turned off in the settings.

[image or embed]
— Gaia Sky (@gaiasky.bsky.social) Jun 16, 2025 at 15:26

Conclusion

This vertex-based trail method avoids the cost and complexity of traditional post-processing while remaining physically intuitive and performant. This represents an ideal fit for real-time astronomical rendering, and for Gaia Sky in particular.

Some improvements to consider for the future could be:

Potential support for color or spectral shifts (red/blue) based on velocity and stretch.
Adaptive fall-off curves could optimize performance.
A GPU culling pass could eliminate distant objects entirely.

Quadruple joins the fight!

Toni Sagrista Selles — Sat, 10 May 2025 00:00:00 +0000

A few days ago I wrote about benchmarking arbitrary precision floating-point libraries in Java. I found out that BigDecimal is not as slow as it is said to be, beating Apfloat at the same precision level by a long margin in most operations. However, for Gaia Sky, I don’t need hundreds of significant digits at all. It turns out 27 significant digits are enough to represent the whole universe with a precision of 1 meter.

The observable universe has a radius of about $4.4 \times 10^{26}$ meters. To express the entire range down to 1 meter, we need to calculate the number of significant digits $d$ as follows:

$$ \begin{align} d &= \log_{10} \left(\frac{R}{\text{precision}}\right) \\ \\ &= \log_{10} \left(\frac{4.4 \times 10^{26}}{1}\right) \\ \\ &= \log_{10}(4.4 \times 10^{26}) \\ \\ &= \log_{10}(4.4) + \log_{10}(10^{26}) \\ \\ \approx 0.643 + 26 &= 26.643 \end{align} $$

In terms of bits, IEEE 754 double precision (64-bit) provides around 15–17 decimal digits of precision, which is enough for the Solar System, but insufficient for the whole universe. In contrast, IEEE 754 quadruple precision (128-bit) provides around 34 decimal digits of precision, which is in fact more than enough. It uses 113 bits of significand precision, $log_{10}(2^{113}) \approx 34$ digits. The range of values we can precisely represent in the universe is $\approx \frac{4.4 \times 10^{26}}{10^{34}} = 4.4 \times 10^{-8}$ meters. This is 4.4 nanometers! As said, this is more than sufficient for our purposes.

Enter Quadruple

Browsing through GitHub I found the Quadruple library, which provides an implementation of 128-bit floating point numbers in Java. The implementation is very compact, and includes addition, subtraction, multiplication, division, and square root. I decided to put it to the test using my JMH benchmark.

I created a new benchmark called “ThreeWay”, which tests these operations (plus allocation) for Apfloat, BigDecimal, and Quadruple. In the arbitrary precision library I’m using only 32 significant digits of precision instead of 34. I do 1 set of 1 second as warm-up, and 5 iterations of 5 seconds for the measurement (see source).

Results

Below are the specs of the system I used to run the tests, and the specific software versions used. This time around I ran the benchmarks in my laptop while it was plugged in. Only the CPU and the memory should play a significant role.

# JMH version: 1.37
# VM version: JDK 21.0.7, OpenJDK 64-Bit Server VM, 21.0.7+6
CPU: Intel(R) Core(TM) i7-8550U (8) @ 4.00 GHz
GPU 1: NVIDIA GeForce GTX 1070 [Discrete]
GPU: Intel UHD Graphics 620 @ 1.15 GHz [Integrated]
Memory: 16.00 GiB
Swap: 8.00 GiB

And here are the benchmark results.

Addition

Three-way Addition results – Interactive view

Of course, Quadruple is compact and only needs to care about 128 bits, while Apfloat and BigDecimal are generic to any precision, so we can expect Quadruple to be faster. And it is.

Subtraction

Three-way Subtraction results – Interactive view

Same with subtraction.

Multiplication

Three-way Multiplication results – Interactive view

And multiplication.

Division

Three-way Division results – Interactive view

Division is also faster with the newcomer.

Allocation (from string)

Finally, the allocation test. First, we test allocation from a string representation of a floating point number.

Three-way Allocation results (from string) – Interactive view

Surprising. Let’s analyze this. We use JOL to find out the instance size of each object.

Quadruple has an instance size of 40 bytes (2 longs, 1 int, 1 boolean, plus header).
BigDecimal has an instance size of also 40 bytes (2 ints, 1 long, 2 references to BigInteger and String, plus header).
Apfloat has an instance size of 24 (3 references plus the object header).

It is unlikely that the issue is the instance size. It most definitely comes down to the code to convert the string into the internal representation of each type. This code seems to be much slower for Quardruple than it is for the others. Let’s see how it fares allocating from a double.

Allocation (from double)

Three-way Allocation results (from double) – Interactive view

The story is reversed. Quadruple is much faster than the others when allocating an object from a double. I never allocate from strings, so this is not that bad actually.

Analysis

There’s not much to say. Quadruple is obviously much faster in a very significant way than the others. This is, of course, to be expected if we consider that Quadruple only deals with float-128 types and does not have to care about higher precisions. It may be enough for your purposes, like it is for mine. If this is the case, it may make sense to use it.

Caveats

There are a couple of important caveats to consider if you want to use Quadruple as it is now:

Only the basic operations are implemented (add, sub, div, mul, sqrt). If you need anything else, you are on your own.
Quadruple instances are mutable. This is a bad design decision in my opinion, and would bar it from adopting further improvements that will land soon to Java like value types (project Valhalla).
Instantiation from String is very slow.

Benchmarking arbitrary precision libraries in Java

Toni Sagrista Selles — Wed, 07 May 2025 00:00:00 +0000

Edit (2025-05-08): Changed some test parameters and re-run the tests. Also added bar plots.

Note: I have since written a new blog which includes Quadruple to the benchmarks, beating both Apfloat and BigDecimal consistently.

I recently set out to compare the performance of Apfloat and BigDecimal for arbitrary precision arithmetic in Java. I use arbitrary precision floating point numbers in key places of the update cycle in Gaia Sky, so it made sense to explore this. My initial approach was a naive benchmark: a simple main() method running arithmetic operations in a loop and measuring the time taken. The results were strongly in favor of BigDecimal, even for large precision values. This was unexpected, as the general consensus I found online suggested that Apfloat is more performant, especially for higher precision operations (hundreds of digits).

To get more accurate and reliable measurements, I decided to implement a proper JMH benchmark. The benchmark project source is available in this repository. The benchmarks test addition, subtraction, multiplication, division, power, natural logarithm, and sine for both Apfloat and BigDecimal at different precision levels.

Why JMH?

JMH is a benchmarking framework specifically designed for measuring performance in Java applications. It is developed by the OpenJDK team and provides a robust methodology for generating reliable and reproducible benchmark results by accounting for JVM warm-up, runtime optimizations, and other factors that can skew measurements. Given the surprising results in the naive implementation, using JMH allowed me to get more accurate measurements and mitigate potential inaccuracies caused by JVM behavior.

The Benchmark Implementation

The JMH benchmark project is structured to measure the average time taken for each arithmetic operation over several iterations and precision levels. Here’s the structure:

Separate benchmarks for addition, subtraction, multiplication, division, natural logarithm, power, and sine, additionally to an allocation test.
Each benchmark tests Apfloat and BigDecimal.
Create the actual objects at benchmark level to factor out allocation costs. Specific benchmark to test allocation overhead.
Settled on four precision levels, on a scale ranging from low and high precision settings, represented as the number of digits. They are 25, 50, 500, and 1000 digits.
Average time mode.
Every benchmark function only runs one operation once. The allocation test creates a couple of objects and consumes them.
One warm-up iterations of one second each to minimize JVM effects (@Warmup(iterations = 1, time = 1)).
Three main iterations of five seconds each for the measurement (@Measurement(iterations = 3, time = 5)).
Send results into Blackhole to prevent JIT optimizations.

Here is an example for the Sin benchmark:

Code: benchmark implementation

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.NANOSECONDS)
@Fork(value = 1)
@Warmup(iterations = 1, time = 1)
@Measurement(iterations = 3, time = 5)
public abstract class BaseBenchmark {

 @State(Scope.Thread)
 public static class BenchmarkState {
 MathContext mc;
 BigDecimal aBD, bBD;
 Apfloat aAF, bAF;

 @Param({ "25", "50", "500", "1000" }) // Add different precision levels here
 int precision;

 @Setup(Level.Trial)
 public void setUp() {
 mc = new MathContext(precision);
 aBD = new BigDecimal("12345.678901234567890123456789012345678934343434343434343434343434343434", mc);
 aAF = new Apfloat("12345.678901234567890123456789012345678934343434343434343434343434343434", precision);
 }
 }
}

public class Sin extends BaseBenchmark {

 @Benchmark
 public void BigDecimalSin(BenchmarkState state, Blackhole bh) {
 var result = BigDecimalMath.sin(state.aBD, state.mc);
 bh.consume(result);
 }

 @Benchmark
 public void ApfloatSin(BenchmarkState state, Blackhole bh) {
 var result = ApfloatMath.sin(state.aAF);
 bh.consume(result);
 }
}

The Results

Below are the specs of the system I used to run the tests and the specific software versions used. Only the CPU and the memory should play a significant role.

# JMH version: 1.37
# VM version: JDK 21.0.7, OpenJDK 64-Bit Server VM, 21.0.7+6
CPU: Intel(R) Core(TM) i7-7700 (8) @ 4.20 GHz
GPU 1: NVIDIA GeForce GTX 1070 [Discrete]
GPU 2: Intel HD Graphics 630 [Integrated]
Memory: 32.00 GiB
Swap: 8.00 GiB

And here are the benchmark results.

Addition

Addition results – Interactive view

We already see that BigDecimal is much faster in all precisions. It is not even close.

Subtraction

Subtraction results – Interactive view

In the subtraction benchmark BigDecimal comes out on top as well.

Multiplication

Multiplication results – Interactive view

The same story repeats for multiplication.

Division

Division results – Interactive view

Again. Division is a notoriously costly operation, but BigDecimal still comes out comfortably on top.

Now, let’s test some more involved arithmetic operations, like the natural logarithm, the sine, and the power function. In Apfloat, those are directly implemented in the library. For BigDecimal, we use the big-math project.

Log

Log results – Interactive view

The logarithm is faster with Apfloat at the higher precision settings, but it BigDecimal still wins in the lower precisions.

Sin

Sin results – Interactive view

The sine is much faster in BigDecimal in all precision settings.

Pow

Pow results – Interactive view

And finally, the power repeats the same story, with BigDecimal sitting comfortably on the throne again.

Allocation

For science, I thought it would be cool to test the allocation overhead, so I prepared the Allocation test, which allocates two instances of either Apfloat or BigDecimal and consumes them.

Allocation results – Interactive view

We see that allocation is very costly in both libraries. However, while Apfloat seems to be roughly constant with the precision, BigDecimal shows a higher cost with 25 digits, the lowest precision setting. I though this was very sus, so I re-ran the test a bunch of times and with more iterations and longer times, and got back the same result. I’m not sure what’s the root cause for this, but it is surprising and intriguing.

Since both Apfloat and BigDecimal are immutable, allocation costs need to be factored in. New objects need to be allocated every time new operands are needed.

Analysis

Contrary to expectations, BigDecimal consistently outperformed Apfloat across all operations and precision levels, including the higher precisions (500 and 1000 digits) where Apfloat was expected to excel. There is a single case when Apfloat is faster, and that is in the high precision natural logarithm benchmark. I think it’s safe to say that this is due to the particular implementation or algorithm being used. Otherwise, the disparity is particularly noticeable in division and sine operations, where Apfloat is significantly slower than BigDecimal. Specifically, BigDecimal was several times faster than Apfloat in most operations and precisions. Those are, in my opinion, significant results.

Finally, allocation seems to be faster with Apfloat, and there’s a weird dependency on the precision for BigDecimal which I found strange.

Questions and Next Steps

I was genuinely surprised by the outcome of these benchmarks, as it contradicts the general consensus regarding Apfloat’s supposed performance advantage in high-precision arithmetic. I am reaching out to the community to validate my methodology and results. Are these findings trustworthy, or did I overlook something crucial in my benchmarking approach? Feedback and insights are very much welcome.

Oh Garmin, not you!

Toni Sagrista Selles — Sat, 29 Mar 2025 00:00:00 +0000

I’ve been a Garmin user for many years, shelling out non-trivial amounts of monies for their sports watches. My first Garmin watch was a Forerunner 10 (black/red). Battery life was abysmal, but back then this was the norm. Today, I’m sporting a Forerunner 255, which I love. It’s not top-of-the-line, but it is more than enough for my modest purposes. These devices have been my trusty companions on countless runs, football matches, and hikes, providing invaluable data without any hidden costs. But now, Garmin has decided to introduce Garmin Connect+, a subscription service priced at €8.99 per month or €89.99 annually. Really, Garmin?

My Garmin Forerunner 255.

This past Thursday, March 27th, Garmin announced a premium plan called Garmin Connect+ in a press release titled “Elevate your health and fitness goals with Garmin Connect+”. This new service promises AI-powered insights through their “Active Intelligence” feature, offering personalized suggestions based on health and activity data. Additionally, it includes a performance dashboard with custom charts, enhanced LiveTrack features, and exclusive app badges. I don’t care for any of that, but these additions may sound appealing to some. I can’t help but feel a sense of déjà vu. Initially, companies introduce new features behind a paywall, assuring users that existing functionalities remain free. But as revenue targets loom, there’s a creeping tendency to lock previously free features behind that same paywall, degrading the user experience. This gradual decline, often referred to as enshittification, has plagued many platforms.

Garmin’s move mirrors trends seen with other fitness platforms like Strava, which have also introduced subscription models for premium features. While it’s understandable that publicly traded companies seek additional revenue streams, it’s disheartening when it comes at the expense of loyal users who have already invested heavily in their products. The general feeling of their customers is clear. You just need to look around the Garmin subreddit, or the Garmin Subscription Model megathread specifically, to know what I’m talking about.

Of course, as these things usually go, Garmin assures us that all existing features will remain free and that Connect+ merely adds new functionalities. I don’t buy it. Forgive me for being skeptical. We’ve seen this play out before, and it often doesn’t end well for the consumer.

Garmin watches are already a significant investment. Introducing a subscription model feels like a cash grab, especially when the features offered seem like they could be standard inclusions. It’s a slippery slope, and I fear we’re at the top, ready to slide down. I hope Garmin proves me wrong and maintains the integrity of its free offerings. But history suggests otherwise, and I can’t help but brace for the inevitable enshittification of a platform I once held in high regard.

One thing is clear to me. My next sports watch will not be a Garmin watch. This move has annihilated my trust in the company. I’ve started researching other options, and I have found Polar and Suunto, both based in Europe, to be excellent alternatives.

Building a local AI assistant with user context

Toni Sagrista Selles — Wed, 26 Mar 2025 00:00:00 +0000

In my last post, I explored the concept of Retrieval-Augmented Generation (RAG) to enable a locally running generative AI model to access and incorporate new information. To achieve this, I used hardcoded documents as context, which were then embedded as vectors and persisted into Chroma DB. These vectors are used during inference to use as context for a local LLM chatbot. But using a few hardcoded sentences is hardly elegant or particularly exciting. It’s alright for educational purposes, but that’s as far as it goes. However, if we need to build a minimally useful system, we need to be more sophisticated than this. In this new post, I set out to create a local Gaia Sky assistant by using the Gaia Sky documentation site and the Gaia Sky homepage as supplementary information, and leveraging Ollama to generate context-aware responses. So, let’s dive into the topic and explain how it all works.

The source code used in this post is available in this repository.

Scraping the websites

The first thing we need to do is extract useful content from our sources. Since we use websites, this is where web scraping comes into play:

We’re using the requests library to fetch the HTML content from a given URL.
First, we get all the internal links for every URL by recursively scraping the content (with BeautifulSoup) and looking for all a tags (anchors) that don’t point to internal anchors. This happens in get_all_doc_links(base_url).
Then, we extract the text from every page by extracting the tags h1, h2, h3, h4, h5, p, li, td, and article. This step required a little bit of trial and error. Once we have the text for each page, we concatenate everything and return it. This is implemented mostly in extract_text_from_page(url). At first, I wrote my own code to retrieve content from a URL and convert it to text. Of course, this code was less than impressive, and even though it worked, the results were not optimal. Since then I experimented with other methods to extract text from websites, like the libraries newspaper3K, goose3, and html2text. I settled for the latter, as it produces Markdown-formatted documents which are particularly well suited for LLMs. You can see the very different results from the previous method by inspecting the history of this post. The initial scraping behavior can still be accessed in the application with the flag --local-scraper.
As we mentioned earlier, we make sure to avoid non-HTML files and internal page anchors (those pesky # URLs that only point to a specific section). This ensures that we only scrape actual webpages that contain relevant data.

With this method, we capture all the relevant documentation—every corner of the site. Since the process is recursive, we don’t miss any pages (unless they’re blocked or have weird redirect loops, but that’s a discussion for another time).

The code box below shows the link gathering, scraping, and text extraction.

import os

# Set user agent enviornment variable
user_agent = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0.0.0 Safari/537.36"
os.environ["USER_AGENT"] = user_agent

import requests, ollama, argparse, readline
from termcolor import colored
from bs4 import BeautifulSoup
from urllib.parse import urljoin, urlparse
from langchain_community.document_loaders import WebBaseLoader
from langchain_chroma import Chroma
from langchain_ollama import OllamaLLM
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.chains import RetrievalQA
from langchain_huggingface import HuggingFaceEmbeddings

headers = {
 "User-Agent": user_agent
}

def get_all_doc_links(base_url):
 """Finds all internal links within the documentation site."""
 visited = set()
 to_visit = {base_url}

 while to_visit:
 url = to_visit.pop()
 print(f"Scraping {colored(url, 'blue')}")
 if url in visited:
 continue
 visited.add(url)

 response = requests.get(url, headers=headers)
 if response.status_code != 200:
 continue

 soup = BeautifulSoup(response.text, "html.parser")
 for link in soup.find_all("a", href=True):
 full_url = urljoin(base_url, link["href"])
 parsed = urlparse(full_url)

 # Ignore anchors (#...) and non-HTML files
 if parsed.fragment or not parsed.path.endswith((".html", "/")):
 continue

 if base_url in full_url and full_url not in visited:
 to_visit.add(full_url)

 return visited

def extract_text_from_page(url):
 """Extracts meaningful text from a given URL using html2text."""
 import html2text
 try:
 response = requests.get(url)
 h = html2text.HTML2Text()
 h.ignore_links = False # Set this to True to ignore links
 return h.handle(response.text)
 except:
 return None

def scrape_urls(base_urls, local_scraper=False):
 """Scrape and extract text content from multiple URLs."""
 all_text = ""

 for base_url in base_urls:
 all_links = get_all_doc_links(base_url)
 print(f"Found {len(all_links)} pages to scrape...")
 print(f"Extracting text from pages...")
 for url in all_links:
 page_text = extract_text_from_page(url)
 if page_text:
 all_text += f"\n\n### {url}\n{page_text}"

 return all_text

 # Main body
 urls = ["https://gaia.ari.uni-heidelberg.de/gaiasky/docs/master/",
 "https://gaiasky.space"]

 print("Starting scraping...")
 doc_text = scrape_urls(urls)

Storing embeddings with ChromaDB

Once we’ve scraped the content, it’s time to turn that raw text into something that the machine can understand. This is where embeddings come into play.

We use the langchain library to split the scraped text into manageable chunks. This is done using the RecursiveCharacterTextSplitter, which splits the content into chunks of a predefined size (to avoid overloading the model). I tuned this by trial and error, and ended up with a chunk size of 1000, and a chunk overlap of 200.
The text is then passed through an embedding model, specifically the “sentence-transformers/all-MiniLM-L6-v2” model from HuggingFace, which transforms it into a high-dimensional vector representation. It is super compact and performs well. I have tried other embed models (see commented out lines at the top of this code block), but in my experience they did not perform better than all-MiniLM-L6-v2.
Finally, we store these vectors in a vector database, Chroma DB. The vectors are indexed so that when we query the database, it can efficiently retrieve the most relevant pieces of text. For the retrieval, I’m using a default of 6 results (search_kwargs in as_retriever() method). I tried with 2 and 3, but it seemed that most models didn’t get enough context.

This is where the magic happens. By converting text into vectors, we enable the chatbot to compare and understand the semantic meaning of different pieces of text. The advantage here is that embeddings can capture the relationships between words and concepts, even if they’re not an exact match — meaning the chatbot can intelligently pull out relevant context to answer your questions.

The relevant code consists of the concatenated texts we produced in the previous step (we called it doc_text), and the location for our disk database:

embed_model = "sentence-transformers/all-MiniLM-L6-v2"
# embed_model = "BAAI/bge-base-en-v1.5"
# embed_model = "HIT-TMG/KaLM-embedding-multilingual-mini-v1"
# embed_model = "Linq-AI-Research/Linq-Embed-Mistral"

def store_embeddings(texts, db_path):
 """Tokenizes, embeds, and stores the texts in a ChromaDB vector store."""
 text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=200)
 documents = text_splitter.create_documents([texts])
 if not documents:
 raise ValueError("No documents found to embed. Check the scraper output.")

 embeddings = HuggingFaceEmbeddings(model_name=embed_model)
 vector_store = Chroma.from_documents(documents, embeddings, persist_directory=db_path)
 return vector_store

print("Storing embeddings in ChromaDB...")
db_path = "chroma_db"
vector_store = store_embeddings(doc_text, db_path)

Querying with Ollama

Now that we’ve got our indexed data, it’s time to query it. But instead of just returning raw search results, we want to use an LLM (Large Language Model) to generate smart, context-aware answers. Enter Ollama.

We use langchain’s create_stuff_documents_chain and create_retrieval_chain to first create a chain to retrieve documents from our vector store and pass them on to the model.
The LLM we’re using is locally hosted through Ollama. The model takes the relevant text retrieved from Chroma and uses that context to generate responses to user queries. The actual model is user-selected at the beginning of the program.
We use a simple interface where the user can ask questions, and the chatbot pulls from the stored content to generate and stream an answer. If it doesn’t have enough context to answer, it will graciously let you know, but more often than not, it’s ready to impress with its knowledge.

By combining Chroma with an LLM, we ensure that the chatbot not only returns raw text but generates contextually appropriate answers. The power of an LLM with RAG lies in its ability to take a set of relevant documents and provide a coherent, informative response. So, when you ask, “What’s the internal reference system of Gaia Sky?” the chatbot can retrieve the most relevant content and synthesize it into a helpful response.

System prompt

One of the key elements in ensuring a Retrieval-Augmented Generation (RAG) system behaves as expected is the system prompt—a set of initial instructions that guide how the language model responds to queries. In our application, the system prompt is designed to enforce a structured and helpful interaction:

system_prompt = (
 "You are a customer support agent, helping users by following directives and answering questions. "
 "Use the given context to answer the question. "
 "If you don't know the answer, say you don't know. "
 "Keep the answer concise and to the point. "
 "Context: {context}"
)

The prompt explicitly states that the LLM is a customer support agent to steer its behavior towards professional responses. It also includes a sentence about context awareness so that it actually uses information given in the context. Then, we ask it to not invent something it does not find in the context, and we demand short and concise answers.

By fine-tuning this system prompt, we set clear boundaries for how the model should behave, ensuring it delivers accurate, relevant, and trustworthy responses based on the retrieved knowledge.

Other parameters

Temperature – The temperature controls the randomness of the model’s responses, with lower values making answers more deterministic and factual, while higher values increase creativity and variability. We use a default low temperature value of 0.3 to get more correct and less creative responses. The temperature can be set with the argument -t.
Number of results – The number of results parameter (n_results or k) determines how many relevant documents are retrieved from the vector database to provide context for the model’s response, with higher values increasing the available information but potentially adding noise. We use the default of 6 results, but this can be changed with the -k argument.

The relevant code that does the actual querying lives in the query_with_ollama(vector_store, llm_model) method. The code below includes the relevant parameters at the top.


system_prompt = (
 "You are a customer support agent, helping users by following directives and answering questions. "
 "Use the given context to answer the question. "
 "If you don't know the answer, say you don't know. "
 "Keep the answer concise and to the point. "
 "Context: {context}"
)

# In the real program, both n_results and temp can be set with program arguments.
# Number of results to retrieve from the vector store.
n_results = 6
# Low temperature to get less creative and more correct responses
temp = 0.3

def query_with_ollama(vector_store, model_name):
 """Uses an Ollama model to retrieve and answer user queries based on stored embeddings."""
 retriever = vector_store.as_retriever(search_kwargs={"k": n_results})
 prompt = ChatPromptTemplate.from_messages(
 [
 ("system", system_prompt),
 ("human", "{input}"),
 ])
 qa_chain = create_stuff_documents_chain(OllamaLLM(model=model_name, temperature=temp, stream=True), prompt)
 chain = create_retrieval_chain(retriever, qa_chain)

 while True:
 query = input(colored("Ask a question (type 'exit' to quit): ", "yellow", attrs=["bold"]))
 if query.lower() in ["exit", "quit", "bye"]:
 break

 for chunk in chain.stream({"input": query}):
 if "answer" in chunk:
 print(chunk["answer"], end="", flush=True)

 print("\n")

This method contains the context retrieval code and the question-answer loop. The actual LLM model to use, stored in llm_model, is user-defined (code not shown here). After starting, the program presents the available models, and the user needs to select one of them. This happens in the chatbot() function, which is the final piece of the puzzle. It gets the vector store object—either we have just ingested it from scraping (if --scrape is used) or we have retrieved it from the database, if the program is called with no arguments. Then, it prompts the user for the model to use, and calls query_with_ollama(...).

def chatbot(vector_store):
 # List available models
 try:
 models = ollama.list()
 model_names = [m.model for m in models.models]
 except:
 print(f"{colored('ERROR', 'red')}: Ollama service is not running.")
 exit(1)

 # Print the available models with index numbers
 print("Available models:")
 for i, name in enumerate(model_names):
 print(f" [{colored(i, 'green')}] {name}")

 # Loop until a valid selection is made
 while True:
 choice = input(f"\nSelect model (default {colored('0', 'green')}): ").strip()

 if choice == "": # Default to 0 if empty input
 llm_model = model_names[0]
 break

 if choice.isdigit(): # Check if input is a number
 index = int(choice)
 if 0 <= index < len(model_names): # Check if index is valid
 llm_model = model_names[index]
 break

 print("Invalid selection. Please enter a valid number.")

 print(f"LLM model: {colored(llm_model, 'yellow', attrs=['bold'])}")
 print(f"Embed model: {colored(embed_model, 'yellow', attrs=['bold'])}")

 print("Connecting to Ollama for queries...")
 query_with_ollama(vector_store, llm_model)

Implementation details

The scraping function is hidden behind a flag (--scrape). When it is used, the websites are scraped, their information is extracted, embedded into vectors and ingested in the database. When no arguments are used, the program starts the chatbot. Of course, you need to run at least once with --scrape before using the chatbot. But if you’ve already scraped the site and generated embeddings, the system will load the existing embeddings from disk, saving you time and resources. It’s efficient, and you don’t have to redo the work every time you launch the chatbot.

If you change the embed model, remember to nuke the database (rm -rf chroma_db) and re-do the scraping!

Finally, by using stream=True in the creation of OllamaLLM, and calling chain.stream() instead of chain.query(), we can stream the response as it is generated, token-by-token. This is the style most web chatbots use.

Evaluation

In this section we test the system with mistral-small:22b-instruct-2409-q4_K_M, a small instruct LLM with capabilities comparable to larger models, and with llama3.1:8b-instruct-q5_K_M, a much smaller but better quantized model(Q5-K-M vs Q4-K-M), also specifically trained for instruct.

I ask each model 3 questions:

The usages of Gaia Sky.
How to install Gaia Sky on Linux.
How are cubemap textures used in Gaia Sky.

Here are links to the relevant sections.

Mistral Small 3
Llama 3.1

Note

You can install these models directly with Ollama:

ollama pull mistral-small:22b-instruct-2409-q4_K_M llama3.1:8b-instruct-q5_K_M

Remember, if you want to try out the chatbot yourself, download it from this repository.

Mistral Small 3 instruct, 22B Q4-K-M

Here we use the general Mistral Small 3 instruct (22B, Q4-K-M) model. It uses Q4_K_M quantization and features 22.2B parameters. I think this is around the limit of what my computer can handle (see conclusion).

On a first pass, we run the program with the --scrape flag to gather the information from the websites and create the embeddings. This corresponds to the output shown below, where all the URLs are fetched and scraped.

$ gsai.py --scrape
Welcome to the Gaia Sky AI assistant! We connect to Ollama to use a local LLM.

Available models:
 [0] mistral-small:22b-instruct-2409-q4_K_M
 [1] llama3.1:8b-instruct-q5_K_M
 [2] gemma3:12b
 [3] qwen2.5:7b
 [4] deepseek-r1:14b
 [5] llama3.1:8b
 [6] mistral-small:24b

Select model (default 0): 0
Using model: mistral-small:22b-instruct-2409-q4_K_M
Embed model: sentence-transformers/all-MiniLM-L6-v2

Starting scraping...
Scraping https://gaia.ari.uni-heidelberg.de/gaiasky/docs/master/
Scraping https://gaia.ari.uni-heidelberg.de/gaiasky/docs/master/Location-log.html
Scraping https://gaia.ari.uni-heidelberg.de/gaiasky/docs/master/Virtual-textures.html
[...]
Scraping https://gaia.ari.uni-heidelberg.de/gaiasky/docs/master/Cubemaps.html
Scraping https://gaia.ari.uni-heidelberg.de/gaiasky/docs/master/Archetypes.html
Found 69 pages to scrape...
Extracting text from pages...
Scraping https://gaiasky.space
Scraping https://gaiasky.space/privacypolicy/
Scraping https://gaiasky.space/news/
[...]
Scraping https://gaiasky.space/news/2023/gaiasky_3.4.0/
Scraping https://gaiasky.space/news/2023/gaiasky_3.5.3/
Scraping https://gaiasky.space/news/page/5/
Scraping https://gaiasky.space/news/page/6/
Found 64 pages to scrape...
Extracting text from pages...
Storing embeddings in ChromaDB...

Done! Run the script without arguments to start the chatbot.

Then, we run gsai.py without the flag to access the chatbot directly and start asking questions. These are documented in the quoted blocks below.

First, I asked about what can Gaia Sky be used for. This is a general type of question, so let’s see how it fares.

Q: State the possible usages of Gaia Sky.

Gaia Sky is a powerful and versatile software tool that can be used for a variety of purposes, especially in the fields of astronomy, education, and scientific research. Here are some of its potential usages:

Educational Tool: Gaia Sky can be used to teach students about astronomy, celestial mechanics, and the universe. It provides an interactive and visually engaging way to learn about different astronomical objects and phenomena.

Research in Astronomy: Scientists and researchers can use Gaia Sky to visualize and analyze complex datasets from space missions like Gaia, allowing them to gain insights into stellar evolution, galactic structure, and other astrophysical processes.

Data Visualization: The software offers advanced data visualization capabilities, enabling users to create detailed 3D models of celestial objects and phenomena. This is particularly useful for presenting research findings or educational content in a compelling manner.

Virtual Tours: Gaia Sky can be used to create virtual tours of the universe, allowing users to explore different regions of space, from our solar system to distant galaxies.

Public Outreach: Astronomy enthusiasts and science communicators can use Gaia Sky to create engaging presentations and demonstrations for public events, such as star parties or science festivals.

Simulations: Researchers can use the software to run simulations of various astronomical scenarios, helping them understand how different factors influence celestial phenomena.

Planetarium Software: Gaia Sky can be used in planetariums to provide immersive and interactive experiences for visitors, enhancing their understanding of astronomy through visual storytelling.

Scientific Collaboration: The software facilitates collaboration among researchers by providing a common platform to share and discuss data, simulations, and findings.

Artistic Visualizations: Artists can use Gaia Sky to create stunning visual representations of astronomical objects and scenes for artistic purposes or multimedia projects.

Space Mission Support: Gaia Sky can be used by teams involved in space missions to plan observations, analyze data, and support mission operations.

These uses highlight the flexibility and power of Gaia Sky as a tool for both educational and professional applications in astronomy and related fields.

This answer is not concise, but it is quite alright. It captures all important features of Gaia Sky and gives a short description for each. However, I think it might be overselling Gaia Sky a tiny bit, but hey. Also, this model is quite slow on my computer, more on that later.

Next, I ask how to install Gaia Sky on Linux. We have several options to install the software on Linux, including APT, DEB and AUR packages, Flatpak, AppImage, and tarballs. Let’s see if the model captures these well.

Q: How do I install Gaia Sky on Linux?

To install Gaia Sky on a Linux system, you have several options depending on your distribution and preferences. Here are the most common methods:

Using Flatpak
Install Flatpak: If you don’t already have Flatpak installed, you can install it using your package manager. For example, on Debian-based systems (like Ubuntu), you would use:
sudo apt update
sudo apt install flatpak
Add the Flathub repository:
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
Install Gaia Sky:
flatpak install flathub space.gaiasky.GaiaSky
Run Gaia Sky:
flatpak run space.gaiasky.GaiaSky
Using AppImage
Download the AppImage: Go to the Gaia Sky downloads page and download the latest .AppImage file for Gaia Sky.
Make the AppImage executable: Open a terminal, navigate to the directory where you downloaded the AppImage, and run:
chmod +x GaiaSky-*.AppImage
Run the AppImage:
./GaiaSky-*.AppImage
Using Unix Installer
Download the installer: Go to the Gaia Sky downloads page and download the .run file for the Unix installer.
Make the installer executable: Open a terminal, navigate to the directory where you downloaded the installer, and run:
chmod +x GaiaSky-*.run
Run the installer:
./GaiaSky-*.run
Follow the on-screen instructions to complete the installation.
Using TAR.GZ Package
Download the TAR.GZ package: Go to the Gaia Sky downloads page and download the .tar.gz file for Gaia Sky.
Extract the package: Open a terminal, navigate to the directory where you downloaded the TAR.GZ file, and run:
tar -xzf GaiaSky-*.tar.gz
Move the extracted folder to your desired installation location (e.g., /opt):
sudo mv GaiaSky /opt/GaiaSky
Create a symbolic link in /usr/local/bin:
sudo ln -s /opt/GaiaSky/gaiasky /usr/local/bin/gaiasky
Run Gaia Sky:
gaiasky
Using Distro-Specific Packages (DEB, RPM, AUR) For distro-specific packages, you can usually install them directly using your package manager:
Debian/Ubuntu:
sudo apt update
sudo apt install gaia-sky
RedHat/Fedora:
sudo dnf install gaia-sky
Arch Linux:
yay -S gaia-sky
Choose the method that best suits your needs and preferences.

Wow, lots of awesome info and formatting here. It gets most of the information right, but it misses the mark on the installation for Debian and RedHat (we have no packages in their repositories), and Arch Linux (the package is actually called gaiasky, not gaia-sky). Other than that, I think this is a good answer.

To be fair, this was a rather open question, so it would be surprising that such a small model were able to capture all the info perfectly.

Finally, I asked for something much more technical and specific—I asked it to explain how cubemaps are used in Gaia Sky.

Q: Explain how cubemap textures are used in Gaia Sky.

Cubemaps are a type of texture that maps an entire environment onto a cube’s six faces, providing a full 360-degree view. In Gaia Sky, cubemaps are primarily used to create realistic and immersive environments for planets and other celestial bodies. Here’s how they are utilized:

Environment Mapping: Cubemaps are employed for environment mapping, which involves projecting the surrounding scene onto a sphere or cube. This technique is particularly useful for rendering reflections on surfaces, such as water or metallic objects, giving them a more realistic appearance.

Skyboxes: One common use of cubemaps in Gaia Sky is to create skyboxes. A skybox is a cube that surrounds the scene and displays an environment map on its faces. This technique is used to simulate distant scenery, such as the sky or outer space, without needing to render far-off objects in detail.

Planetary Surfaces: Cubemaps can also be applied to planetary surfaces to create detailed and varied textures. By mapping different environments (like oceans, mountains, or forests) onto the faces of a cube, Gaia Sky can generate diverse and realistic planetary landscapes.

Lighting and Shadows: Cubemaps are used in lighting calculations to simulate how light interacts with the environment. This includes creating realistic shadows and highlights based on the surrounding scene, enhancing the overall visual quality of the rendered objects.

Implementation Steps:

Creation of Cubemap Textures:

Developers create cubemap textures using specialized software that can render 360-degree environments. These tools often allow for the creation of high-quality, seamless textures.

Loading Cubemaps in Gaia Sky:

The created cubemaps are then loaded into Gaia Sky. This process involves specifying which texture should be applied to which face of the cube (positive X, negative X, positive Y, negative Y, positive Z, and negative Z).

Application in Shaders:

The shaders in Gaia Sky are programmed to use these cubemaps for various effects, such as reflections, environment lighting, and skybox rendering. This is done by sampling the cubemap texture within the shader code based on the viewing direction or other parameters.

Benefits:

Realism: Cubemaps help create highly realistic environments by providing detailed textures and reflections.

Performance: Instead of rendering distant objects, which can be computationally expensive, cubemaps allow for quick and efficient simulation of the environment.

Flexibility: They can be easily swapped out to change the appearance of an environment without altering the underlying geometry.

In summary, cubemap textures play a crucial role in Gaia Sky by enabling the creation of immersive, realistic environments through techniques like environment mapping and skybox rendering.

This is perhaps too detailed, and sources information from outside of the context, but it is mostly correct, if a little verbose. I’m happy with this answer.

All in all, I think this Mistral model is a little too verbose. You can see specks of larger models, like the good formatting, or the effort to produce complete and exhaustive answers. However, the accuracy is not always on point. I wonder if using a model with better quantization (Q6?) would solve this.

Llama 3.1 instruct, 8B Q5-K-M

Now we test the Llama 3.1 instruct (8B, Q5-K-M) model, an instruct-specific model that is much smaller than Mistral Small 3 (24B). However, this time around I stepped up the quantization a bit by using Q5_K_M, which should improve accuracy over the basic Q4_0.

First, let’s ask about the usages:

Q: State the possible usages of Gaia Sky.

Gaia Sky is a free, open-source planetarium software that can be used for various purposes, including:

Educational purposes: Gaia Sky can be used in educational settings to teach astronomy and astrophysics concepts to students.

Research and scientific visualization: The software’s advanced features make it suitable for researchers who need to visualize complex astronomical data, such as star catalogs, galaxy distributions, or planetary systems.

Public outreach and engagement: Gaia Sky can be used in public events, exhibitions, or planetariums to showcase the wonders of the universe and promote interest in astronomy among the general public.

Personal entertainment: Users can simply enjoy exploring the night sky, viewing celestial objects, and experiencing the thrill of space travel within the software’s virtual environment.

Artistic and creative projects: Gaia Sky’s capabilities make it an attractive tool for artists, designers, and filmmakers who need to create realistic astronomical environments or simulate complex astrophysical phenomena.

Professional training and development: The software can be used by professionals in fields like astronomy, space exploration, or related industries to improve their knowledge and skills through interactive simulations and visualizations.

These are just a few examples of the many possible usages of Gaia Sky. Its versatility and flexibility make it an excellent tool for various applications across different domains.

This is a good response. Not too long, and very well organized. Everything in it is correct, too. Also, Llama 3.1 was very fast in generating it.

The next question is about the Linux installation.

Q: How do I install Gaia Sky on Linux?

To install Gaia Sky on Linux, you can follow these steps for the different package types:

Flatpak
Install Flatpak if you haven’t already:
sudo apt-get install flatpak
Add the Flathub repository:
flatpak remote-add --if-not-exists flathub https://flathub.org/repo/flathub.flatpakrepo
Install Gaia Sky from Flathub:
flatpak install flathub space.gaiasky.GaiaSky
Run Gaia Sky:
flatpak run space.gaiasky.GaiaSky
AppImage
Download the AppImage file from the official website: https://gaiasky.space/downloads/
Make the AppImage executable:
chmod +x gaiasky_x86_64.AppImage
Run Gaia Sky:
./gaiasky_x86_64.AppImage
Unix Installer (tarball)
Download the tarball from the official website: https://gaiasky.space/downloads/
Extract the contents of the tarball to a directory of your choice:
tar -xvf gaiasky.tar.gz
Run Gaia Sky:
./gaiasky.sh
DEB (Debian and derivatives)
Download the DEB package from the official website: https://gaiasky.space/downloads/
Install the package using dpkg:
sudo dpkg -i gaiasky_3.6.7_amd64.deb
Run Gaia Sky:
gaiasky
RPM (RedHat and derivatives)
Download the RPM package from the official website: https://gaiasky.space/downloads/
Install the package using rpm:
sudo rpm -i gaiasky-3.6.7-x86_64.rpm
Run Gaia Sky:
gaiasky
AUR (Arch Linux and derivatives)
Install the AUR helper of your choice (e.g., yay, yay-bin).
Clone the Gaia Sky AUR package repository:
git clone https://aur.archlinux.org/gaiasky.git
Navigate to the cloned repository directory:
cd gaiasky
Build and install the package:
makepkg -si
Run Gaia Sky:
gaiasky

This answer is fracking excellent. It nails everything. It includes all Linux packages we offer, and the instructions for each are on point. I’m super impressed. The quality of this response is much better than the one generated by the (much larger) Mistral Small 3 22B.

Finally, we ask it what cubemaps are used for:

Q: How are cubemap textures used in Gaia Sky?

In Gaia Sky, cubemap textures are used to display planetary surfaces and other 3D models with high detail and realism. Here’s an overview of how they work:

What is a cubemap?

A cubemap is a type of texture that represents the surface of a 3D object from all six sides (top, bottom, left, right, front, and back). It’s called a “cubemap” because it’s essentially a cube with six faces, each containing a different view of the object.

How are cubemaps used in Gaia Sky?

In Gaia Sky, cubemaps are used to display planetary surfaces, such as Earth or other planets. When you load a planet into the game, Gaia Sky generates a cubemap texture that represents the surface of the planet from all six sides. This allows for more detailed and realistic rendering of the planet’s surface.

How do cubemaps improve graphics in Gaia Sky?

Cubemaps offer several benefits over traditional textures:

Higher detail: Cubemaps can display much higher levels of detail than traditional textures, making planetary surfaces look more realistic.

Realistic lighting: Cubemaps take into account the way light interacts with the surface of the planet, creating a more realistic and immersive experience.

Improved performance: Because cubemaps are pre-computed and stored in memory, they can be rendered much faster than traditional textures, reducing the load on your graphics card.

How to use cubemap textures in Gaia Sky

To use cubemap textures in Gaia Sky, you’ll need to:

Load a planet or other 3D model into the game.

Adjust the graphics settings to enable cubemaps (this may require updating to a newer version of Gaia Sky).

Select the desired level of detail and lighting for the cubemap texture.

By using cubemap textures, Gaia Sky can create stunningly realistic planetary surfaces that draw you in and make you feel like you’re exploring the cosmos itself!

The format and wording of this answer look like it comes from a larger model. However, I think the content misses the point a bit. It does not look like it sources this information from the Cubemaps section in our documentation, which may be an issue with the embedding, the context size, or the context retrieval. I tried reducing the number of results from 10 to 5 before running the query again:

How are cubemaps used in Gaia Sky?

Cubemaps are used in Gaia Sky to texture planets, moons, and other spherical or semi-spherical objects. They help eliminate artifacts that happen at the poles with UV sphere models, providing a more seamless appearance. Cubemaps can be used instead of regular equirectangular (spherically projected) images.

This time around, the answer is much more concise and correct.

In the global picture, I like the responses of Llama better than those provided by Mistral. Also, Mistral is almost 3 times larger than Llama (22B vs 8B), so it is much slower on my machine.

Performance

In my case, I’m pretty limited by my hardware, and responses tend to be slow with the larger model (Mistral). The speed is quite good with Llama (8B) though. True, I’m running this a very old computer for today’s standards (Intel(R) Core(TM) i7-7700 (8) @ 4.20 GHz, NVIDIA GeForce GTX 1070 8 GB, 32 GB RAM), so it is what it is.

Here are the performance stats for both models. I asked them both to generate a short text (~100 words) about the meaning of life.

Model	Total Duration	Prompt Eval Rate	Eval Count	Eval Duration	Eval Rate
`mistral-small:22b-instruct-2409-q4_K_M`	24.575599226s	53.56 tokens/s	67 token(s)	23.194348749s	2.89 tokens/s
`llama3.1:8b-instruct-q5_K_M`	7.825027246s	109.12 tokens/s	119 token(s)	7.526260082s	15.81 tokens/s

I found the responses provided by smaller models (1B to 5B) to be mostly garbage. On larger instruct models with decent quantization, accuracy is greatly improved at the expense of your precious time.

Conclusion

In this post, we’ve expanded on our previous post and developed an AI chatbot able to answer questions on Gaia Sky. The information is sourced from the Gaia Sky homepage and documentation. With the power of web scraping, embeddings, and Ollama, we’ve brought the Gaia Sky website and docs to life with smart, context-aware interactions. No more endless searches for answers. Instead, we now have a chatbot that knows exactly where to look and can provide concise, relevant responses. So go ahead and try it out—who knew scraping could be this useful?

It’s pretty obvious to me that such small models are able to get most of the questions almost right, but they are most definitely not suited for professional settings. In this case, larger models (at least 70B, I reckon) would be required. For our purposes here, Mistral Small 3 instruct (22B) and Llama 3.1 instruct (8B) are enough.

I don’t want to finish without mentioning off-the-shelf options that accomplish the same goals. There exist no-code software packages like AnythingLLM and Open WebUI that support local models via Ollama and integrate RAG. I actually recommend using these if you are not too interested in the nitty-gritty details.

Local LLM with Retrieval-Augmented Generation

Toni Sagristà Sellés — Mon, 24 Mar 2025 00:00:00 +0000

Edit (2025-03-26): Added some words about next steps in conclusion.

Edit (2025-03-25): I re-ran the example with a clean database and the results are better. I also cleaned up the code a bit.

Over the past few months I have been running local LLMs on my computer with various results, ranging from ‘unusable’ to ‘pretty good’. Local LLMs are becoming more powerful, but they don’t inherently “know” everything. They’re trained on massive datasets, but those are typically static. To make LLMs truly useful for specific tasks, you often need to augment them with your own data–data that’s constantly changing, specific to your domain, or not included in the LLM’s original training. The technique known as RAG aims to bridge this problem by embedding context information into a vector database that is later used to provide context to the LLM, so that it can expand its knowledge beyond the original training dataset. In this short article, we’ll see how to build a very primitive local AI chatbot powered by Ollama with RAG capabilities.

The source code used in this post is available here.

Ingredients and set-up

In order to build our AI chatbot, we need the following ingredients:

Ollama, to facilitate access to different LLMs.
Chroma DB, our vector storage for our context.
Langchain, to facilitate the integration of our RAG application with the LLM.

First, we need to install Ollama. On Arch Linux, it is in the extra repository. If you happen to have an NVIDIA GPU that supports CUDA, you’re in luck. You can speed-up your inference but a significant factor! Otherwise, you can still just use the CPU. I have a (rather old) NVIDIA GPU, so I’m installing ollama-cuda as well.

pacman -S ollama ollama-cuda

Next, we need some LLM models to run locally. First, start the Ollama service.

systemctl start ollama.service

You can check whether the service is running with curl:

$ curl http://localhost:11434
Ollama is running%

Then, you need to get some LLM models. I have some installed that I use regularly:

$ ollama list
NAME ID SIZE MODIFIED
llama3.1:8b 46e0c10c039e 4.9 GB 2 hours ago
gemma3:12b 6fd036cefda5 8.1 GB 10 days ago
phi4-mini:latest 78fad5d182a7 2.5 GB 10 days ago
deepseek-coder-v2:16b 63fb193b3a9b 8.9 GB 10 days ago
llama3.2:1b baf6a787fdff 1.3 GB 4 weeks ago
mistral-small:24b 8039dd90c113 14 GB 7 weeks ago
phi4:latest ac896e5b8b34 9.1 GB 7 weeks ago

I would recommend using llama3.1:8b (link) for this experiment, especially if you run a machine with little RAM. It is quite compact and works reasonably well¹. Additionally, some models like gemma3:12b do not support embeddings, so you need to choose carefully.

Get llama3.1:8b with:

ollama pull llama3.1:8b

That’s it for our LLM. Now, let’s create a new directory for our RAG chatbot project, and let’s install the dependencies.

mkdir rag-chat && cd rag-chat
pipenv install ollama langchain langchain-ollama chromadb overrides

This should install python-ollama, Langchain together with the -ollama hook, and Chroma DB.

Ollama model setup

The code we display below is available in this repository. Now, let’s create a file, rag-chat.py.

First, our chatbot will list all the available models and will ask for the model we want to use. This part is simple:

#!python
from langchain_ollama import OllamaEmbeddings, OllamaLLM
import chromadb, requests, os, ollama

OLLAMA_URL = "http://localhost:11434"

models = ollama.list()
model_names = []
for m in models.models:
 model_names.append(m.model)

print("Available models:\n -", '\n - '.join(model_names))

# Ask the user the LLM model to use
llm_model = input("\nModel to use: ")

Vector storage

Chroma DB is a vector database designed and optimized for storing and searching vector embeddings, which are crucial in RAG. Vector embeddings are numerical representations of data (text, images, audio, etc.). They capture the meaning or semantic information of the data. They are generated by machine learning models (often transformer models like those used in LLMs). These models are trained to map similar data points to nearby points in a high-dimensional vector space. That’s precisely what we need to do with our context data for it to be useful to the LLM. And to that purpose we use Chroma DB.

So, we need to initialize the Chroma DB client with a persistent storage, and get it ready to embed our context data. We decide to persist our storage to disk, in the directory chroma_db/.

# Configure ChromaDB
# Initialize the ChromaDB client with persistent storage in the current directory
chroma_client = chromadb.PersistentClient(path=os.path.join(os.getcwd(), "chroma_db"))

# Custom embedding function for ChromaDB (using Ollama)
class ChromaDBEmbeddingFunction:
 """
 Custom embedding function for ChromaDB using embeddings from Ollama.
 """
 def __init__(self, langchain_embeddings):
 self.langchain_embeddings = langchain_embeddings

 def __call__(self, input):
 # Ensure the input is in a list format for processing
 if isinstance(input, str):
 input = [input]
 return self.langchain_embeddings.embed_documents(input)

# Initialize the embedding using Ollama embeddings, with the chosen model
embedding = ChromaDBEmbeddingFunction(
 OllamaEmbeddings(
 model=llm_model,
 base_url=OLLAMA_URL
 )
)

# Define a collection for the RAG workflow
collection_name = "rag_collection_demo"
collection = chroma_client.get_or_create_collection(
 name=collection_name,
 metadata={"description": "A collection for RAG with Ollama"},
 embedding_function=embedding # Embedding function we defined before
)

Provide context for retrieval

Once we have Chroma DB set up, with our collection and embedding function, we need to populate it. Here, we’ll just use an array with text, but you could fetch this information from files easily.

In my example, I use a totally made up text (1), the abstract to a paper about characterizing atmospheres with JWST (2), and some information about the new Gaia Sky website. The first is made up, so the model can’t know it. The second is a paper which came out much later than the model was published. The third is about the new Gaia Sky website, which was also created after the model.

# Function to add documents to the ChromaDB collection
def add_documents_to_collection(documents, ids):
 """
 Add documents to the ChromaDB collection.

 Args:
 documents (list of str): The documents to add.
 ids (list of str): Unique IDs for the documents.
 """
 collection.add(
 documents=documents,
 ids=ids
 )

# Example: Add sample documents to the collection
documents = [
 "The Mittius is a sphere of radius 2 that is used to disperse light in all
 directions. The Mittius is very powerful and sometimes emits light in various
 wavelengths on its own. It is a completely fictional object whose only purpose
 is testing RAG in a local LLM.",

 "The newly accessible mid-infrared (MIR) window offered by the James Webb Space
 Telescope (JWST) for exoplanet imaging is expected to provide valuable information to
 characterize their atmospheres. In particular, coronagraphs on board the JWST
 Mid-InfraRed instrument (MIRI) are capable of imaging the coldest directly
 imaged giant planets at the wavelengths where they emit most of their flux. The
 MIRI coronagraphs have been specially designed to detect the NH3 absorption
 around 10.5 microns, which has been predicted by atmospheric models. We aim to
 assess the presence of NH3 while refining the atmospheric parameters of one of the
 coldest companions detected by directly imaging GJ 504 b. Its mass is still a matter
 of debate and depending on the host star age estimate, the companion could either
 be placed in the brown dwarf regime or in the young Jovian planet regime. We present
 an analysis of MIRI coronagraphic observations of the GJ 504 system. We took advantage
 of previous observations of reference stars to build a library of images and to
 perform a more efficient subtraction of the stellar diffraction pattern. We detected
 the presence of NH3 at 12.5 sigma in the atmosphere, in line with atmospheric model
 expectations for a planetary-mass object and observed in brown dwarfs within a
 similar temperature range. The best-fit model with Exo-REM provides updated values
 of its atmospheric parameters, yielding a temperature of Teff = 512 K and radius of
 R = 1.08 RJup. These observations demonstrate the capability of MIRI coronagraphs to
 detect NH3 and to provide the first MIR observations of one of the coldest directly
 imaged companions. Overall, NH3 is a key molecule for characterizing the atmospheres
 of cold planets, offering valuable insights into their surface gravity. These
 observations provide valuable information for spectroscopic observations planned
 with JWST.",

 "Gaia Sky has a new website built with Hugo. It contains download pages for all new
 and old versions of the software, and a full listing of all the catalogs and datasets
 offered with the software. The datasets can be downloaded in-app with the provided
 dataset manager."

]
doc_ids = ["doc_mittius", "doc_paper_jwst", "doc_gaiasky_web"]

# Documents only need to be added once or whenever an update is required. 
# This line of code is included for demonstration purposes:
add_documents_to_collection(documents, doc_ids)

Finally, the chatbot logic

Now we have our vector storage set up. Let’s build the chat logic!

# Function to query the ChromaDB collection
def query_chromadb(query_text, n_results=3):
 """
 Query the ChromaDB collection for relevant documents.

 Args:
 query_text (str): The input query.
 n_results (int): The number of top results to return.

 Returns:
 list of dict: The top matching documents and their metadata.
 """
 results = collection.query(
 query_texts=[query_text],
 n_results=n_results
 )
 return results["documents"], results["metadatas"]

# Function to interact with the Ollama LLM
def query_ollama(prompt):
 """
 Send a query to Ollama and retrieve the response.

 Args:
 prompt (str): The input prompt for Ollama.

 Returns:
 str: The response from Ollama.
 """
 llm = OllamaLLM(model=llm_model)
 return llm.invoke(prompt)

# RAG pipeline: Combine ChromaDB and Ollama for Retrieval-Augmented Generation
def rag_pipeline(query_text):
 """
 Perform Retrieval-Augmented Generation (RAG) by combining ChromaDB and Ollama.

 Args:
 query_text (str): The input query.

 Returns:
 str: The generated response from Ollama augmented with retrieved context.
 """
 # Step 1: Retrieve relevant documents from ChromaDB
 retrieved_docs, metadata = query_chromadb(query_text)
 context = " ".join(retrieved_docs[0]) if retrieved_docs else "No relevant documents found."

 # Step 2: Send the query along with the context to Ollama
 augmented_prompt = f"Context: {context}\nQuestion: {query_text}\nAnswer: "
 # Uncomment next line if you want to see the context
 # print(augmented_prompt)

 response = query_ollama(augmented_prompt)
 return response

# User query loop
while True:
 query = input("Ask a question (or type 'exit' to quit): ")
 if query.lower() == "exit":
 break
 response = rag_pipeline(query)
 print(response)

That’s it! In this part, we ask the user to provide a query. Then, the script uses this query to fetch some context from our local Chroma DB, adds the context to the prompt (in augmented_prompt), and sends the query to Ollama. Then, we wait for the response and print it out.

Here is an example output. I ask it about JWST and exoplanets, the imaginary object ‘Mittius’, and the new Gaia Sky website. I have highlighted the response lines:

$ rag-chat.py
Available models:
 - llama3.1:8b
 - gemma3:12b
 - phi4-mini:latest
 - deepseek-coder-v2:16b
 - llama3.2:1b
 - mistral-small:24b
 - phi4:latest
Model to use: llama3.1:8b
Insert of existing embedding ID: doc_mittius
Insert of existing embedding ID: doc_paper_jwst
Insert of existing embedding ID: doc_gaiasky_web
Add of existing embedding ID: doc_mittius
Add of existing embedding ID: doc_paper_jwst
Add of existing embedding ID: doc_gaiasky_web

Ask a question (or type 'exit' to quit): JWST and exoplanets.
The James Webb Space Telescope (JWST) will play a significant role in exoplanet
research by providing new capabilities for characterizing their atmospheres.
The telescope\'s Mid-InfraRed instrument (MIRI) coronagraphs are capable of
imaging cold directly imaged giant planets and detecting specific molecules,
such as ammonia (NH3), which is a key indicator of surface gravity.

In the provided text, JWST is mentioned in the context of:

1. The newly accessible mid-infrared (MIR) window offered by JWST for exoplanet
 imaging.
2. The MIRI coronagraphs on board JWST are capable of detecting NH3 around
 10.5 microns.
3. Future spectroscopic observations planned with JWST will be guided by the
 valuable information obtained from these observations.

Overall, JWST is expected to provide new insights into exoplanet atmospheres
and help scientists refine their understanding of these distant worlds.

Ask a question (or type 'exit' to quit): What is the mittius?
A completely fictional object used to test Reasoning and Argument Generation
(RAG) in a local Large Language Model (LLM).

Ask a question (or type 'exit' to quit): What is the radius of the Mittius?
The answer can be found in the first sentence of the text:

"The Mittius is a sphere of radius 2 that is used to disperse light in all directions."

So, the radius of the Mittius is 2.

Ask a question (or type 'exit' to quit): How was the new Gaia Sky website built?
The new Gaia Sky website was built with Hugo.

Ask a question (or type 'exit' to quit): exit

As you can see, even this small 8B parameter model pretty much nails all three answer. In the third question (radius of the Mittius), it even provides the exact sentence that it sourced the subsequent answer from. Good work, Llama3.1!

Conclusion

As we’ve seen, with very little effort we can build a rudimentary RAG system on top of Ollama. This enables us to use context information in our queries in an automated manner, with the help of Chroma DB. This post only highlights the very basic notions required to get RAG to work in a very bare-bones manner. Further concepts to explore include different embedding models, quantization, system prompts, and more.

In our small test, we’ve used the Llama3.1 8B model, which is rather small. Using a larger model, like Gemma3 (12B), DeepSeek-R1 (14B), or even Mistral-small (24B), should improve the results at the expense of performance.

The code in this post is partially based on this medium article.

https://artificialanalysis.ai/models/llama-3-1-instruct-8b ↩︎

Gaia Sky gets new website

Toni Sagrista Selles — Wed, 19 Feb 2025 00:00:00 +0000

Over the past few weeks I’ve been working on a new standalone website for Gaia Sky. It uses the same technology stack as my personal website (essentially Hugo), so it is a static website generated from templates and content. This is enough for Gaia Sky. I am no graphic designer or UX person, but I tried my best to make it look potable.

The new Gaia Sky website, gaiasky.space

Why did I create a standalone website instead of keeping the old section in the ZAH site? A few reasons:

It is much easier to update and maintain.
We now have a news section to make announcements of new releases, datasets, and whatnot. People can follow it with RSS.
We can use our own domain, gaiasky.space. This was in the back of my mind for a long time.
We can verify our Flathub package. The impossibility to get Gaia Sky verified in Flathub is what prompted it all, as we obviously do not control the domain uni-heidelberg.de associated with the old app ID de.uni_heidelberg.zah.GaiaSky. The new ID space.gaiasky.GaiaSky is already verified.

As an extra perk, the new site allowed us to integrate things that were previously scattered in several places. I have written scripts to generate pages for:

gaiasky.space/downloads/releases – old releases, together with their packages and release notes.
gaiasky.space/resources/datasets – a full listing of the datasets available in Gaia Sky, with their descriptions and additional metadata.

The site is responsive and should render fine on PC and mobile. If you are interested, give it a visit: gaiasky.space.

Rendering volume aurorae and nebulae

Toni Sagrista Selles — Wed, 18 Dec 2024 00:00:00 +0000

A few weeks ago someone created an issue in the Gaia Sky Codeberg repository requesting the addition of aurorae to the Earth. They used as an example the aurora add-on in Cosmographia¹, which, at the time, looked to me like it was using some kind of billboard particle system to generate the effect. I never thought it looked particularly good for an aurora; I thought Gaia Sky could do better. So I set on a quest to implement a better looking aurora object in Gaia Sky. This wee quest would involve implementing three very different approaches to aurora rendering over the span of a few weeks, plus more than half a dozen 3D volume nebulae.

In this post, I present the three different methods I implemented to render aurorae. But before anything, I need to take a small detour and talk about nebulae.

Li’l nebulae detour

If every story should start at the beginning and be told in full, I must first relate how this little quest took a little detour, at the beginning, to explore the rendering of volume nebulae. The OP mentioned in the Codeberg issue that he also gets requests to fly into nebulae during his tours, and Gaia Sky only contained billboards so far. Since Gaia Sky already had the necessary infrastructure to render ray-marched volumes using a screen quad in a post-processing effect, I decided that addressing this issue first made sense, as it seemed to require a much lower effort and friction.

Typically, volumes are rendered using an in-scene model (normally a box) that encapsulates the volume fully. The fragments that hit the model faces are sent to through the pipeline, and the volume ray-marching or ray-tracing is implemented in the respective fragment shader. In our approach, we’ll use a post-processing stage, which means that we create a full screen quad and we run a fragment shader for every pixel in our screen or window. The main challenges of this approach are:

Ground the model in the scene world. Since we are essentially painting on an empty, flat canvas, we need to ground our 3D volume in our 3D world. This step is taken care of by carefully selecting the coordinate system and constructing the ray direction for each fragment and camera and object positions correctly.
Interaction with the depth buffer. We need our volume to be occluded by objects closer to the camera, and the opaque portions of our volume need to occlude further away parts of the scene. To fix this, we pass in the current depth buffer as a texture so that we can sample it at every pixel. Additionally, we also pass in the current scene frame buffer so that we have full control over the blending of our volume with the rest of the scene.

To get started, I browsed through shadertoy in search for a good implementation of a ray-marched shader that I could use as a base for my nebulae. I found several, like the Dusty Nebula 4 by Duke, and the Supernova Remnant, also by Duke. This Duke guy seems to be the most prolific shadertoy author when it comes to large-scale dusty objects, and by a large margin. It’s not even close, so props to him and thank you very much for your work. All of his shaders seem to be licensed under CC-BY-NC-SA 3.0, which is perfect for modification and reuse in Gaia Sky.

So, taking some of these shaders as a baseline, I created new ones for the Cat’s Eye nebula, the Hourglass nebula, the Trifid nebula, the Butterfly nebula, and the Crab nebula, among others. You can see some of them here.

My implementation of the Crab nebula as a volume. See it live on Shadertoy.

Once the Shadertoy shaders were ready, I did a little more work on the post-processing ray-marching infrastructure to accommodate the new members. So far, I had only ever used it for the black holes. The final step was translating the shaders from Shadertoy to Gaia Sky. This was not too difficult, as they were already created with that step in mind.

Anatomy of a nebula shader

As we mentioned, currently the nebulae are implemented in the post-processor. This means that they are rendered in image space after the main rendering stage, which renders the geometry, has finished.Post-processors get as input data the current frame buffer, the depth buffer, and whatever additional textures are needed for the particular effect (such as one or more noise textures). Additionally, they get positions and orientations as uniforms or varyings computed in the vertex stage.

All nebula shaders have the same main parts:

Set up – here we prepare the data for the raymarching step. The ray direction, the camera position and the object position are all computed/fetched here. We also recover the depth for the current fragment from the depth buffer. The set-up is typically done in the main() function. In the following code we have an example of how the set-up stage is done.

Code: set-up example

#version 330 core

// Color buffer.
uniform sampler2D u_texture0;
// Depth buffer (log).
uniform sampler2D u_texture1;
// Position of our camera with respect to the object to render.
uniform vec3 u_pos;

// Texture coordinates.
in vec2 v_texCoords;
// Ray from camera to fragment.
in vec3 v_ray;

// Output color.
layout(location = 0) out vec4 fragColor;

void main() {
 // Ray direction.
 vec3 rayDir = normalize(v_ray);
 // Camera position.
 vec3 camPos = u_pos;
 // Position of object. We always put the position of the object at the origin.
 vec3 objPos = vec3(0.0, 0.0, 0.0);
 // Sample depth buffer.
 float depth = texture(u_texture1, v_texCoords).r;
 depth *= length(rayDir);
 // Color of pre-existing scene.
 vec3 col = texture(u_texture0, v_texCoords).rgb;

 // Here would go the call to ray-marching.

 // Finally, the blending would go here.
}

Ray marching – here is where the actual ray-marching happens. In ray marching, we shoot a ray for every pixel, divide it into samples, and gather information at every step. This information is usually the density and color of the medium we are rendering.

Intersection check – since we implement this in image space (we shoot a ray for each pixel in our window), we first need to check whether our ray intersects the bounding volume containing the object or not. In nebulae, this volume is typically a sphere. Simply enough, this is done with a straightforward ray-sphere intersection function, which returns the intersection near and far distances if there is an intersection at all, or nothing if the ray does not hit the sphere. The code below shows how to do the ray-sphere intersection.
Code: ray-sphere intersection example
```
bool RaySphereIntersect(vec3 org, vec3 dir, out float near, out float far) {
 float b = dot(dir, org);
 float c = dot(org, org) - 8.0;
 float delta = b * b - c;
 if (delta < 0.0)
 return false;
 float deltasqrt = sqrt(delta);
 near = -b - deltasqrt;
 far = -b + deltasqrt;
 return far > 0.0;
}
```

Main loop – here we loop over a maximum number of iterations and accumulate the density. Every cycle advances our position through the ray by a given step distance. The code below illustrates a rather typical main loop. The following code completes the main() function to include the call to the main loop (raymarching()) and the alpha blending at the very bottom.

Code: main loop example

#define ITERATIONS 50

vec4 raymarch(vec3 camPos, vec3 rayDir, vec3 objPos, float depth) {
 // Ray direction.
 vec3 rd = rayDir;
 // Ray origin.
 vec3 ro = camPos;

 // ld, td: local, total density
 // w: weighting factor
 float ld = 0.0, td = 0.0, w = 0.0;

 // t: length of the ray
 // d: distance function
 float d = 1.0, t = 0.0;

 const float h = 0.1;

 vec4 sum = vec4(0.0);

 float min_dist = 0.0, max_dist = 0.0;

 // If our ray intersects the sphere...
 if (RaySphereIntersect(ro, rd, min_dist, max_dist)) {
 t = min_dist * step(t, min_dist);

 // Raymarch loop
 for (int i = 0; i < ITERATIONS; i++) {
 vec3 pos = ro + t * rd;

 // Loop break conditions.
 // Here we check the current distance agains the depth and the
 // maximum distance.
 if (t > depth || t > max_dist) break;

 // Evaluate distance function.
 // The map function returns the density at a given position.
 float d = map(pos);

 // Point light calculations.
 vec3 ldst = vec3(0.0) - pos;

 // Here, the color of light depends on the distance from the center
 // of the object. Typically, this is more complicated than what's here.
 vec3 lightColor = mix(colorA, colorB, ldst);

 if (d < h) {
 // Compute local density.
 ld = h - d;

 // Compute weighting factor.
 w = (1.0 - td) * ld;

 // Accumulate density.
 td += w + 1.0 / 200.0;

 vec4 col = vec4(0.3, 0.3, 0.3, td);

 // Emission.
 sum += sum.a * vec4(sum.rgb, 0.0) * 0.2;

 // Uniform scale density.
 col.a *= 0.2;
 // Colour by alpha.
 col.rgb *= col.a;
 // Alpha blend in contribution.
 sum = sum + col * (1.0 - sum.a);
 }
 td += 1.0 / 90.0;

 // Optimize step size near the camera.
 t += max(d * 0.1 * max(min(length(ldst), length(ro)), 1.0), 0.01);
 }

 // Simple scattering.
 sum *= 1. / exp(ld * 0.2) * 0.4;
 sum = clamp(sum, 0.0, 1.0);
 sum.xyz = sum.xyz * sum.xyz * (3.0 - 2.0 * sum.xyz);
 }
 return vec4(sum.xyz, td);
}

Blending – finally, we have the blending. Most nebulae are blended using regular alpha blending, but implementing additive blending is trivial.

Code: alpha blending example

void main() {
 vec3 rayDir = normalize(v_ray);
 vec3 camPos = u_pos * 2.0e-7;
 vec3 objPos = vec3(0.0, 0.0, 0.0);
 float depth = texture(u_texture1, v_texCoords).r;
 depth *= length(rayDir);
 vec3 col = texture(u_texture0, v_texCoords).rgb;

 // Ray marching.
 vec4 rmcol = raymarch(camPos, rayDir, objPos, depth);

 // Alpha blending.
 fragColor = vec4(col * (1.0 - rmcol.a) + rmcol.rgb * rmcol.a, 1.0);
}

This covers the basics of our nebulae shaders. The video below shows some of the nebulae that I have adapted for Gaia Sky and that will be available in the NGC2000 catalog when the next version is out, soon™.

Your browser does not support the video tag.

Six of the new nebulae implemented in Gaia Sky.

Now, back to the aurora.

Approaching aurora rendering

Aurorae are natural light displays caused by interactions between charged particles from the Sun (solar wind) and the Earth’s magnetosphere and atmosphere. At the poles, magnetic field lines converge and funnel these particles into the upper atmosphere (thermosphere), where they collide with oxygen and nitrogen molecules. The collisions excite the molecules, causing electrons to jump to higher energy states. The excited atoms are unstable, so as the electrons return to their normal states, they release packets of energy in the form of photons. The color (frequency) of these photons depends on the type of gas and altitude: green from oxygen (~100-150 km), red from oxygen higher up, where lower density causes slower de-excitation (~200+ km), and blue or purple from nitrogen. The shifting, curtain-like shapes result from changes in the solar wind and Earth’s magnetic field.

A picture of an aurora borealis in Greenland. The green, red and blue colors can be seen in different parts of the curtains. Photo by Visit Greenland on Pixnio, CC0.

Aurorare are, then, purely volumetric and perfectly emissive phenomena. This fact strongly points to a volume ray-caster being the best method for rendering aurorare, but implementing volume rendering shaders is hard, so I explored some alternative options first. I have very creatively labeled them 1, 2 and 3.

1. Mesh w/ custom shader

I like to simplify problems to their very basics. Even though aurora curtains have volume (width), their extent is limited when compared to their height or their footprint length. So, obviously, my first thought was to implement aurorae as a curved mesh with a special shader to render the curtain elements. I knew it wouldn’t look super good, but maybe it was good enough.

I set up the infrastructure and created the object. At first, I used an uncapped cylinder mesh. The faces are unculled, since they need to be visible from both sides. Once the mesh was in place, I created the fragment shader which would render the actual emissive elements. This shader is available on Shadertoy: Base Aurora Shader.

I loaded the shader and applied it to the mesh. The results were… underwhelming. The shader itself is ok, but when applied to the flat surface of the cylinder, it just does not cut it. When the camera moves the effect is totally shattered and you can tell right away that this is a flat object.

The aurora shader on a cylindrical mesh, seen from afar.

A close-up. Meh.

It was clear to me that this wasn’t good enough. So I went browsing on shadertoy again.

2. Isolated noise aurora

I’ll start by saying that this second effort went nowhere, but had the pleasant side effect of having implemented the true in-scene, bounding box-based volume infrastructure.

The base idea was to adapt a pre-existing ray-marching shader to be rendered as a volume in Gaia Sky. The shader in question is here.

The isolated aurora shader ray-marches an aurora using fractal brownian motion noise as footprints. Looks good in shadertoy, not so much in Gaia Sky. See it live on Shadertoy.

That shader contains some code to to intersect the ray with an axis-aligned bounding box (AABB), so naturally I implemented the necessary infrastructure to be able to ship external volume rendering shaders on bounding meshes. And it kind of worked, but I could never make it behave exactly the way I wanted. I had several problems with the integration in Gaia Sky, so instead of spending lots of time and effort into solving them only to end up with a half-assed, bad-looking aurora, I decided to try the next approach.

3. True volume aurora

This is the good one. It is based on Lawlor and Genetti’s 2011 paper “Interactive Volume Rendering Aurora on the GPU”². Key innovations include a height-based aurora model, a GPU-friendly atmosphere model, efficient ray marching with distance fields, and interactive frame rates (20–80 FPS). The aurora footprints are initially represented as splines, which are then fed into a stem-like fluid simulator to generate the true footprint textures. Then, they sample the rays and stretch the footprints in the vertical direction using a physically-based vertical electron deposition function. They also accelerate the ray-marching step using a 2D signed distance field (SDF) texture generated from the footprint.

I implemented all of this and the results were quite good. I drew the footprints myself using Gimp. I use cubemaps for the footprints and the signed distance function texture, as they eliminate the artifacts at the poles typical of regular UV mapping on spheres. The aurora volume itself is modeled with a flipped sphere that wraps around the Earth. I generate it programmatically.

I generated the SDF textures with sdfer, a small CLI utility that generates distance fields from raster images.

Left: the footprint texture, created with Gimp. Center: the signed distance function texture. Right: the final render.

By far, the largest problem I faced was in the UV coordinate mapping. In order to sample the 2D textures from the 3D samples in the ray marching steps, I implemented a projection function that maps 3D points to 2D UV coordinates.

vec2 uv(vec3 p, vec3 rayDir, vec3 camPos) {
 p = normalize(p);
 return vec2(0.5 - atan(-p.z, -p.x) / (PI2), asin(-p.y) / PI + 0.5);
}

However, when the camera was inside the volume, this gave me mirrored ghosting aurorae in their opposite direction relative to the camera. I struggled to solve this issue for a while, until I realized that I just had to enforce that the sampling point had to be on the right side of the ray:

vec2 uv(vec3 p, vec3 rayDir, vec3 camPos) {
 if (dot(rayDir, p) > 0.000) {
 // Ok.
 p = normalize(p);
 return vec2(0.5 - atan(-p.z, -p.x) / (PI2), asin(-p.y) / PI + 0.5);
 } else {
 // Invalid result.
 return vec2(5.0, 5.0);
 }
}

This solved the artifacts. I then implemented some additional suggestions by O. Lawlor (the paper’s author), and I got it to a state where I really liked it. See the video below:

Your browser does not support the video tag.

The final volume aurora in Gaia Sky.

Future work

A few things can be improved in the current aurora:

Render the aurorae to an off-screen buffer. This would unlock a series of benefits:
- Render at lower resolution to increase performance.
- Apply blur and glow to mitigate jittering artificts.
- Full control over blending with scene.
Proper color modeling. The base and high colors are now passed in from a JSON descriptor file. This could be improved by adopting a physical model of the upper atmosphere and generating the colors accordingly.

Conclusion

In this post, I have shown how the new volume nebulae and aurorae in Gaia Sky came about. I have given some implementation details, and discussed their pros and cons. In the end, I think the time spent on this was well spent, as the results were quite satisfactory.

SPICE-Enhanced Comosgraphia Mission Visualization Tool, NAIF: https://naif.jpl.nasa.gov/naif/cosmographia.html ↩︎
Interactive Volume Rendering Aurora on the GPU, Journal of WSCG: https://www.researchgate.net/publication/220200705_Interactive_Volume_Rendering_Aurora_on_the_GPU ↩︎

Langur Monkey - Toni Sagristà Sellés

GGUF quantization guide

Naming conventions

Unquantized formats

Legacy quantization formats (Q*_0, Q*_1)

K-Quant formats (modern default)

I-Quant formats (aggressive compression)

Special formats

Decision guide

Conclusions

LM Studio on systemd linger

Yet another architectural update for Play Kid

The debug panel, showing the machine state and a code disassembly.

Play Kid with the top menu bar and the ‘Open ROM’ menu entry.

Game Boy emulator tech stack update

Debug panel

The new integrated debugger features a real-time disassembly view and breakpoint management.

Playing around with the new Play Kid UI based on egui.

Dependency hell

Conclusion

Play Kid

Play Kid running different games with different color palettes.

Downloads

Build

Run

Operation

Debug panel

The debug panel, showing the machine state and a code disassembly.

Playing around with the Play Kid debug UI.

CLI args

SDL2 version

The debug mode.

Useful links

Implementing a Game Boy emulator

Me playing Super Mario Land, levels 2 to 4, with my own emulator.

Foreword

Game Boy essentials

The CPU

Memory

The PPU

The APU

Cartridges

UI and debug mode

Future work

Conclusions

Google *unkills* JPEG XL?

Let’s recap

Chromium’s new stance

About JPEG XL

Conclusion

Motion trails in Gaia Sky

Vertex-based stretching of stars in supraluminal travel in Gaia Sky.

Traditional approach

Our approach

The shader

0. Initial orientation

1. Rotation and velocity projection

2. Estimate screen-space movement

3. Calculate trail stretch amount

4. Distance-based fadeout

5. Apply trail effect

6. Final assembly

Results

Conclusion

Quadruple joins the fight!

Enter Quadruple

Results

Addition

Three-way Addition results – Interactive view

Subtraction

Three-way Subtraction results – Interactive view

Multiplication

Three-way Multiplication results – Interactive view

Division

Three-way Division results – Interactive view

Allocation (from string)

Three-way Allocation results (from string) – Interactive view

Allocation (from double)

Three-way Allocation results (from double) – Interactive view

Analysis

Legacy quantization formats (`Q_0`, `Q_1`)

Google unkills JPEG XL?