Category: Rust

In my re-implementation of the Gaia Sky level-of-detail (LOD) catalog generation in Rust I have been able to roughly halve the processing time, and, even though I do not have concrete numbers yet, everything points towards a drastic decrease in memory usage as well. In this project, I need to read a metric ton of gzipped csv Gaia catalog files, parse and process them into a functional in-memory catalog with cartesian positions, velocity vectors, RGB colors, etc. Then I need use them to generate an octree that represents the LOD structure, and finally write another metric ton of binary files back to disk. Using memory mapped files helps a lot in avoiding copies and speeding up the reading and writing operations; that’s something I tried out in the Java version and have come to also re-implement in Rust. Here’s the thing though: working with memory mapped files in Java is super straightforward. In Rust? Not so much. And the lack of available documentation and examples does not help. I was actually unable to find any working snippets with all the parts I needed, so I’m documenting it in this post in case someone else is in the same situation I was.

Lately I’ve taken an interest in the Rust programming language. I read the epub version of the Rust book over summer and found it quite well-written and overall interesting. Recently, I got myself a hard copy that sits on my desktop for quick queries, as I’ve been re-implementing the Gaia Sky catalog generation in Rust. The reasons for this are varied, but basically they come down to hardware. We need to load a humongous amount of stars (~1.6 billion) into memory, our old server, which had 2 TB of RAM, is being replaced by a newer, faster, and smaller (memory-wise, ~700 GB) machine in which we won’t be able to generate with the current Java implementation. But this is a story for another post. Today I’m here to document the completely mouseless Rust development environment I’ve been putting together as a sort of side project, to aid in my re-implementation.