(GPT-)[https://github.com/evisdrenova/gpt-rs]rs is a rust implementation of a large language model, specifically GPT-3. I built GPT-rs while working through Sebastican Raschka's Build a Large Language Model from Scratch.
The Build a Large Language Model from Scratch book is implemented in Python but I thought it would be more fun to do it in Rust.
The implementation covers the major parts of an LLM integration. Specifically, pre-training, the LLM architecture and post-training/fine-tuning.
In conjunction with the project, I also wrote a blog detailing each step of the process with code examples. You can find that (here)[https://github.com/evisdrenova/gpt-rs/blob/main/blog.md].
I had two motivations for doing this project:
- Get a deeper understanding of LLM's work by implementing one myself
- Refine my rust skills
I think I was able to accomplish both.
It's actually amazing to understand the math and inner workings of an LLM and then see it work in production and how accurate it is (despite not always being that accurate). At the same time, it's clear that we are so far away from AGI or any other general intelligence.
LLMs truly are next token prediction mechanisms trained on input-target pairs of sentences.
But maybe that's all we are too?