micro_llm

This project is a demonstration of technical exploration in building a minimalistic language model (LLM) from scratch. The focus is on breaking down abstractions and dependencies to reveal the foundational algorithms and processes behind language models. Through this work, I aimed to showcase skills in designing neural networks, implementing custom layers, and working with minimal dependencies while solving real-world challenges like handling large datasets (Project Gutenberg, Civ5 Civilopedia).

One of the hidden challenges of building a language model from scratch is the underlying software on which it is built. This project aims to demystify the architecture of LLMs by reducing reliance on pre-built frameworks and libraries, at the same time providing a stable skeleton for more technical and mathematical exploration. This keeps the focus on the core learning workflow rather than more generic software engineering.

Key Highlights:

• Implementation of core neural network components, such as custom linear layers and backpropagation, without relying on high-level abstractions.
• Modular design for easy experimentation with optimization algorithms and custom network layers.
• Integration of external datasets and workflows for advanced text generation purposes.
• Emphasis on understanding the underlying architecture of LLMs by reducing reliance on pre-built frameworks.

Core Features:

• Modular design for easy experimentation with language modeling concepts.
• Built-in utilities for exploring dataset integration and custom neural network layers.
• Focused on incrementally demystifying how language models function from basic mathematical principles.

Workflow & Usage:

• Scripts available in the run/ directory demonstrate specific workflows:
  • Execute any script with python3.10 -m run.<script_name>.
• Optimization algorithms are replaceable and fully customizable for exploration and educational purposes.

Technical Stack:

• Python 3.10 as the foundation for all development tasks.
• Minimal dependency usage (numpy, pandas, sqlalchemy, torch, etc.) to underscore efficiency and focus on core algorithms.
• Compatibility ensured for modern hardware with CUDA support (torch==1.13.1+cu117 for GPU acceleration).