Status: [LIVE] Endpoints found: https://abubakarsiddik31.github.io/llm-from-scratch/](https://abubakarsiddik31.github.io/llm Source: https://github.com/abubakarsiddik31/llm-from-scratch
# Hands-On LLM Implementation ### Build everything from scratch. Ship to production. **A practical, implementation-first guide to training and deploying Large Language Models** [![Python](https://img.shields.io/badge/Python-3.10+-blue.svg)](https://www.python.org/) [![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-red.svg)](https://pytorch.org/) [![License](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE) πŸ“– **Read it as a book:** [https://abubakarsiddik31.github.io/llm-from-scratch/](https://abubakarsiddik31.github.io/llm-from-scratch/) *(built with mdBook from [`book/`](book/))* [Projects](#-roadmap) β€’ [Getting Started](#-getting-started) β€’ [Directory Structure](#-directory-structure)
--- ## Overview This roadmap is for people who **understand the theory** and want to **build**. You'll implement every component of an LLM pipeline from scratch: > **Tokenization β†’ Pre-training β†’ Fine-tuning β†’ Optimization β†’ Production Deployment** **Philosophy:** learn by doing. Each project builds on the previous one, and the last one is a fully deployed LLM application. --- ## Prerequisites | Requirement | Details | |-------------|---------| | **GPU** | CUDA-capable (8GB+ VRAM recommended) | | **Python** | 3.10+ | | **Env** | `uv`, `conda` or `mamba` | | **Knowledge** | Neural networks & attention mechanism (theoryβ€”we'll implement the rest) | --- ## Roadmap
### 10 Phases β€’ 32 Projects β€’ From Zero to Production
### Phase 1 β€” Foundation: Build Your First LLM | Project | Topic | Status | |---------|-------|--------| | **1** | Character-Level GPT | βœ… Complete | | **2** | BPE Tokenizer | βœ… Complete | | **3** | Contextual Embeddings (BERT-style) | βœ… Complete | | **3B** | SimCSE (Sentence Embeddings) | βœ… Complete | | **4** | Pre-train 125M Model | βœ… Complete | **Focus:** Multi-head attention, Transformer blocks, Training loop, Text generation, Masked Language M