3,555 prompts from top AI collections
← Back to Market
[LIVE] [V1jey04/gpt2-124m-from-scratch] 2 endpoint(s)
ChatGPT API Proxy/ChatGPT
2,234 characters
Status: [LIVE] Endpoints found: https://huggingface.co/spaces/V1jey04/gpt https://huggingface.co/V1jey04/gpt https://huggingface.co/spaces/V1jey04/gpt2-124m-from-scratch)** Source: https://github.com/V1jey04/gpt2-124m-from-scratch # GPT-2-size chat model, built from scratch A 124M-parameter language model and its tokenizer, written from scratch and trained on Kaggle's free T4 GPUs for $0. It was pretrained on 2.42B tokens of educational web text, then fine-tuned on 276M tokens of conversations, and it now chats on a laptop CPU. **[Try it in your browser](https://huggingface.co/spaces/V1jey04/gpt2-124m-from-scratch)** (runs on your device, nothing to install) · **[Model weights and card](https://huggingface.co/V1jey04/gpt2-124m-from-scratch)** <img src="docs/tokenizer.gif" width="360" alt="Animation: the bytes of 'What is photosynthesis?' merge into four tokens"> No `transformers` or `tokenizers` library is used to build or train the model. The tokenizer is standard-library Python, and the model is plain PyTorch. | Part | File | What it does | |---|---|---| | Tokenizer | `tokenizer/bpe.py` | Byte-level BPE: 50,000 learned pieces plus 4 chat tokens | | Model | `model/gpt.py` | GPT-2-small shape: attention, MLP, blocks, loss, KV-cached sampling | | Training | `train.py` | Two-GPU DDP, fp16, gradient accumulation, exact checkpoint and resume | | Data | `data/` | Streams, tokenizes and shards FineWeb-Edu and the chat datasets | | Chat | `chat.py` | Terminal chat with streaming output | | Browser demo | `hf_release/web/` | ONNX Runtime Web + a JavaScript port of the tokenizer, KV cache in a web worker | | Benchmarks | `benchmark.py` | Zero-shot comparison against OpenAI's GPT-2 124M | ## Results ![Validation loss during pretraining and chat fine-tuning](docs/training.png) | Stage | Data | Compute | Validation loss | |---|---|---|---| | Pretraining | 2.42B tokens of FineWeb-Edu (9,236 steps × 262,144 tokens) | 2× T4, 22.6 h, two Kaggle sessions | 10.96 → 3.297 | | Chat fine-tuning | 276M tokens of SmolTalk, Dolly and OpenAssistant (4,217 steps) | 2× T4, 2 h 40 min | 2.74 → 1.445 | ### Against OpenAI's GPT-2 (124M) ![Benchmark bars: this model vs GPT-2](docs/benchmarks.png) | Test
Download .txt