[LIVE] [V1jey04/gpt2-124m-from-scratch] 2 endpoint(s)
ChatGPT
API Proxy/ChatGPT
2,234 characters
Status: [LIVE]
Endpoints found:
https://huggingface.co/spaces/V1jey04/gpt
https://huggingface.co/V1jey04/gpt
https://huggingface.co/spaces/V1jey04/gpt2-124m-from-scratch)**
Source: https://github.com/V1jey04/gpt2-124m-from-scratch
# GPT-2-size chat model, built from scratch
A 124M-parameter language model and its tokenizer, written from scratch and trained on Kaggle's
free T4 GPUs for $0. It was pretrained on 2.42B tokens of educational web text, then fine-tuned
on 276M tokens of conversations, and it now chats on a laptop CPU.
**[Try it in your browser](https://huggingface.co/spaces/V1jey04/gpt2-124m-from-scratch)** (runs on your device, nothing to install) ·
**[Model weights and card](https://huggingface.co/V1jey04/gpt2-124m-from-scratch)**
<img src="docs/tokenizer.gif" width="360" alt="Animation: the bytes of 'What is photosynthesis?' merge into four tokens">
No `transformers` or `tokenizers` library is used to build or train the model. The tokenizer is
standard-library Python, and the model is plain PyTorch.
| Part | File | What it does |
|---|---|---|
| Tokenizer | `tokenizer/bpe.py` | Byte-level BPE: 50,000 learned pieces plus 4 chat tokens |
| Model | `model/gpt.py` | GPT-2-small shape: attention, MLP, blocks, loss, KV-cached sampling |
| Training | `train.py` | Two-GPU DDP, fp16, gradient accumulation, exact checkpoint and resume |
| Data | `data/` | Streams, tokenizes and shards FineWeb-Edu and the chat datasets |
| Chat | `chat.py` | Terminal chat with streaming output |
| Browser demo | `hf_release/web/` | ONNX Runtime Web + a JavaScript port of the tokenizer, KV cache in a web worker |
| Benchmarks | `benchmark.py` | Zero-shot comparison against OpenAI's GPT-2 124M |
## Results

| Stage | Data | Compute | Validation loss |
|---|---|---|---|
| Pretraining | 2.42B tokens of FineWeb-Edu (9,236 steps × 262,144 tokens) | 2× T4, 22.6 h, two Kaggle sessions | 10.96 → 3.297 |
| Chat fine-tuning | 276M tokens of SmolTalk, Dolly and OpenAssistant (4,217 steps) | 2× T4, 2 h 40 min | 2.74 → 1.445 |
### Against OpenAI's GPT-2 (124M)

| Test