[LIVE] [yashgarg4/nanogpt-tinystories] 1 endpoint(s)
Gemini
API Proxy/Gemini
2,210 characters
Status: [LIVE]
Endpoints found:
https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/yashgarg4/nanogpt
Source: https://github.com/yashgarg4/nanogpt-tinystories
# nanogpt-tinystories
**A ~30M-parameter GPT-2-style transformer, built from scratch in pure PyTorch and
trained on TinyStories until it writes coherent little children's stories.**




[](https://colab.research.google.com/github/yashgarg4/nanogpt-tinystories/blob/main/notebooks/train_colab.ipynb)
No `transformers` library, no `AutoModel`, no pretrained weights. Every core
component — tokenization, embeddings, causal self-attention, the transformer
block, the training loop, and sampling — is implemented **and explained** by hand.
The goal is **understanding transformers from the inside**. The code is heavily commented to match.
## What it does
Trained from **random initialization** on TinyStories, it continues a prompt like
`"Once upon a time"` into a short, mostly-coherent story — a real result from this
repo's model (val loss 1.88, temperature 0.8, top-k 200):
> Once upon a time, there was a little girl named Lily. She loved to play with her
> jewelry and pretend she was a princess. One day, she went to the park to play
> with her jewelry. But when she saw her toy box, a big dog tried to take it from
> her. Lily was sad and started to cry. Her mommy told her not to worry and that
> the dog was not a real friend... Later that day, Lily's friend came over to play.
> Lily was so happy to have her shiny jewelry box back. She promised to always be
> careful and respectful with it. From that day on, Lily and her friends played
> with her jewelry box, and the dog never grabbed it again.
More in [`samples/`](samples/): the same model at different temperatures
([`gallery.md`](samples/gallery.md))