Status: [LIVE]
Endpoints found:
https://huggingface.co/AkshithAI/project-828-gpt
Source: https://github.com/AkshithAI/project_828
# Project 828 - MoE Transformer
[](https://www.python.org/downloads/)
[](https://pytorch.org/)
[](https://huggingface.co/AkshithAI/project-828-gpt-base)
[](src/kernels/)
[](https://www.deepspeed.ai/)
[](LICENSE)
**Mixture-of-Experts Transformer with Custom Triton Kernels & Advanced Training Pipeline**
A Mixture-of-Experts (MoE) transformer implementation featuring custom Triton CUDA kernels, Grouped Query Attention, RoPE positional encoding with YaRN extension, Gigatoken Rust/SIMD tokenization, Liger-Kernel integration, and distributed training support.
[Features](#features) • [Model Architecture](#model-architecture) • [🤗 HF Model Release](#3987m-model-configuration-baseline--v1--hugging-face-release) • [Custom Triton Kernels](#custom-triton-kernels) • [Quick Start](#quick-start) • [Training](#training) • [Results](#training-experiments--results)
---
- **Model Release (400M Base)**: [🤗 Hugging Face - `AkshithAI/project-828-gpt-base`](https://huggingface.co/AkshithAI/project-828-gpt-base)
- **Training Report** : [Phase 1 Pretraining H200 W&B Report](https://wandb.ai/akshithmarepally-akai/828_pretraining_h200/reports/Phase-1-Pretraining-H200:-Training-Dynamics-and-MoE-Routing--VmlldzoxNzQ4MTcwOA==)
## Table of Contents
- [Features](#features)
- [Model Architecture](#model-architecture)
- [Architecture Overview](#architecture-overview)
- [828M Model Configuration (Current Target)](#828m-model-configuration-current-target)
- [398.7M Model Configuration (Baseline