Status: [LIVE] Endpoints found: https://huggingface.co/AkshithAI/project-828-gpt Source: https://github.com/AkshithAI/project_828
# Project 828 - MoE Transformer [![Python 3.8+](https://img.shields.io/badge/python-3.8+-blue.svg)](https://www.python.org/downloads/) [![PyTorch](https://img.shields.io/badge/PyTorch-2.0+-ee4c2c.svg)](https://pytorch.org/) [![Hugging Face](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-project--828--gpt--base-yellow.svg)](https://huggingface.co/AkshithAI/project-828-gpt-base) [![Triton Kernels](https://img.shields.io/badge/Triton_Kernels-Enabled-orange.svg)](src/kernels/) [![DeepSpeed](https://img.shields.io/badge/DeepSpeed-Enabled-green.svg)](https://www.deepspeed.ai/) [![License](https://img.shields.io/badge/License-TBD-lightgrey.svg)](LICENSE) **Mixture-of-Experts Transformer with Custom Triton Kernels & Advanced Training Pipeline** A Mixture-of-Experts (MoE) transformer implementation featuring custom Triton CUDA kernels, Grouped Query Attention, RoPE positional encoding with YaRN extension, Gigatoken Rust/SIMD tokenization, Liger-Kernel integration, and distributed training support. [Features](#features) • [Model Architecture](#model-architecture) • [🤗 HF Model Release](#3987m-model-configuration-baseline--v1--hugging-face-release) • [Custom Triton Kernels](#custom-triton-kernels) • [Quick Start](#quick-start) • [Training](#training) • [Results](#training-experiments--results) --- - **Model Release (400M Base)**: [🤗 Hugging Face - `AkshithAI/project-828-gpt-base`](https://huggingface.co/AkshithAI/project-828-gpt-base) - **Training Report** : [Phase 1 Pretraining H200 W&B Report](https://wandb.ai/akshithmarepally-akai/828_pretraining_h200/reports/Phase-1-Pretraining-H200:-Training-Dynamics-and-MoE-Routing--VmlldzoxNzQ4MTcwOA==)
## Table of Contents - [Features](#features) - [Model Architecture](#model-architecture) - [Architecture Overview](#architecture-overview) - [828M Model Configuration (Current Target)](#828m-model-configuration-current-target) - [398.7M Model Configuration (Baseline