DeepSpeed

Model Setup

Training API

Inference API

Checkpointing API

ZeRO API

Mixture of Experts (MoE)

Transformer Kernel API

Pipeline Parallelism

Optimizers

Learning Rate Schedulers

Flops Profiler

Autotuning

Memory Usage

Indices and tables