OpenAI vs. Deepseek vs. Qwen: Comparing Open Source LLM Architectures

TL;DR

The video compares architectural differences among three open-weight LLMs: GPT-OSS (Mixture of Experts, 120B/20B parameters, 131k token context via YARN applied during pretraining), Qwen3 (dense and MoE variants, 36T pretraining tokens, 32k context extended to 128k via inference-time YARN scaling), and DeepSeek V3/V3.

Key insights

• GPT‑OSS applies YARN (rope frequency scaling) during pretraining so its weights natively support 131k token contexts; DeepSeek trains context length in stages (32k → 128k) via finetuning; Qwen3 trains
• DeepSeek V3 uses Multi‑head Latent Attention (MLA) – compresses keys/values into a latent space before caching, then decompresses during inference – delivering greater memory savings and modeling perf
• Qwen3 replaces the QKV bias (static offset) with QK Norm (dynamic rescaling of query/key vectors) to keep attention scores stable at scale.