LLM training dynamics & data influence
What is LLM training dynamics & data influence?
This trend covers the techniques and processes that shape how language models learn from data and get modified after training, including fine-tuning methods, reinforcement learning approaches, and ways to influence model behavior without retraining from scratch.
Understanding how models are trained and adapted is critical for evaluating their reliability, controlling their behavior, and assessing whether improvements require expensive retraining or can be achieved through cheaper modifications.
References
- Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning — ArXiv
- From Spectra to Joint Schedules in LLM Pre-training: 3+3(+2) Scaling-Law Regimes — ArXiv
- Policy Gradient for LLMs, Explained Visually — Hacker News
- Pretraining Latent Information Feedback Transformers with Teacher Supervision — ArXiv
- AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation — ArXiv
- Alpha Diffusion Language Models: Factorization Alone Is Not the Problem — ArXiv
- Topic Modeling in the Era of Embeddings and LLMs — Medium: Large Language Models
- What and Whose Knowledge? Measuring Epistemic Diversity in Large Language Models — Hacker News