Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws

Scaling laws describe the relationship between the size of language models and their capabilities. Unlike prior studies that evaluate a model's capability via loss or benchmarks, we estimate the number of knowledge bits a model stores. We focus on factual knowledge represented as tuples, such as (USA, capital, Washington D.C.) from a Wikipedia page. Through multiple controlled datasets, we establi

[stealthinu]

LLMのパラメータと記憶についての結構重要そうなまとめの論文の話。attentionが記憶しているっぽいとか良いデータと悪いデータ混在してても良いデータにマーク付けるとよいとか色々。