Deep Learning ideas that have stood the test of time

Deep Learning is such a fast-moving field and the huge number of research papers and ideas can be overwhelming. The goal of this post is to review ideas that have stood the test of time. These ideas, or improvements of them, have been used over and over again. They’re known to work. If you were to start in Deep Learning today, understanding and implementing each of these techniques would probably

[Nyoho]

「時の試練に耐えてきた、よく採用されたアイデア」のレビュー

[misshiki]

2012：AlexNetとドロップアウト、2013：深層強化学習、2014：Attention＋seq2seq、2014：Adam最適化、2014/2015：GAN、2015：ResNet、2017：Transformers、2018：BERT、2019/2020～：巨大言語モデル（GPT-3など）、自己教師あり学習（SimCLRなど）？

[knok]

活性化関数についての言及がないなあ

[sh19e]

“The clearest trend throughout the history of Deep Learning is perhaps that of the bitter lesson. Algorithmic advances for better parallelization (= more data) and more model parameters win over "smarter techniques" again and again. This trend seems to continue well into 2020 where GPT-3,”

[stealthinu]

ディープラーニング関連の論文で振り返って特に重要だったものだけピックアップ。Dropout/深層強化学習/Attention/Adam/GAN/ResNet/Transformer/BERT/GPT-3と納得いく一覧だがBNが無い。CNNはこのリスト以前か。

[nikutaiha]

deep learning

[chck1245]

分野のメインストリーム、読むべき論文集

[bluescreen]

こういうまとめはありがたいですね。今後も発展していくだろうから追記していってほしいな。

[yukisno]

G検定対策になりそう