DataLoader With Multiple Workers Leaks Memory PyTorch Developer podcast

Artwork

Tech PyTorch Edward Yang Team PyTorch Deep Learning Machine Learning

Indhold leveret af PyTorch, Edward Yang, and Team PyTorch. Alt podcastindhold inklusive episoder, grafik og podcastbeskrivelser uploades og leveres direkte af PyTorch, Edward Yang, and Team PyTorch eller deres podcastplatformspartner. Hvis du mener, at nogen bruger dit ophavsretligt beskyttede værk uden din tilladelse, kan du følge processen beskrevet her https://da.player.fm/legal.

PyTorch Developer Podcast « »
DataLoader with multiple workers leaks memory

3y ago 16:38

Del

MP3•Episode hjem

Indhold leveret af PyTorch, Edward Yang, and Team PyTorch. Alt podcastindhold inklusive episoder, grafik og podcastbeskrivelser uploades og leveres direkte af PyTorch, Edward Yang, and Team PyTorch eller deres podcastplatformspartner. Hvis du mener, at nogen bruger dit ophavsretligt beskyttede værk uden din tilladelse, kan du følge processen beskrevet her https://da.player.fm/legal.

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.

Further reading.

A nice summary of the full issue https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
DataLoader architecture RFC https://github.com/pytorch/pytorch/issues/49440
Cinder Python https://github.com/facebookincubator/cinder

… continue reading

82 episoder

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Artwork

DataLoader with multiple workers leaks memory

PyTorch Developer Podcast

33 subscribers

published 3y ago

Del

MP3•Episode hjem

Indhold leveret af PyTorch, Edward Yang, and Team PyTorch. Alt podcastindhold inklusive episoder, grafik og podcastbeskrivelser uploades og leveres direkte af PyTorch, Edward Yang, and Team PyTorch eller deres podcastplatformspartner. Hvis du mener, at nogen bruger dit ophavsretligt beskyttede værk uden din tilladelse, kan du følge processen beskrevet her https://da.player.fm/legal.

Today I'm going to talk about a famous issue in PyTorch, DataLoader with num_workers > 0 causes memory leak (https://github.com/pytorch/pytorch/issues/13246). This bug is a good opportunity to talk about DataSet/DataLoader design in PyTorch, fork and copy-on-write memory in Linux and Python reference counting; you have to know about all of these things to understand why this bug occurs, but once you do, it also explains why the workarounds help.

Further reading.

A nice summary of the full issue https://github.com/pytorch/pytorch/issues/13246#issuecomment-905703662
DataLoader architecture RFC https://github.com/pytorch/pytorch/issues/49440
Cinder Python https://github.com/facebookincubator/cinder

… continue reading

82 episoder

#Tech #PyTorch #Edward Yang #Team PyTorch #Deep Learning #Machine Learning

Alle episoder

×

Velkommen til Player FM!

Player FM is scanning the web for high-quality podcasts for you to enjoy right now. It's the best podcast app and works on Android, iPhone, and the web. Signup to sync subscriptions across devices.

Lyt til 500+ emner