SOURCE-LINKED INTELLIGENCE

FoldPipe: Bounded Remote Streaming of Native Molecular Shards with Asynchronous Prefetch

arXiv · AI, language, vision and robotics · article · Aug 27, 2026 · UTC

Training molecular machine-learning models on ephemeral or memory-constrained accelerator instances can require repeatedly retrieving preprocessed molecular graphs from remote storage. FoldPipe is a lightweight Python orchestration layer for already-sharded PyTorch and PyTorch Geometric data. It retrieves one shard ahead in a background thread while the consumer trains on the current shard, keeping the number of live shard payloads bounded with respect to total dataset size. Asynchronous prefetch and bounded buffering are established systems techniques rather than novel scheduling algorithms.

Read original source ↗ Open in workspace

recordType: paper
region: Global

Evidence & attribution

arXiv · AI, language, vision and robotics · 2026-08-27T12:15:16.000Z

First collected: 2026-09-21T08:51:59.673Z. This is not the publication date.