Need help with prefetch_generator?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

justheuristic
196 Stars 18 Forks The Unlicense 16 Commits 1 Opened issues

Description

Simple package that makes your generator work in background thread

Services available

!
?

Need anything else?

Contributors list

# 3,177
Jupyter...
Python
lasagne
Keras
10 commits
# 352,954
pytorch
medical...
medical...
C
1 commit
# 620,603
C++
C
Shell
Jupyter...
1 commit

prefetch_generator

Simple package that makes your generator work in background thread.

Quick usage example (ipython notebook)

Install:

  • pip install prefetch_generator
  • No dependencies apart from standard libraries
  • Works with both python2 and python3 (pip3 install)

Description:

This is a single-function package that transforms arbitrary generator into a background-thead generator that prefetches several batches of data in a parallel background thead.

This is useful if you have a computationally heavy process (CPU or GPU) that iteratively processes minibatches from the generator while the generator consumes some other resource (disk IO / loading from database / more CPU if you have unused cores).

By default these two processes will constantly wait for one another to finish. If you make generator work in prefetch mode (see examples below), they will work in parallel, potentially saving you your GPU time.

We personally use the prefetch generator when iterating minibatches of data for deep learning with tensorflow and theano ( lasagne, blocks, raw, etc.).

based on http://stackoverflow.com/questions/7323664/python-generator-pre-fetch

Usage:

This package contains two objects - BackgroundGenerator(anyothergenerator[,maxprefetch = something]) - @background([maxprefetch=somethind]) decorator

the usage is either

for batch in BackgroundGenerator(my_minibatch_iterator):
    doit()

or

@background()
def iterate_minibatches(some_param):
    while True:
        X = read_heavy_file()
        X = do_helluva_math(X)
        y = wget_from_pornhub()
        do_pretty_much_anything()
        yield X_batch, y_batch

More details are written in the BackgroundGenerator doc help(BackgroundGenerator)

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.