Better directory iterator and faster os.walk(), now in the Python 3.5 stdlib
.. image:: https://img.shields.io/pypi/v/scandir.svg :target: https://pypi.python.org/pypi/scandir :alt: scandir on PyPI (Python Package Index)
.. image:: https://travis-ci.org/benhoyt/scandir.svg?branch=master :target: https://travis-ci.org/benhoyt/scandir :alt: Travis CI tests (Linux)
.. image:: https://ci.appveyor.com/api/projects/status/github/benhoyt/scandir?branch=master&svg=true :target: https://ci.appveyor.com/project/benhoyt/scandir :alt: Appveyor tests (Windows)
scandir()is a directory iteration function like
os.listdir(), except that instead of returning a list of bare filenames, it yields
DirEntryobjects that include file type and stat information along with the name. Using
scandir()increases the speed of
os.walk()by 2-20 times (depending on the platform and file system) by avoiding unnecessary calls to
os.stat()in most cases.
scandirhas been included in the Python 3.5 standard library as
os.scandir(), and the related performance improvements to
os.walk()have also been included. So if you're lucky enough to be using Python 3.5 (release date September 13, 2015) you get the benefit immediately, otherwise just
download this module from PyPI_, install it with
pip install scandir, and then do something like this in your code:
.. code-block:: python
# Use the built-in version of scandir/walk if possible, otherwise # use the scandir module version try: from os import scandir, walk except ImportError: from scandir import scandir, walk
PEP 471, which is the PEP that proposes including
scandirin the Python standard library, was
acceptedin July 2014 by Victor Stinner, the BDFL-delegate for the PEP.
This
scandirmodule is intended to work on Python 2.7+ and Python 3.4+ (and it has been tested on those versions).
Python's built-in
os.walk()is significantly slower than it needs to be, because -- in addition to calling
listdir()on each directory -- it calls
stat()on each file to determine whether the filename is a directory or not. But both
FindFirstFile/
FindNextFileon Windows and
readdiron Linux/OS X already tell you whether the files returned are directories or not, so no further
statsystem calls are needed. In short, you can reduce the number of system calls from about 2N to N, where N is the total number of files and directories in the tree.
In practice, removing all those extra system calls makes
os.walk()about 7-50 times as fast on Windows, and about 3-10 times as fast on Linux and Mac OS X. So we're not talking about micro-optimizations. See more benchmarks in the "Benchmarks" section below.
Somewhat relatedly, many people have also asked for a version of
os.listdir()that yields filenames as it iterates instead of returning them as one big list. This improves memory efficiency for iterating very large directories.
So as well as a faster
walk(), scandir adds a new
scandir()function. They're pretty easy to use, but see "The API" below for the full docs.
Below are results showing how many times as fast
scandir.walk()is than
os.walk()on various systems, found by running
benchmark.pywith no arguments:
==================== ============== ============= System version Python version Times as fast ==================== ============== ============= Windows 7 64-bit 2.7.7 64-bit 10.4 Windows 7 64-bit SSD 2.7.7 64-bit 10.3 Windows 7 64-bit NFS 2.7.6 64-bit 36.8 Windows 7 64-bit SSD 3.4.1 64-bit 9.9 Windows 7 64-bit SSD 3.5.0 64-bit 9.5 Ubuntu 14.04 64-bit 2.7.6 64-bit 5.8 Mac OS X 10.9.3 2.7.5 64-bit 3.8 ==================== ============== =============
All of the above tests were done using the fast C version of scandir (source code in
_scandir.c).
Note that the gains are less than the above on smaller directories and greater on larger directories. This is why
benchmark.pycreates a test directory tree with a standardized size.
walk() ~~~~~~
The API for
scandir.walk()is exactly the same as
os.walk(), so just
read the Python docs_.
scandir() ~~~~~~~~~
The full docs for
scandir()and the
DirEntryobjects it yields are available in the
Python documentation here_. But below is a brief summary as well.
scandir(path='.') -> iterator of DirEntry objects for given path
Like
listdir,
scandircalls the operating system's directory iteration system calls to get the names of the files in the given
path, but it's different from
listdirin two ways:
Instead of returning bare filename strings, it returns lightweight
DirEntryobjects that hold the filename string and provide simple methods that allow access to the additional data the operating system may have returned.
It returns a generator instead of a list, so that
scandiracts as a true iterator instead of returning the full list immediately.
scandir()yields a
DirEntryobject for each file and sub-directory in
path. Just like
listdir, the
'.'and
'..'pseudo-directories are skipped, and the entries are yielded in system-dependent order. Each
DirEntryobject has the following attributes and methods:
name: the entry's filename, relative to the scandir
pathargument (corresponds to the return values of
os.listdir)
path: the entry's full path name (not necessarily an absolute path) -- the equivalent of
os.path.join(scandir_path, entry.name)
is_dir(*, follow_symlinks=True): similar to
pathlib.Path.is_dir(), but the return value is cached on the
DirEntryobject; doesn't require a system call in most cases; don't follow symbolic links if
follow_symlinksis False
is_file(*, follow_symlinks=True): similar to
pathlib.Path.is_file(), but the return value is cached on the
DirEntryobject; doesn't require a system call in most cases; don't follow symbolic links if
follow_symlinksis False
is_symlink(): similar to
pathlib.Path.is_symlink(), but the return value is cached on the
DirEntryobject; doesn't require a system call in most cases
stat(*, follow_symlinks=True): like
os.stat(), but the return value is cached on the
DirEntryobject; does not require a system call on Windows (except for symlinks); don't follow symbolic links (like
os.lstat()) if
follow_symlinksis False
inode(): return the inode number of the entry; the return value is cached on the
DirEntryobject
Here's a very simple example of
scandir()showing use of the
DirEntry.nameattribute and the
DirEntry.is_dir()method:
.. code-block:: python
def subdirs(path): """Yield directory names not starting with '.' under given path.""" for entry in os.scandir(path): if not entry.name.startswith('.') and entry.is_dir(): yield entry.name
This
subdirs()function will be significantly faster with scandir than
os.listdir()and
os.path.isdir()on both Windows and POSIX systems, especially on medium-sized or large directories.
The Python docs for scandir_
PEP 471_, the (now-accepted) Python Enhancement Proposal that proposed adding
scandirto the standard library -- a lot of details here, including rejected ideas and previous discussion
Please send flames, comments, and questions about scandir to Ben Hoyt:
http://benhoyt.com/
File bug reports for the version in the Python 3.5 standard library
here_, or file bug reports or feature requests for this module at the GitHub project page:
https://github.com/benhoyt/scandir