Need help with fdir?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

thecodrr
934 Stars 32 Forks MIT License 215 Commits 9 Opened issues

Description

⚡ The fastest directory crawler & globbing library for NodeJS. Crawls 1m files in < 1s

Services available

!
?

Need anything else?

Contributors list

# 36,276
directo...
Electro...
mac
bittorr...
190 commits
# 31,989
ecmascr...
Babel
Angular
Vue.js
3 commits
# 17,849
interva...
Koa
sass-fr...
plex
2 commits
# 3,867
JavaScr...
aws-lam...
js
lambda
1 commit
# 553,430
Python
JavaScr...
Node.js
fs
1 commit
# 66,214
directo...
Gulp
Babel
test-fr...
1 commit
# 7,784
exec
sequeli...
React
SQLite
1 commit

The Fastest Directory Crawler & Globber for NodeJS

The Fastest: Nothing similar (in the NodeJS world) beats

fdir
in speed. It can easily crawl a directory containing 1 million files in < 1 second.

💡 Stupidly Easy:

fdir
uses expressive Builder pattern to build the crawler increasing code readability.

🤖 Zero Dependencies*:

fdir
only uses NodeJS
fs
&
path
modules.

🕺 Astonishingly Small: < 2KB in size gzipped & minified.

🔥 All Node Versions Supported: Unlike other similar libraries that have dropped support for Node versions < 10,

fdir
supports all versions >= 6.

🖮 Hackable: Extending

fdir
is extremely simple now that the new Builder API is here. Feel free to experiment around.

*

picomatch
must be installed manually by the user to support globbing.

Support

Do you like this project? Support me by donating, creating an issue, becoming a stargazer, or opening a pull request. Thanks.

Status

This project is not dead. In terms of performance, I don't think there is anything more to be done. This is as fast a NodeJS Directory Crawler as you can get. I could be wrong though.

What's next: 1. There is still no Async Iterator API (contributions welcome). 2. We need to discuss whether its feasible (or even sensible) to have event emitters in different parts of the crawling process. 3. Make globbing pluggable 4. Finalize and freeze the API 5. Improve documentation (maybe we can make a proper documentation website?)

Contributions are welcome in all these.

🚄 Quickstart

Installation

You can install using

npm
:
$ npm i fdir

or Yarn:

$ yarn add fdir

Usage

const { fdir } = require("fdir");

// create the builder const api = new fdir().withFullPaths().crawl("path/to/dir");

// get all files in a directory synchronously const files = api.sync();

// or asynchronously api.withPromise().then((files) => { // do something with the result here. });

Documentation:

I have written in-depth documentation here.

📊 Benchmarks:

Specs:

  • CPU: Intel i7 7th Generation (7700HQ)
  • RAM: 16 GB
  • Storage: 256 GB SSD
  • OS: Manjaro Linux
  • Directory Size: < 2k files

Notes:

  • Some people asked that I benchmark
    no-op
    (without options) version of
    fdir
    .
    • I did and found no performance difference. The results were identical. (I didn't include it here as it wasn't anything special.)
  • Some other people were doubtful about the authenticity of these results due to frequency scaling, process overload, disk warmup, etc.
    • So I have updated the benchmark with new results that should resolve all those doubts. Here's the process I followed:
    • Hard shutdown the laptop (a couple of times just to be sure) to clear disk, ram cache etc.
    • Login directly to a TTY (avoiding any unnecessary process from starting).
    • Disable CPU Scaling using
      sh
      $ sudo cpupower frequency-set --governor performance
      
    • Run the benchmark

The Fastest Globber

Last updated: May 13, 2020 (fdir v3.3.0)

$ yarn bench:glob

glob pattern used:

**.js
&
**/**.js

Node v13.13.0

| Synchronous | Asynchronous | | :-------------------------------------------------------------------: | :--------------------------------------------------------------------: | | | |

The Fastest Directory Crawler

Last updated: May 10, 2020 (fdir v3.0.0)

$ yarn bench

Node v14.2.0:

| Synchronous | Asynchronous | | :---------------------------------------------------------------------: | :----------------------------------------------------------------------: | | | |

Node v8.7.0:

Older versions of fdir (1.x & 2.x) used synchronous

lstat
call (
lstatSync
) in the asynchronous API to acheive speed on Node < 10. This has been fixed in fdir 3.0.0.

| Synchronous | | | :--------------------------------------------------------------------: | :---------------------------------------------------------------------: | | | |

🦮 LICENSE

Copyright © 2020 Abdullah Atta under MIT. Read full text here.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.