Reproducible Data Science at Scale!
Pachyderm is a tool for version-controlled, automated, end-to-end data pipelines for data science. If you need to chain together data scraping, ingestion, cleaning, munging, wrangling, processing, modeling, and analysis in a sane way, while ensuring the traceability and provenance of your data, Pachyderm is for you. If you have an existing set of scripts which do this in an ad-hoc fashion and you're looking for a way to "productionize" them, Pachyderm can make this easy for you.
You can also refer to our complete documentation to see tutorials, check out example projects, and learn about advanced features of Pachyderm.
Keep up to date and get Pachyderm support via: - Follow us on Twitter. - Join our community Slack Channel to get help from the Pachyderm team and other users.
To get started, sign the Contributor License Agreement.
You should also check out our contributing guide.
Send us PRs, we would love to see what you do! You can also check our GH issues for things labeled "help-wanted" as a good place to start. We're sometimes bad about keeping that label up-to-date, so if you don't see any, just let us know.
WE'RE HIRING! Love Docker, Go and distributed systems? Learn more about our open positions
Pachyderm automatically reports anonymized usage metrics. These metrics help us understand how people are using Pachyderm and make it better. They can be disabled by setting the env variable
falsein the pachd container.
Pachyderm has moved some components of Pachyderm Platform to a source-available limited license.
We remain committed to the culture of open source, developing our product transparently and collaboratively with our community, and giving our community and customers source code access and the ability to study and change the software to suit their needs.
Under the Pachyderm Community License, you can access the source code and modify or redistribute it; there is only one thing you cannot do, and that is use it to make a competing offering.
Check out our License FAQ Page for more information.