Need help with cascalog?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

nathanmarz
1.4K Stars 181 Forks Other 1.4K Commits 41 Opened issues

Description

Data processing on Hadoop without the hassle.

Services available

!
?

Need anything else?

Contributors list

# 91,793
Clojure
Clojure...
epl
taoenss...
347 commits
# 34,883
Java
Clojure
Shell
big-dat...
256 commits
# 222,904
Clojure
applesc...
Zsh
notifie...
131 commits
# 147,483
Clojure
Amazon ...
Shell
epl
61 commits
# 142,726
Clojure
histogr...
CSS
Shell
59 commits
# 134,664
test-fr...
Perl
Clojure
Amazon ...
15 commits
# 409,082
Clojure
Shell
MongoDB
mongodb...
8 commits
# 152,548
Clojure
Clojure...
reactjs
React
3 commits
# 100,198
Shell
Clojure
TeX
mcmc
3 commits
# 132,509
Clojure
Shell
TeX
Jupyter...
3 commits
# 576,194
Clojure
Shell
HTML
2 commits
# 459,475
Clojure
Shell
Scala
2 commits
# 328,764
Clojure
semanti...
Shell
interop
2 commits
# 249,479
unoffic...
Telegra...
Erlang
messeng...
2 commits
# 152,762
Clojure
Shell
CSS
2 commits
# 155,517
C++
Clojure
sh
mac
2 commits
# 121,412
Clojure
pymongo
MongoDB
health-...
2 commits
# 58,104
Clojure
Shell
edn
epl
1 commit
# 373,229
Java
Shell
Clojure
Emacs
1 commit
# 54,084
Clojure
Clojure...
Perl
cpu-mon...
1 commit

Cascalog

Build Status

Cascalog is a fully-featured data processing and querying library for Clojure or Java. The main use cases for Cascalog are processing "Big Data" on top of Hadoop or doing analysis on your local computer. Cascalog is a replacement for tools like Pig, Hive, and Cascading and operates at a significantly higher level of abstraction than those tools.

Follow the getting started steps, check out the tutorial, and you'll be running Cascalog queries on your local computer within 5 minutes.

Getting Started with JCascalog

To get started with JCascalog, Cascalog's pure-Java API, see this wiki page. The jcascalog.Playground class has in-memory datasets that you can play with to learn the basics.

Latest Version

The latest release version of Cascalog is hosted on Clojars:

Current Version

Getting started with Clojure Cascalog

The best way to get started with Cascalog is experiment with the toy datasets that ship with the project. These datasets are served from memory and can be played with purely from the REPL. Just follow these steps and you'll be on your way:

  1. Install leiningen
  2. Make sure you have Java 1.6 (run
    java -version
    )
  3. Start a new leiningen project with
    lein new 
    , replacing
    
    
  4. Include dependency on Cascalog in your project by adding
    [cascalog/cascalog-core "2.1.0"]
    into your project's
    project.clj
    file.
  5. Work through the examples in the Getting Started Guide.

Using Cascalog within a project

Cascalog is hosted at Clojars, and some of its dependencies are hosted at Conjars. Both Clo/Con-jars are maven repos that's easy to use with maven or leiningen.

To include Cascalog in your leiningen or cake project, add the following to your

project.clj
:

General

[cascalog/cascalog-core "3.0.0"] ;; under :dependencies
[org.apache.hadoop/hadoop-core "1.2.1"] ;; under :dev-dependencies

Leiningen 2.0

:repositories {"conjars" "http://conjars.org/repo"}
:dependencies [cascalog/cascalog-core "3.0.0"]
:profiles { :provided {:dependencies [[org.apache.hadoop/hadoop-core "1.2.1"]]}}

Leiningen < 2.0

:dependencies [cascalog/cascalog-core "3.0.0"]
:dev-dependencies [[org.apache.hadoop/hadoop-core "1.2.1"]]

Note that Cascalog is compatible with Clojure 1.2.0, 1.2.1, 1.3.0, 1.4.0, and 1.5.1.

Documentation and Issue Tracker

Come chat with us in the Google group: cascalog-user

Or in the #cascalog or #cascading rooms on freenode!

Priorities for Cascalog development

  1. Replicated and bloom joins
  2. Cross query optimization: push constants and filters down into subqueries when possible

Acknowledgements

YourKit is kindly supporting open source projects with its full-featured Java Profiler. YourKit, LLC is the creator of innovative and intelligent tools for profiling Java and .NET applications. Take a look at YourKit's leading software products: YourKit Java Profiler and YourKit .NET Profiler.

Cascalog is based off of a very early branch of cascading-clojure project (http://github.com/clj-sys/cascading-clojure). Special thanks to Bradford Cross and Mark McGranaghan for their work on that project. Much of that code appears within Cascalog in either its original form or a modified form.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.