Need help with Naiad?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.
MicrosoftResearch

Description

The Naiad system provides fast incremental and iterative computation for data-parallel workloads

443 Stars 74 Forks Apache License 2.0 58 Commits 10 Opened issues

Services available

Need anything else?

Installation steps

Naiad is distributed as a Visual Studio 2013 solution, comprising the following projects:

  • "Naiad", which contains the core Naiad system and support libraries, and builds Microsoft.Research.Naiad.dll. If you create your own project that uses Naiad, you must add a reference to this library.

  • "Lindi", a LINQ-like library for data-parallel iterative computation.

  • "DifferentialDataflow", a LINQ-like library supporting incremental execution of iterative computation.

  • "AzureSupport", a library of Naiad components that supports writing to and reading from Windows Azure Storage.

  • "Examples", which contains several example Naiad applications and builds Examples.exe.

  • In addition, the "ClusterSubmission" solution contains experimental support for launching Naiad programs on Azure and YARN clusters.

These projects can be built and executed using any of the following implementations of the .NET framework:

  • Microsoft .NET Framework 4.5

To build Naiad, follow the steps below that correspond to your setup.

  • Building using Visual Studio
  1. Open the Naiad.sln file in this directory using Visual Studio.
  2. On the menu bar, click Build > Build Solution, or press F6.
  • Building using MSBuild (on Windows, without Visual Studio)
  1. Ensure that MSBuild is installed on your local machine. You can get MSBuild by installing the Microsoft .NET Framwork, version 4.5. Typically, the file MSBuild.exe can be found in the path:

    C:\Windows\Microsoft.NET\Framework64\v4.0.30319\MSBuild.exe

    However, this path may differ, depending on your local configuration.

  2. Open a command prompt, and change to the directory containing the Naiad.sln file.

  3. At the command prompt, run MSBuild.exe.

Getting started with the example programs

As part of this source distribution, the Examples project demonstrates how to write simple applications using Naiad. After building the Naiad solution, you will find the executable Examples.exe in the Examples\bin\Debug directory.

Examples.exe uses command-line arguments to select an example, and it is recommended that you run it from a command prompt or terminal. For example, to run the word count example, follow these steps:

  1. Open a command prompt or terminal, and change to the Examples\bin\Debug directory.

  2. At the command prompt, type "Examples.exe" to see the usage and available examples.

  3. At the command prompt, type "Examples.exe wordcount" to get started with the wordcount example.

Known issues with Naiad

This release of Naiad is an alpha release. The amount of testing the code has been subjected to is limited, mostly to programs we have written. The email address [email protected] goes to each of the project members, and is a great place to send questions and comments.

There are several specific issues we know about now, and want to point out to avoid headaches for everyone involved:

  1. In distributed mode, we strongly recommend that each process call OnNext the same numbers of times before calling OnCompleted. Several aspects of the graph manager logic consult the epoch numbers associated with local inputs. Ideally, all process call OnNext the same number of times; passing no parameters to OnNext causes the epoch to advance with no input supplied, and is a great way to do this.

  2. Naiad currently requires custom datatypes to be structs, rather than classes. This is not too hard to fix, but does require us to swing through the code and ensure that every class has an appropriate default constructor, which all structs have. Structs also stress the GC a lot less, and are a lot easier to auto-serialize. Related to this, if you want to use distributed Naiad, the auto-generated Serialization/Deserialization code requires all of your fields to be public and mutable (ie: not readonly). This is because the generated code attempts to set these values on a newly minted struct, and can’t do this without these settings. More details on writing custom serialization code in upcoming blog posts.

As Naiad is a research prototype, there are many parts of it that are likely to change. We’ve tried hard to put interfaces in place that will remain relatively stable, but there are certainly aspects of the interfaces we expect will change. Information about programming examples, the current state of the code, known problems, and upcoming fixes and features should be available at the project web page:

http://research.microsoft.com/Naiad/,

or at the related blog:

http://bigdataatsvc.wordpress.com/

Contacting us

If you have questions or comments, please feel welcome to email us at

[email protected]

We'd love to hear your thoughts on the project. Thanks!

The Naiad team.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.