Need help with aistore?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

NVIDIA
384 Stars 39 Forks MIT License 4.8K Commits 1 Opened issues

Description

AIStore: scalable storage for AI applications

Services available

!
?

Need anything else?

Contributors list

AIStore is a lightweight object storage system with the capability to linearly scale-out with each added storage node and a special focus on petascale deep learning.

License Go Report Card

AIStore (AIS for short) is a built from scratch, lightweight storage stack tailored for AI apps. AIS consistently shows balanced I/O distribution and linear scalability across arbitrary numbers of clustered servers, producing performance charts that look as follows:

I/O distribution

The picture above comprises 120 HDDs.

The ability to scale linearly with each added disk was, and remains, one of the main incentives behind AIStore. Much of the development is also driven by the ideas to offload dataset transformation and other I/O intensive stages of the ETL pipelines.

Features

  • scale-out with no downtime and no limitation;
  • comprehensive HTTP REST API to GET and PUT objects, create, destroy, list and configure buckets, and more;
  • Amazon S3 API to run unmodified S3 apps;
  • FUSE client (
    aisfs
    ) to access AIS objects as files;
  • arbitrary number of extremely lightweight access points;
  • easy-to-use CLI based on auto-completions;
  • automated cluster rebalancing upon: changes in cluster membership, drive failures and attachments, bucket renames;
  • N-way mirroring (RAID-1), Reed–Solomon erasure coding, end-to-end data protection.
  • ETL offload: running user-defined extract-transform-load workloads on (and by) performance-optimized storage cluster;

Also, AIStore:

  • can be deployed on any commodity hardware;
  • supports single-command infrastructure and software deployment on Google Cloud Platform via ais-k8s GitHub repo;
  • supports Amazon S3, Google Cloud, and Microsoft Azure backends (and all S3, GCS, and Azure-compliant object storages);
  • can ad-hoc attach and "see" (read, write, list, cache, evict) datasets hosted by other AIS clusters;
  • provides unified global namespace across multiple backends:

AIStore

  • can be used as a fast cache for GCS and S3; can be populated on-demand and/or via
    prefetch
    and
    download
    APIs;
  • can be used as a standalone highly-available protected storage;
  • includes MapReduce extension for massively parallel resharding of very large datasets;
  • supports existing PyTorch and TensorFlow-based training models.

AIS runs natively on Kubernetes and features open format - thus, the freedom to copy or move your data from AIS at any time using the familiar Linux

tar(1)
,
scp(1)
,
rsync(1)
and similar.

For AIStore white paper and design philosophy, for introduction to large-scale deep learning and the most recently added features, please see AIStore Overview (where you can also find six alternative ways to work with existing datasets). Videos and animated presentations can be found at videos.

Finally, getting started with AIS takes only a few minutes.

Deployment options

There is a vast spectrum of possible deployments - primarily due to the fact that the essential prerequisites boil down to having Linux with a disk. This results in a practically unlimited set of options from all-in-one (AIS gateway + AIS target) docker container to a petascale bare-metal cluster of any size, and from a single development VM or workstation to multiple racks of high-end servers.

The table below contains a few concrete examples:

| Deployment option | Targeted audience and objective | | --- | ---| | Local playground | AIS developers and development, Linux or Mac OS | | Minimal production-ready deployment | This option utilizes preinstalled docker image and is targeting first-time users or researchers (who could immediately start training their models on smaller datasets) | | Easy automated GCP/GKE deployment | Developers, first-time users, AI researchers | | Large-scale production deployment | Requires Kubernetes and is provided (documented, automated) via a separate repository: ais-k8s |

Further, there's the capability referred to as global namespace. Simply put, as long as there’s HTTP connectivity, AIS clusters can be easily interconnected to “see” - i.e., list, read, write, cache, evict - each other's datasets. This ad-hoc capability, in turn, makes it possible to start small and gradually/incrementally build high-performance shared storage comprising petabytes.

For more discussion on supported deployments, please refer to Getting Started.

Observability

There are multiple ways to monitor all aspects of AIS operation.

For starters, AIS collects, logs, and reports via StatsD a fairly large and growing number of counters, latencies and throughputs including (but not limited to) those stats that reflect cluster recovery and global rebalancing, all extended long-running operations, and, of course, the basic read, write, list transactions, and more.

Logging interval is called

stats_time
(default
10s
) and is configurable on the level of both each specific node and the entire cluster.

In particular, all eXtended actions support generic API and CLI to show both common counters (byte and object numbers) as well as operation-specific extended statistics.

Batch operations that may take many seconds (minutes, sometimes hours) to execute are called eXtended actions or xactions. Examples include erasure coding or n-way mirroring a dataset, resharding and reshuffling a dataset, and many more.

In addition, AIS subsystems integrate their own, subsystem-specific, stats - e.g.:

Finally, global rebalance (that gets triggered by any membership changes - nodes joining, leaving, going down, etc.) can be further "visualized" via

ais show rebalance
CLI.

As far as Graphite/Grafana and Prometheus, AIS integrates with these popular backends via StatsD - the daemon for easy but powerful stats aggregation.

Scripts for easy deployment of both Graphite and Grafana are included (see below). StatsD can be connected to Graphite, which then can be used as a data source for Grafana to get a visual overview of the statistics and metrics. The same is true for Prometheus that is available for easy integration via its official exporter.

For local non-containerized deployments, use

./deploy/dev/local/deploy_grafana.sh
to start Graphite and Grafana containers. Local deployment scripts will automatically "notice" the presence of the containers and will send statistics to the Graphite.

For local docker-compose based deployments, make sure to use

-grafana
command-line option. The
./deploy/dev/docker/deploy_docker.sh
script will then spin-up Graphite and Grafana containers.

In both of these cases, Grafana will be accessible at localhost:3000.

For information on AIS statistics, please see Statistics, Collected Metrics, Visualization

AIStore includes

aisloader
- the tool to stress-test and benchmark storage performance. For background, command-line options, and usage, please see Load Generator and How To Benchmark AIStore.

Debug-Mode Observability

For development and, more generally, for any non-production deployments AIS supports building with debug, for instance:

$ MODE=debug make deploy

As usual, debug builds incorporate more runtime checks and extra logging. But in addition AIS debug build provides a special API endpoint at

hostname:port/debug/vars
that can be accessed (via browser or Curl) at any time to display the current values of:
  • all stats counters (including error counters)
  • all latencies including keepalive
  • mountpath capacities
  • mountpath (disk) utilizations
  • total number of goroutines
  • memory stats

and more.

Notation

hostname:port
stands for TCP endpoint of any deployed AIS node, gateway or storage target.

Example output:

# curl hostname:port/debug/vars
{
"ais.ios": {"/ais/mp1:util%": 20, "/ais/mp2:util%": 23, "/ais/mp3:util%": 22, "/ais/mp4:util%": 25},
"ais.stats": {"kalive.ns": 735065, "lst.n": 45, "lst.ns": 2892015, "num-goroutines": 27, "put.n": 1762, "put.ns": 1141380, "put.redir.ns": 16596465, "up.ns.time": 30012389406},
"cmdline": ["/bin/aisnode","-config=.ais/ais.json","-local_config=.ais/ais_local.json","-role=target"],
"memstats": {"Alloc":43209256,"TotalAlloc":57770120,"Sys":75056128,"Lookups":0,"Mallocs":215893,"Frees":103090,"HeapAlloc":43209256, ...}
...
}

AIStore Configuration

Configuration consists of cluster-wide (global) defaults and node-specific values - the latter includes node's own hostnames (or IP addresses) and mountpaths (disks).

Optionally and in addition, there's also a node-specific "config override" - a set of values that were changed for this node from global (inherited) defaults.

AIS configuration is further described and illustrated in this readme.

It is important to note that configuring cluster for production requires careful consideration. For example, AIS supports 3 (three) logical networks and will, therefore, benefit, performance-wise, if provisioned with up to 3 isolated physical networks or VLANs. The logical networks are:

  • user (aka public)
  • intra-cluster control
  • intra-cluster data

with the corresponding JSON names, respectively:

  • hostname
  • hostname_intra_control
  • hostname_intra_data

For AIS Kubernetes deployments we recommended Cilium CNI.

Assorted Tips

  • To enable an optional AIStore authentication server, execute
    $ AUTH_ENABLED=true make deploy
    . For information on AuthN server, please see AuthN documentation.
  • In addition to AIStore - the storage cluster, you can also deploy aisfs - to access AIS objects as files, and AIS CLI - to monitor, configure and manage AIS nodes and buckets.
  • AIS CLI is an easy-to-use command-line management tool supporting a growing number of commands and options (one of the first ones you may want to try could be
    ais show cluster
    - show the state and status of an AIS cluster). The CLI is documented in the readme; getting started with it boils down to running
    make cli
    and following the prompts.
  • For more testing commands and options, please refer to the testing README.
  • For
    aisnode
    command-line options, see: command-line options.
  • For helpful links and/or background on Go, AWS, GCP, and Deep Learning: helpful links.
  • And again, run
    make help
    to find out how to build, run, and test AIStore and tools.

Guides and References

Selected Package READMEs

License

MIT

Author

Alex Aizman (NVIDIA)

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.