massively parallel experimentation with Jupyter and AWS Lambda 🐑🌩📒
Eigensheep is a python package, with a very easy setup process, that lets you effortlessly run Jupyter Notebook cells on AWS Lambda, enabling massive parallelism. To instantly provision and run your code on 1000 tiny VMs, prefix a cell with
%%eigensheep -n 1000.
Eigensheep gives your Lambda code full access both to packages from PyPi, and to layers from Lambda Layers, including typically tricky-to-install things like
%%eigensheepto run it on AWS Lambda
tqdmfor interactively displaying progress
Here we compare the task of capturing screenshots of the 50 most popular websites (according to Moz) with Pyppeteer. In the first bar, we're doing this sequentially with a Python
forloop. In the second one, each website is run as a different Lambda. The estimated cost of the full sequential test (at the current
us-east-1price) is $0.0051 (or 0.07% of the monthly free quota). The estimated cost of the full parallel test is $0.0073 (or 0.11% of the monthly free quota).
Open up your Terminal and install
pip install eigensheep
Open a Jupyter notebook with
jupyter notebookand create a new Python notebook. Eigensheep supports both Python 2 and Python 3. Run the following code in a cell:
Follow the on-screen instructions to configure AWS credentials. AWS credentials will be saved to
eigensheepprofile for subsequent invocations. Eigensheep uses AWS CloudFormation so you only need a few clicks to get started (see our guided video walkthrough).
Once Eigensheep is set up, you can run any code on Lambda by prefixing the cell with
%%eigensheep. You can include dependencies from
%%eigensheep, for example
%%eigensheep requests numpy. You can invoke a cell multiple times concurrently with the
-nparameter, for example
%%eigensheep -n 100.
Q: Why is this library called Eigensheep?
The name comes from the classic math joke:
What do you call a baby eigensheep?
A lamb, duh.
Q: Does this work on Python 2 and Python 3?
A: Both Python 2 and Python 3 are supported. If the library is imported from a Python 2.x notebook, the Lambda runtime will default to "python2.6". If the library is imported from a Python 3.x notebook, the Lambda runtime defaults to "python3.6". This can be manually overridden with the "--runtime" option.
Q: Can I use this to do GPU stuff?
A: Currently the AWS Lambda execution environment does not expose access to any GPU acceleration. Eigensheep probably won't be that useful for training deep neural nets.
Q: How much does it cost to run stuff on AWS Lambda?
A: Unlike a traditional VM, you don't get charged while you're idling and not actively computing. You don't have to worry about accidentally forgetting to turn off a machine, and provisioning a VM takes only milliseconds rather than minutes.
AWS provides a pretty generous Free Tier for Lambda which does not expire after 12 months. It's 400,000 GB-seconds/month. That's 36 continuous hours of a single maxed out 3108MB Lambda job for free every month. Alternatively, it's about 20 minutes of 100 concurrent maxed out instances. After that it's about $7 for every subsequent free-tier equivalent.
Q: Can this be used for web scraping?
A: Yes, Eigensheep can be used for web scraping. However, note that different Lambda VM instances often share the same IP address.
Q: Can Eigensheep be used for long running computations?
A: The maximum allowed duration of any Lambda job is 15 minutes. Eigensheep works best for tasks which can be broken up into smaller chunks.
Q: What are the security implications of using Eigensheep?
A: The Eigensheep CloudFormation stack creates an IAM User, Access Key, and Lambda Role with as few permissions as possible. If the access keys are compromised, the attacker only has access to a bucket containing Eigensheep-specific content, and can not use it to access any of your other AWS resources.
The IAM User can only read/write from a specific bucket earmarked for use with Eigensheep, and can only update a specific lambda function (all the different variants are stored as different versions on a single Lambda function). The Lambda function only has access to the specific bucket and the ability to write to CloudWatch logs and XRay tracing streams.
All of the access keys can be revoked and all of the resources can be removed simply by deleting the CloudFormation stack from the AWS console.
Q: Where does Eigensheep store its configuration?
A: Eigensheep stores its access keys and configuration in the
~/.aws/configfile under the
Q: Can I use Eigensheep without installing the CloudFormation Stack?
A: Yes. Although it's a bit more complicated to set up. You can use any AWS access key and secret, so long as it has the ability to modify/invoke a Lambda named "EigensheepLambda" (which must be manually created). You must also create an S3 bucket named "eigensheep-YOURACCOUNTID", where YOURACCOUNTID is your numerical AWS account ID.
usage: %%eigensheep [-h] [-n N] [--memory MEMORY] [--timeout TIMEOUT] [--runtime RUNTIME] [--layer LAYER] [--reinstall] [--no_install] [--clean] [--rm] [--name NAME] [--verbose] [deps [deps ...]]
Jupyter cell magic to invoke cell on AWS Lambda
positional arguments: deps dependencies to be installed via pip
optional arguments: -h, --help show this help message and exit -n N number of parallel lambdas to invoke --memory MEMORY amount of memory in 64MB increments from 128 up to 3008 --timeout TIMEOUT lambda execution timeout in seconds up to 900 (15 minutes) --runtime RUNTIME lambda runtime (python3.7, python2.7) defaults configured based on host environment --layer LAYER ARNs of lambda layers to include --reinstall regenerate lambda configuration and dependencies --no_install do not install dependencies if configration not found --clean clear all deployed lambda configurations --rm remove a specific lambda configuration --name NAME store the lambda for later use with
eigensheep.invoke--verbose show additional information from lambda invocation
eigensheep.map("do_stuff", [1, 2, 3, 4])
boto3, and countless Stackoverflow answers.