Need help with aws-data-wrangler?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

awslabs
1.7K Stars 262 Forks Apache License 2.0 1.3K Commits 21 Opened issues

Description

Pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

Services available

!
?

Need anything else?

Contributors list

AWS Data Wrangler

Pandas on AWS

Easy integration with Athena, Glue, Redshift, Timestream, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

AWS Data Wrangler

An AWS Professional Service open source initiative | [email protected]

Release Python Version Code style: black License

Checked with mypy Coverage Static Checking Build Status Documentation Status

| Source | Downloads | Installation Command | |--------|-----------|----------------------| | PyPi | PyPI Downloads |

pip install awswrangler
| | Conda | Conda Downloads |
conda install -c conda-forge awswrangler
|

⚠️ For platforms without PyArrow 3 support (e.g. EMR, Glue PySpark Job, MWAA):
➡️

pip install pyarrow==2 awswrangler

Powered By

Table of contents

Quick Start

Installation command:

pip install awswrangler

⚠️ For platforms without PyArrow 3 support (e.g. EMR, Glue PySpark Job, MWAA):
➡️

pip install pyarrow==2 awswrangler
import awswrangler as wr
import pandas as pd
from datetime import datetime

df = pd.DataFrame({"id": [1, 2], "value": ["foo", "boo"]})

Storing data on Data Lake

wr.s3.to_parquet( df=df, path="s3://bucket/dataset/", dataset=True, database="my_db", table="my_table" )

Retrieving the data directly from Amazon S3

df = wr.s3.read_parquet("s3://bucket/dataset/", dataset=True)

Retrieving the data from Amazon Athena

df = wr.athena.read_sql_query("SELECT * FROM my_table", database="my_db")

Get a Redshift connection from Glue Catalog and retrieving data from Redshift Spectrum

con = wr.redshift.connect("my-glue-connection") df = wr.redshift.read_sql_query("SELECT * FROM external_schema.my_table", con=con) con.close()

Amazon Timestream Write

df = pd.DataFrame({ "time": [datetime.now(), datetime.now()],
"my_dimension": ["foo", "boo"], "measure": [1.0, 1.1], }) rejected_records = wr.timestream.write(df, database="sampleDB", table="sampleTable", time_col="time", measure_col="measure", dimensions_cols=["my_dimension"], )

Amazon Timestream Query

wr.timestream.query(""" SELECT time, measure_value::double, my_dimension FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3 """)

Read The Docs

Community Resources

Please send a Pull Request with your resource reference and @githubhandle.

Logging

Enabling internal logging examples:

import logging
logging.basicConfig(level=logging.INFO, format="[%(name)s][%(funcName)s] %(message)s")
logging.getLogger("awswrangler").setLevel(logging.DEBUG)
logging.getLogger("botocore.credentials").setLevel(logging.CRITICAL)

Into AWS lambda:

import logging
logging.getLogger("awswrangler").setLevel(logging.DEBUG)

Who uses AWS Data Wrangler?

Knowing which companies are using this library is important to help prioritize the project internally.

Please send a Pull Request with your company name and @githubhandle if you may.

What is Amazon SageMaker Data Wrangler?

Amazon SageMaker Data Wrangler is a new SageMaker Studio feature that has a similar name but has a different purpose than the AWS Data Wrangler open source project.

  • AWS Data Wrangler is open source, runs anywhere, and is focused on code.

  • Amazon SageMaker Data Wrangler is specific for the SageMaker Studio environment and is focused on a visual interface.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.