Need help with databay?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

Voyz
177 Stars 10 Forks Apache License 2.0 245 Commits 2 Opened issues

Description

Databay is a Python interface for scheduled data transfer. It facilitates transfer of (any) data from A to B, on a scheduled interval.

Services available

!
?

Need anything else?

Contributors list

No Data

This library is currently being beta-tested. See something that's broken? Did we get something wrong? Create an issue and let us know!

Databay title

Databay is a Python interface for scheduled data transfer. It facilitates transfer of (any) data from A to B, on a scheduled interval.

Installation

pip install databay

Documentation

See full Databay documentation.

Or more specifically:

Features

Overview

In Databay, data transfer is expressed with three components:

  • Inlets
    - for data production.
  • Outlets
    - for data consumption.
  • Links
    - for handling the data transit between inlets and outlets.

Scheduling is implemented using third party libraries, exposed through the

BasePlanner
interface. Currently two
BasePlanner
implementations are available - using Advanced Python Scheduler and Schedule.

A simple example:

# Data producer
inlet = HttpInlet('https://some.test.url.com/')

Data consumer

outlet = MongoOutlet('databay', 'test_collection')

Data transfer between the two

link = Link(inlet, outlet, datetime.timedelta(seconds=5))

Start scheduling

planner = ApsPlanner(link) planner.start()

Every 5 seconds this snippet will pull data from a test URL, and write it to MongoDB.

Example use:

Databay showcase gif


While Databay comes with a handful of built-in inlets and outlets, its strength lies in extendability. To use Databay in your project, create concrete implementations of

Inlet
and
Outlet
classes that handle the data production and consumption functionality you require. Databay will then make sure data can repeatedly flow between the inlets and outlets you create. Extending inlets and extending outlets is easy and has a wide range of customization. Head over to Extending Databay section for a detailed explanation or to Examples for real use cases.

Supported Python Versions

| Python Version | <3.6 | 3.6 | 3.7 | 3.8 | 3.9 | |---------------- |------ |----- |----- |----- |----- | | Supported | ❌ | ✅ | ✅ | ✅ | ✅ |

Community Contributions

We aim to support the ecosystem of Databay users by collating and promoting inlets and outlets that implement popular functionalities. We encourage you to share the inlets and outlets you write with the community - start by reading the guidelines on contributing to the Databay community.

Did you write a cool inlet or outlet that you'd like to share with others? Put it on a public repo, send us an email and we'll list it here!

[email protected]

Inlets

  • FileInlet - File input inlet (built-in).
  • HttpInlet - Asynchronous http request inlet using aiohttp (built-in).

Outlets

Requests

The following are inlets and outlets that others would like to see implemented. Feel free to build an item from this list and share your implementation! Let us know if you'd like to add an item to this list.

Roadmap

See full Databay Roadmap. Bare in mind this a live document that is shared to give you an idea of what can be expected in the future releases, rather than a locked schedule. Priorities and order of implementation may change without warning.

v1.0

  1. Beta test the pre-release.
  2. ~~Complete 100% test coverage.~~
  3. ~~Add more advanced examples.~~
  4. Release v1.0.
  5. Buy a carrot cake and celebrate.

v1.1

  1. Filters and translators - callbacks for processing data between inlets and outlets.
  2. Advanced scheduling - conditional, non uniform intervals.

Licence

See LICENSE

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.