ThreatIngestor

.. image:: https://inquest.net/images/inquest-badge.svg :target: https://inquest.net/ :alt: Developed by InQuest .. image:: https://travis-ci.org/InQuest/ThreatIngestor.svg?branch=master :target: https://travis-ci.org/InQuest/ThreatIngestor :alt: Build Status .. image:: https://readthedocs.org/projects/threatingestor/badge/?version=latest :target: http://inquest.readthedocs.io/projects/threatingestor/en/latest/?badge=latest :alt: Documentation Status .. image:: https://api.codacy.com/project/badge/Grade/a989bb12e9604d5a9577ce71848e7a2a :target: https://app.codacy.com/app/InQuest/ThreatIngestor :alt: Code Health .. image:: https://api.codacy.com/project/badge/Coverage/a989bb12e9604d5a9577ce71848e7a2a :target: https://app.codacy.com/app/InQuest/ThreatIngestor :alt: Test Coverage .. image:: http://img.shields.io/pypi/v/ThreatIngestor.svg :target: https://pypi.python.org/pypi/ThreatIngestor :alt: PyPi Version

An extendable tool to extract and aggregate IOCs_ from threat feeds.

Integrates out-of-the-box with ThreatKB_ and MISP, and can fit seamlessly into any existing worflow with SQS, Beanstalk, and

custom plugins

Currently used by InQuest Labs IOC-DB: https://labs.inquest.net/iocdb.

Overview

ThreatIngestor can be configured to watch Twitter, RSS feeds, or other sources, extract meaningful information such as malicious IPs/domains and YARA signatures, and send that information to another system for analysis.

.. image:: https://inquest.readthedocs.io/projects/threatingestor/en/latest/_images/mermaid-multiple-operators.png :target: https://inquest.readthedocs.io/projects/threatingestor/en/latest/workflows.html :alt: ThreatIngestor flowchart with several sources feeding into multiple operators.

Try it out now with this

quick walkthrough

ThreatIngestor walkthroughs

labs.inquest.net/iocdb

Installation

ThreatIngestor requires Python 3.6+, with development headers.

Install ThreatIngestor from PyPI::

pip install threatingestor

Install optional dependencies for using some plugins, as needed::

pip install threatingestor[all]

View the

full installation instructions

Usage

Create a new

config.yml

config.example.yml

threatingestor config.yml

By default, it will run forever, polling each configured source every 15 minutes.

View the

full ThreatIngestor documentation

Plugins

ThreatIngestor uses a plugin architecture with "source" (input) and "operator" (output) plugins. The currently supported integrations are:

Sources ~~~~~~~

• Beanstalk work queues 

  __

• Git repositories 

  __

• GitHub repository search 

  __

• RSS feeds 

  __

• Amazon SQS queues 

  __

• Twitter 

  __

• Generic web pages 

  __

Operators ~~~~~~~~~

• Beanstalk work queues 

  __

• CSV files 

  __

• MISP 

  __

• MySQL table 

  __

• SQLite database 

  __

• Amazon SQS queues 

  __

• ThreatKB 

  __

• Twitter 

  __

View the

full ThreatIngestor documentation

Threat Intel Sources

Looking for some threat intel sources to get started? InQuest has a Twitter List with several accounts that post C2 domains and IPs: https://twitter.com/InQuest/lists/ioc-feed. Note that you will need to apply for a Twitter developer account to use the ThreatIngestor Twitter Source. Take a look at

config.example.yml

For quicker setup, RSS feeds can be a great source of intelligence. Check out this example

RSS config file

Support

If you need help getting set up, or run into any issues, feel free to open an Issue. You can also reach out to

@InQuest

We'd love to hear any feedback you have on ThreatIngestor, its documentation, or how you're putting it to work for you!

Contributing

Issues and pull requests are welcomed. Please keep Python code PEP8 compliant. By submitting a pull request you agree to release your submissions under the terms of the LICENSE_.

.. ThreatKB: https://github.com/InQuest/ThreatKB .. _LICENSE: https://github.com/InQuest/threat-ingestors/blob/master/LICENSE .. _full ThreatIngestor Documentation: https://inquest.readthedocs.io/projects/threatingestor/ .. _SQS: https://aws.amazon.com/sqs/ .. _Beanstalk: https://beanstalkd.github.io/ .. _MISP: https://www.misp-project.org/ .. _custom plugins: https://inquest.readthedocs.io/projects/threatingestor/en/latest/developing.html .. _IOCs: https://en.wikipedia.org/wiki/Indicatorof_compromise .. _full installation instructions: https://inquest.readthedocs.io/projects/threatingestor/en/latest/installation.html .. _Issue: https://github.com/InQuest/ThreatIngestor/issues .. [email protected]: https://twitter.com/InQuest .. _quick walkthrough: https://inquest.readthedocs.io/projects/threatingestor/en/latest/welcome.html#try-it-out .. _ThreatIngestor walkthroughs: https://inquest.net/taxonomy/term/42 .. _RSS config file: https://github.com/InQuest/ThreatIngestor/blob/master/rss.example.yml .. _labs.inquest.net/iocdb: https://labs.inquest.net/iocdb