Need help with node-metainspector?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

gabceb
123 Stars 52 Forks MIT License 74 Commits 20 Opened issues

Description

Node npm for web scraping purposes. It scrapes a given URL, and returns you its title, meta description, meta keywords, an array with all the links, all the images in it, etc. Inspired by the metainspector Ruby gem

Services available

!
?

Need anything else?

Contributors list

# 88,979
Ruby
CSS
Shell
esi
31 commits
# 270,500
JavaScr...
Shell
7 commits
# 208,603
Node.js
smtp-se...
JavaScr...
HTML
6 commits
# 462,178
C#
pagespe...
c-sharp
html-mi...
3 commits
# 205,615
JavaScr...
2 commits
# 220,191
Less
Node.js
Go
Shell
1 commit
# 82,991
JavaScr...
HTTP
HTML
hosting
1 commit
# 728,115
JavaScr...
1 commit
# 140,158
Common ...
Erlang
talks
blockli...
1 commit
# 9,793
autohot...
ahk
bitwise...
Nette
1 commit
# 643,182
JavaScr...
microse...
esi
HTML
1 commit
# 87,041
HTML
osx
Clojure
Heroku
1 commit
# 306,373
Shell
pcapng
tcpdump
Haskell
1 commit

status

Node-Metainspector

MetaInspector is an npm package for web scraping purposes. You give it an URL, and it lets you easily get its title, links, images, description, keywords, meta tags....

Metainspector is inspired by the Metainspector gem by jaimeiniesta

This version requires node v6 or higher, as some dependencies make use of various bits of ES6 functionality. The 1.x.x versions are compatible with v0.x - v4 releases of node, and should be used instead for older applications.

Scraped data

client.url                  # URL of the page
client.scheme               # Scheme of the page (http, https)
client.host                 # Hostname of the page (like, markupvalidator.com, without the scheme)
client.rootUrl              # Root url (scheme + host, i.e http://simple.com/)
client.title                # title of the page, as string
client.links                # array of strings, with every link found on the page as an absolute URL
client.author               # page author, as string
client.keywords             # keywords from meta tag, as array
client.charset              # page charset from meta tag, as string
client.description          # returns the meta description, or the first long paragraph if no meta description is found
client.image                # Most relevant image, if defined with og:image
client.images               # array of strings, with every img found on the page as an absolute URL
client.feeds                # Get rss or atom links in meta data fields as array
client.ogTitle              # opengraph title
client.ogDescription        # opengraph description
client.ogType               # Open Graph Object Type
client.ogUpdatedTime        # Open Graph Updated Time
client.ogLocale             # Open Graph Locale - for languages

Options

timeout - Defines the time Metainspector will wait for the url to respond in ms
maxRedirects - Specifies the number of redirects Metainspector will follow
limit - The limit in the number of bytes Metainspector will download when querying a site

Usage

var MetaInspector = require('node-metainspector');
var client = new MetaInspector("http://www.google.com", { timeout: 5000 });

client.on("fetch", function(){ console.log("Description: " + client.description);

console.log("Links: " + client.links.join(","));

});

client.on("error", function(err){ console.log(err); });

client.fetch();

TO DO

Finish implementation of the properties below:

Add absolutify url function to return all urls as an absolute url

client.internal_links # array of strings, with every internal link found on the page as an absolute URL client.external_links # array of strings, with every external link found on the page as an absolute URL

ZOMG Fork! Thank you!

You're welcome to fork this project and send pull requests. Just remember to include tests.

Copyright (c) 2009-2012 Gabriel Cebrian, released under the MIT license

Bitdeli Badge

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.