Gammo - A pure-Ruby HTML5 parser

Gammo provides a pure Ruby HTML5-compliant parser and XPath support for traversing the DOM tree built by Gammo. The implementation of the HTML5 parsing algorithm in Gammo conforms the WHATWG specification. Given an HTML string, Gammo parses it and builds DOM tree based on the tokenization and tree-construction algorithm defined in WHATWG parsing algorithm, these implementations are provided without any external dependencies.

Gammo, its naming is inspired by Gumbo. But Gammo is a fried tofu fritter made with vegetables.

require 'gammo'
require 'open-uri'

parser = URI.open('https://google.com') { |f| Gammo.new(f.read) }
document = parser.parse #=> #<:node::document>
puts document.xpath('//title').first.inner_text #=> 'Google'
</:node::document>

Overview

Features

• Tokenization: Gammo has a tokenizer for implementing the tokenization algorithm.
• Parsing: Gammo provides a parser which implements the parsing algorithm by the above tokenization and the tree-construction algorithm.
• Node: Gammo provides the nodes which implement WHATWG DOM specification partially.
• DOM Tree Traversal: Gammo provides a way of DOM tree traversal.
• Performance: Gammo does not prioritize performance, and there are a few potential performance notes.

Tokenizaton

Gammo::Tokenizer

Gammo::Tokenizer#next_token

Here is a simple example for performing only the tokenizer.

def dump_for(token)
  puts "data: #{token.data}, class: #{token.class}"
end

tokenizer = Gammo::Tokenizer.new('')
dump_for tokenizer.next_token #=> data: html, class: Gammo::Tokenizer::DoctypeToken
dump_for tokenizer.next_token #=> data: input, class: Gammo::Tokenizer::StartTagToken
dump_for tokenizer.next_token #=> data: frameset, class: Gammo::Tokenizer::StartTagToken
dump_for tokenizer.next_token #=> data: end of string, class: Gammo::Tokenizer::ErrorToken

The parser described below depends on this tokenizer, it applies the WHATWG parsing algorithm to the tokens extracted by this tokenization in order.

Token types

The tokens generated by the tokenizer will be categorized into one of the following types:

Parsing

Gammo::Parser

A successfully parsed parser has the

document

Gammo::Parser#parse

document

require 'gammo'
require 'pp'

document = Gammo.new('').parse
def dump_for(node, strm)
  strm << node.to_h
  return unless node && (child = node.first_child)
  while child
    dump_for(child, (strm.last[:children] ||= []))
    child = child.next_sibling
  end
  strm
end
pp dump_for(document, [])

Notes

Currently, it's not possible to traverse the DOM tree with css selector or xpath like Nokogiri. However, Gammo plans to implement these features in the future.

Node

The nodes generated by the parser will be categorized into one of the following types:

For some nodes such as

Gammo::Node::Element

Gammo::Node::Document

Gammo::Node#next_sibling

Gammo::Node#first_child

Gammo::Node#append_child

Gammo::Node#remove_child

DOM Tree Traversal

Currently, XPath 1.0 is the only way for traversing DOM tree built by Gammo. CSS selector support is also planned but not having any ETA.

XPath 1.0 (experimental)

Gammo has an original lexer/parser for XPath 1.0, it's provided as a helper in the DOM tree built by Gammo. Here is a simple example:

document = Gammo.new('').parse
node_set = document.xpath('//input[@type="button"]') #=> "<:xpath::nodeset>"

node_set.length #=> 1
node_set.first #=> "<:node::element>"
</:node::element></:xpath::nodeset>

Since this is implemented by full scratch, Gammo is providing this support as a very experimental feature. Please file an issue if you find bugs.

Example

Before proceeding at the details of XPath support, let's have a look at a few simple examples. Given a sample HTML text and its DOM tree:

document = Gammo.new(<




  
namusyaka.com
  
Here is a sample web site.
  

• hello
• world

• Google google.com
• GitHub github.com/namusyaka

EOS

The following XPath expression gets all

li

document.xpath('//li').each do |elm|
  puts elm.inner_text
end

The following XPath expression gets all

li

ul

id=links

document.xpath('//ul[@id="links"]/li').each do |elm|
  puts elm.inner_text
end

The following XPath expression gets each text node for each

li

ul

id=links

document.xpath('//ul[@id="links"]/li/text()').each do |elm|
  puts elm.data
end

Axis Specifiers

In the combination with Gammo, the axis specifier indicates navigation direction within the DOM tree built by Gammo. Here is list of axes. As you can see, Gammo fully supports the all of axes.

Node Test

Node tests consist of specific node names or more general expressions. Although particular syntax like

:

Also note that the

processing-instruction

Operators

• The

  /

  ,

  //

  and

  []

  are used in the path expression.
• The union operator

  |

  forms the union of two node sets.
• The boolean operators:

  and

  ,

  or

• The arithmetic operators:

  +

  ,

  -

  ,

  *

  ,

  div

  and

  mod

• Comparison operators:

  =

  ,

  !=

  ,

  <

  ,

  >

  ,

  <=

  ,

  >=

Functions

XPath 1.0 defines four data types (nodeset, string, number, boolean) and there are various functions based on the types. Gammo supports those functions partially, please check it to be supported before using functions.

Node set functions

String Functions

Boolean Functions

Number Functions

CSS Selector

TBD.

Performance

As mentioned in the features at the beginning, Gammo doesn't prioritize its performance. Thus, for example, Gammo is not suitable for very performance-sensitive applications (e.g. performing Gammo parsing synchronously from an incoming request from an end user). Instead, the goal is to work well with batch processing such as crawlers. Gammo places the highest priority on making it easy to parse HTML by peforming it without depending on native-extensions and external gems.

References

This was developed with reference to the following softwares.

• x/net/html: I've been working on this package, it gave me strong reason to make this happen.
• Blink: Blink gave me great impression about tree construction.
• html5lib-tests: Gammo relies on this test.

License

The gem is available as open source under the terms of the MIT License.

Release History

• v0.2.0
  • XPath 1.0 support #4
• v0.1.0
  • Initial Release