highscore

by domnikl

domnikl / highscore

Find and rank keywords in text

212 Stars 22 Forks Last release: Not found 139 Commits 17 Releases

Available items

No Items, yet!

The developer of this repository has not created any items for sale yet. Need a bug fixed? Help with integration? A different license? Create a request here:

highscore

Easily find and rank keywords in long texts.

2019-10-19 I archived this project as there hasn't been any recent activity and I have moved on to other projects. You can still download and use it though.

Features

  • configurable to rank different types of words different (uppercase, long words, etc.)
  • rate based on amount (%) of vowels and consonants in a string
  • directly get keywords from String objects
  • blacklist words via a plain text file, String or an Array of words (per language if needed)
  • optionally, configure a whitelist and only words from that list will get ranked
  • uses word stemming if necessary (requires the
    fast-stemmer
    or
    stemmer
    gem)
  • merge together Keywords from multiple sources
  • contains a CLI tool that operates on STDIN/OUT and is configurable via parameters
  • can use
    bloomfilter-rb
    gem for better performance (optional)
  • words on the bonus list will receive a higher score

Installation

  • [sudo] gem install highscore

For better blacklist perfomance, use the

bloomfilter-rb
gem:
  • [sudo] gem install bloomfilter-rb

To use word stemming, you need to have the

fast-stemmer
(C extension) or
stemmer
gem installed:
  • [sudo] gem install fast-stemmer
  • [sudo] gem install stemmer

Examples

text = Highscore::Content.new "foo bar"
text.configure do
  set :multiplier, 2
  set :upper_case, 3
  set :long_words, 2
  set :long_words_threshold, 15
  set :short_words_threshold, 3      # => default: 2
  set :bonus_multiplier, 2           # => default: 3
  set :vowels, 1                     # => default: 0 = not considered
  set :consonants, 5                 # => default: 0 = not considered
  set :ignore_case, true             # => default: false
  set :word_pattern, /[\w]+[^\s0-9]/ # => default: /\w+/
  set :stemming, true                # => default: false
end

get only the top 50 keywords

text.keywords.top(50).each do |keyword| keyword.text # => keyword text keyword.weight # => rank weight (float) end

you could also just use a string

keywords = "Foo bar baz".keywords(blacklist) do set :multiplier, 10 end

The result is either a Highscore::Keywords object or an Array (depending on whether it's sorted or not) that you can iterate over. Each object in there is a Highscore::Keyword that has methods

text
and
weight
.
keywords = "Foo bar is not bar baz".keywords(Highscore::Blacklist.load(['baz']))

keywords.rank.each do |k| puts "#{k.text} #{k.weight}" end

prints:

Foo 3.0

bar 2.0

not 1.0

Have a look at

bin/highscore
, you can run highscore on your CLI and feed it with text on STDIN.

Blacklisting and Whitelisting

Using a custom blacklist to ignore keywords

# setting single words
blacklist = Highscore::Blacklist.new
blacklist << 'foo'

load a string/array

blacklist = Highscore::Blacklist.load "a string" blacklist = Highscore::Blacklist.load %w{an array}

loading from a file (separated by whitespace)

blacklist = Highscore::Blacklist.load_file "blacklist.txt"

loading the default blacklist (falls back automatically if not explicit given)

blacklist = Highscore::Blacklist.load_default_file

inject the blacklist into the content class

content = Highscore::Content.new "a string", blacklist

Using a whitelist instead of ranking all words

# construct and inject it just like a blacklist
whitelist = Highscore::Whitelist.load %w{these are valid keywords}
content = Highscore::Content.new "invalid words", whitelist

Using bonus words

# construct and inject it just like a blacklist
bonuslist = Highscore::Bonuslist.load %w{bonus words}
content = Highscore::Content.new "A string with bonus words in it", bonuslist

I18n

# Load a default blacklist
blacklist_default = Highscore::Blacklist.load "mister"
text = Highscore::Content.new "oui mister interesting", blacklist_default
text.keywords.top(3).join " "

=> prints "interesting oui"

Load a rudimentary blacklist for French

blacklist_francais = Highscore::Blacklist.load "oui" text.add_wordlist blacklist_francais, :french text.keywords(:lang => :fr).top(3).join " "

=> prints "interesting mister"

Author

Original author: Dominik Liebler [email protected]

License

(The MIT License)

Copyright (c) 2017 Dominik Liebler and contributors

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the 'Software'), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.