A vector similarity database
Simbase is a redis-like vector similarity database. You can add, get, delete vectors to/from it, and then retrieve the most similar vectors within one vector set or between two vector sets.
Current version is v0.1.0-beta1.
Simbase use a concept model as below:
+ - - - + +----------->| Basis |
A real example follow the model below:
+ - - - - - + + - - - - - - - -+ +--->| Articles |
This graph shows
To build the project, you need install leiningen first, and then
After the uberjar is created, you can start the system
You can use redis-cli directly for administration tasks.
Or you can use redis client bindings in different language directly in a programming way.
dest = redis.Redis(host='localhost', port=7654) dest.execute_command('bmk', 'ba', 'a', 'b', 'c') dest.execute_command('vmk', 'ba', 'va') dest.execute_command('rmk', 'va', 'va', 'cosinesq')
var redis = require("redis"), client = redis.createClient(7654, 'localhost');
client.send_command('bmk', ['ba', 'a', 'b', 'c']) client.send_command('vmk', ['ba', 'va']) client.send_command('rmk', ['va', 'va', 'cosinesq'])
For example, we need to recommend articles to users, we may follow below steps:
> bmk b2048 t1 t2 t3 ... t2047 t2048 > vmk b2048 article > vmk b2048 userprofile > rmk userprofile article cosinesq
> vadd article 1 0.11 0.112 0.1123... > vadd article 2 0.21 0.212 0.2123... ...
> vadd userprofile 1 0.11 0.112 0.1123... > vadd userprofile 2 0.21 0.212 0.2123... ...
> rrec userprofile 2 article
All commands are explained in next section.
Then you can use redis-cli to connect to simbase directly
List all basis in system
bmk basisname components...
bmk b512 universe time space human animal plant...
Create a basis
brev basisname components...
brev b512 plant animal human space time universe...
Revise a basis
Vector set related
List all vector set with one basis
vmk basisname vecsetname
vmk b512 article
Create a vector set
vget vecsetname vecid
vget article 12345678
Get the vector for the article with id 12345678
vadd vecsetname vecid components...
vadd article 12345678 0.1 0.12 0.123 0.1234 0.12345 0.123456...
add the value for the article vector with id 12345678
vset vecsetname vecid components...
vset article 12345678 0.1 0.12 0.123 0.1234 0.12345 0.123456...
set the value for the article vector with id 12345678
vacc vecsetname vecid components...
vacc article 12345678 0.1 0.12 0.123 0.1234 0.12345 0.123456...
accumulate the value for the article vector with id 12345678
vrem vecsetname vecid
vrem article 12345678
remove the vector with id 12345678 from article vector set
List all recommendation targets with the inputed vecset as source
rmk vecsetname1 vecsetname2 funcscore
rmk userprofile article cosinesq
Create a recommendation to article by userprofile and it use cosinesq as score function. Currently score functions you can choice are: 'cosinesq' and 'jensenshannon'
rrec vecsetname1 vecid vecsetname2
rrec userprofile 87654321 article
Recommend articles for user 87654321
Although Simbase can store arbitrary vectors, but score functions may apply some constraints on vectors.
For example, if you adopt "jensenshannon" as your score function, you should assure your vector is a probability distribution, i.e. the sum of all components equals to one.
The write operation is handled in a single thread per basis, and comparison between any two vectors is needed, so the write operation is scaled at O(n).
We had a non-final performance test for the dense vectors on an i7-cpu Macbook, it can easily handle 100k 1k-dimensional vectors with each write operation in under 0.14 sec; and if the linear scale ratio can hold, it means Simbase can handle 700k dense vectors with each write operation in under 1 sec.
Since the data is all in memory, the read operation is pretty fast.
We are still in the process of tuning the performance of the sparse vectors.
Simbase is dual licensed under the Apache License 2.0 and Eclipse Public License 1.0. Simbase is free for commercial use and distribution under the terms of either license.
Special thanks for Feng Sheng, we borrowed lots of code from his great project http-kit ( https://github.com/http-kit/http-kit/ ).
Also thanks for Kunwei Zhang from Tsinghua Univ. for his smart idea.