Need help with pqstream?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

tmc
450 Stars 22 Forks MIT License 109 Commits 26 Opened issues

Description

pqstream turns your postgres database into an event stream

Services available

!
?

Need anything else?

Contributors list

# 16,361
Go
gRPC
GraphQL
openapi
65 commits
# 228,951
CSS
Shell
golang
gRPC
6 commits
# 73,034
Go
golang
gRPC
postgre...
5 commits
# 201,887
CSS
remarka...
golang
gRPC
4 commits
# 480,343
SQL
Go
Shell
gRPC
1 commit
# 165,205
golang
gRPC
CSS
time-se...
1 commit
# 621,047
Go
Shell
golang
gRPC
1 commit
# 279,062
CSS
Vim
golang
gRPC
1 commit

pqstream

pqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.

ci status go report card coverage

installation

$ go get -u github.com/tmc/pqstream/cmd/{pqs,pqsd}

basic usage

create an example database:

$ createdb dbname
# echo "create table notes (id serial, created_at timestamp, note text)" | psql dbname

connect the agent:

$ pqsd -connect postgresql://user:[email protected]/dbname

connect the cli:

sh
$ pqs

at this point you will see streams of database operations rendered to stdout:

(in a psql shell):

dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# update notes set note = 'here is an updated note' where id=1;
UPDATE 1
dbname=# delete from notes where id = 1;
DELETE 1
dbname=#

our client should now show our operations:

sh
$ pqs
{"schema":"public","table":"notes","op":"INSERT","id":"1","payload":{"created_at":null,"id":1,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"INSERT","id":"2","payload":{"created_at":null,"id":2,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"UPDATE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"},"changes":{"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"DELETE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"}}

field redaction

If there's a need to prevent sensitive fields (i.e. PII) from being exported the

redactions
flag can be used with
pqsd
:
$ pqsd -connect postgresql://user:[email protected]/dbname -redactions='{"public":{"users":["first_name","last_name","email"]}}'

The

redactions
is encoded in JSON and conforms to the following layout:
json
'{"schema":{"table":["field1","field2"]}}'`

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.