Need help with pqstream?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

tmc
459 Stars 22 Forks MIT License 109 Commits 27 Opened issues

Description

pqstream turns your postgres database into an event stream

Services available

!
?

Need anything else?

Contributors list

# 17,128
Go
gRPC
GraphQL
openapi
65 commits
# 235,867
Shell
golang
gRPC
postgre...
6 commits
# 66,708
golang
gRPC
bash-pr...
bash-co...
5 commits
# 209,215
remarka...
golang
gRPC
postgre...
4 commits
# 128,138
Go
Shell
gRPC
postgre...
1 commit
# 168,820
golang
gRPC
C
time-se...
1 commit
# 622,410
Go
Shell
golang
gRPC
1 commit
# 284,605
Vim
golang
gRPC
React
1 commit

pqstream

pqstream is a program that streams changes out of a postgres database with the intent of populating other systems and enabling stream processing of data sets.

ci status go report card coverage

installation

$ go get -u github.com/tmc/pqstream/cmd/{pqs,pqsd}

basic usage

create an example database:

$ createdb dbname
# echo "create table notes (id serial, created_at timestamp, note text)" | psql dbname

connect the agent:

$ pqsd -connect postgresql://user:[email protected]/dbname

connect the cli:

sh
$ pqs

at this point you will see streams of database operations rendered to stdout:

(in a psql shell):

dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# insert into notes values (default, default, 'here is a sample note');
INSERT 0 1
dbname=# update notes set note = 'here is an updated note' where id=1;
UPDATE 1
dbname=# delete from notes where id = 1;
DELETE 1
dbname=#

our client should now show our operations:

sh
$ pqs
{"schema":"public","table":"notes","op":"INSERT","id":"1","payload":{"created_at":null,"id":1,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"INSERT","id":"2","payload":{"created_at":null,"id":2,"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"UPDATE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"},"changes":{"note":"here is a sample note"}}
{"schema":"public","table":"notes","op":"DELETE","id":"1","payload":{"created_at":null,"id":1,"note":"here is an updated note"}}

field redaction

If there's a need to prevent sensitive fields (i.e. PII) from being exported the

redactions
flag can be used with
pqsd
:
$ pqsd -connect postgresql://user:[email protected]/dbname -redactions='{"public":{"users":["first_name","last_name","email"]}}'

The

redactions
is encoded in JSON and conforms to the following layout:
json
'{"schema":{"table":["field1","field2"]}}'`

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.