Safely store secrets in Git/Mercurial/Subversion
Safely store secrets in a VCS repo (i.e. Git, Mercurial, Subversion or Perforce). These commands make it easy for you to Gnu Privacy Guard (GPG) encrypt specific files in a repo so they are "encrypted at rest" in your repository. However, the scripts make it easy to decrypt them when you need to view or edit them, and decrypt them for use in production. Originally written for Puppet, BlackBox now works with any Git or Mercurial repository.
A slide presentation about an older release is on SlideShare.
Join our mailing list: https://groups.google.com/d/forum/blackbox-project
Suppose you have a VCS repository (i.e. a Git or Mercurial repo) and certain files contain secrets such as passwords or SSL private keys. Often people just store such files "and hope that nobody finds them in the repo". That's not safe.
With BlackBox, those files are stored encrypted using GPG. Access to the VCS repo without also having the right GPG keys makes it worthless to have the files. As long as you keep your GPG keys safe, you don't have to worry about storing your VCS repo on an untrusted server. Heck, even if you trust your server, now you don't have to trust the people that do backups of that server, or the people that handle the backup tapes!
Rather than one GPG passphrase for all the files, each person with access has their own GPG keys in the system. Any file can be decrypted by anyone with their GPG key. This way, if one person leaves the company, you don't have to communicate a new password to everyone with access. Simply disable the one key that should no longer have access. The process for doing this is as easy as running 2 commands (1 to disable their key, 1 to re-encrypt all files.)
Automated processes often need access to all the decrypted files. This is easy too. For example, suppose Git is being used for Puppet files. The master needs access to the decrypted version of all the files. Simply set up a GPG key for the Puppet master (or the role account that pushes new files to the Puppet master) and have that user run
blackbox_postdeployafter any files are updated.
If you don't have a GPG key, set it up using instructions such as: Set up GPG key. \ Now you are ready to go.
cdinto a Git, Mercurial, Subversion or Perforce repository and run
blackbox_initialize.
If a file is to be encrypted, run
blackbox_register_new_fileand you are done.
Add and remove keys with
blackbox_addadminand
blackbox_removeadmin.
To view and/or edit a file, run
blackbox_edit; this will decrypt the file and open with whatever is specified by your $EDITOR environment variable. \ When you close the editor the file will automatically be encrypted again and the temporary plaintext file will be shredded. \ If you need to leave the file decrypted while you update you can use the
blackbox_edit_startto decrypt the file and
blackbox_edit_endwhen you want to "put it back in the box."
OBVIOUSLY we don't want secret things like SSL private keys and passwords to be leaked.
NOT SO OBVIOUSLY when we store "secrets" in a VCS repo like Git or Mercurial, suddenly we are less able to share our code with other people. Communication between subteams of an organization is hurt. You can't collaborate as well. Either you find yourself emailing individual files around (yuck!), making a special repo with just the files needed by your collaborators (yuck!!), or just deciding that collaboration isn't worth all that effort (yuck!!!).
The ability to be open and transparent about our code, with the exception of a few specific files, is key to the kind of collaboration that DevOps and modern IT practitioners need to do.
make copy-installwill copy the bin files into $PREFIX/bin, default is /usr/local (uninstall with
make copy-uninstall).
make symlinks-installwill make symlinks of the bin files into $PREFIX/bin, default is /usr/local (uninstall with
make copy-uninstall) (useful when doing development)
sudo port install vcs_blackbox
brew install blackbox
make packages-rpm; now you can distribute the RPM via local methods. (Requires fpm.)
make packages-deb; now you can distribute the DEB via local methods. (Requires fpm.)
antigen bundle StackExchange/blackboxto your .zshrc
zgen load StackExchange/blackboxto your .zshrc where you're loading your other plugins.
nix-env -i blackbox
pkgin in scm-blackbox
| Name: | Description: | |-------------------------------------|-------------------------------------------------------------------------| |
blackbox_edit| Decrypt, run $EDITOR, re-encrypt a file | |
blackbox_edit_start| Decrypt a file so it can be updated | |
blackbox_edit_end| Encrypt a file after blackboxeditstart was used | |
blackbox_cat| Decrypt and view the contents of a file | |
blackbox_view| Like blackboxcat but pipes to
lessor $PAGER | | `blackboxdiff
| Diff decrypted files against their original crypted version | |blackboxinitialize
| Enable blackbox for a GIT or HG repo | |blackboxregisternewfile
| Encrypt a file for the first time | |blackboxderegisterfile
| Remove a file from blackbox | |blackboxlistfiles
| List the files maintained by blackbox | |blackboxlistadmins
| List admins currently authorized for blackbox | |blackboxdecryptfile
| Decrypt a file | |blackboxdecryptallfiles
| Decrypt all managed files (INTERACTIVE) | |blackboxpostdeploy
| Decrypt all managed files (batch) | |blackboxaddadmin
| Add someone to the list of people that can encrypt/decrypt secrets | |blackboxremoveadmin
| Remove someone from the list of people that can encrypt/decrypt secrets | |blackboxshredallfiles
| Safely delete any decrypted files | |blackboxupdateallfiles
| Decrypt then re-encrypt all files. Useful after keys are changed | |blackbox_whatsnew ` | show what has changed in the last commit for a given file |
BlackBox automatically determines which VCS you are using and does the right thing. It has a plug-in architecture to make it easy to extend to work with other systems. It has been tested to work with many operating systems.
git-- The Git
hg-- Mercurial
svn-- SubVersion (Thanks, Ben Drasin!)
p4-- Perforce
.blackboxdirectory is intact
To add or fix support for a VCS system, look for code at the end of
bin/_blackbox_common.sh
To add or fix support for a new operating system, look for the case statements in
bin/_blackbox_common.shand
bin/_stack_lib.shand maybe
tools/confidence_test.sh
BlackBox can be used with Cygwin or MinGW.
BlackBox assumes that
blackbox-admins.txtand
blackbox-files.txtwill have LF line endings. Windows users should be careful to configure Git or other systems to not convert or "fix" those files.
If you use Git, add the following lines to your
.gitattributesfile:
**/blackbox-admins.txt text eol=lf **/blackbox-files.txt text eol=lf
The latest version of
blackbox_initializewill create a
.gitattributesfile in the
$BLACKBOXDATAdirectory (usually
.blackbox) for you.
Cygwin support requires the following packages:
Normal operation:
Development (if you will be adding code and want to run the confidence test)
MinGW (comes with Git for Windows) support requires the following:
Normal operation:
MINTTYinstead of windows console. You'll be executing blackbox from the Git Bash prompt.
download.batonce it has completed run
install.batthen add the path for those tools to your PATH (ex:
PATH=%PATH%;c:\GnuWin32\bin)
Development:
make test)
GPG has many different ways to encrypt a file. BlackBox uses the mode that lets you specify a list of keys that can decrypt the message.
If you have 5 people ("admins") that should be able to access the secrets, each creates a GPG key and adds their public key to the keychain. The GPG command used to encrypt the file lists all 5 key names, and therefore any 1 key can decrypt the file.
To remove someone's access, remove that admin's key name (i.e. email address) from the list of admins and re-encrypt all the files. They can still read the .gpg file (assuming they have access to the repository) but they can't decrypt it any more.
What if they kept a copy of the old repo before you removed access? Yes, they can decrypt old versions of the file. This is why when an admin leaves the team, you should change all your passwords, SSL certs, and so on. You should have been doing that before BlackBox, right?
Why don't you use symmetric keys? In other words, why mess with all this GPG key stuff and instead why don't we just encrypt all the files with a single passphrase. Yes, GPG supports that, but then we are managing a shared password, which is fraught with problems. If someone "leaves the team" we would have to communicate to everyone a new password. Now we just have to remove their key. This scales better.
How do automated processes decrypt without asking for a password? GPG requires a passphrase on a private key. However, it permits the creation of subkeys that have no passphrase. For automated processes, create a subkey that is only stored on the machine that needs to decrypt the files. For example, at Stack Exchange, when our Continuous Integration (CI) system pushes a code change to our Puppet masters, they run
blackbox_postdeployto decrypt all the files. The user that runs this code has a subkey that doesn't require a passphrase. Since we have many masters, each has its own key. And, yes, this means our Puppet Masters have to be very secure. However, they were already secure because, like, dude... if you can break into someone's puppet master you own their network.
If you use Puppet, why didn't you just use hiera-eyaml? There are 4 reasons:
eval $(gpg-agent --daemon)
blackbox_edit_start FILENAME
vim FILENAME
blackbox_edit_end FILENAME
git commit -aor
hg commit
Wait... it can be even easier than that! Run
blackbox_edit FILENAME, and it'll decrypt the file in a temp file and call
$EDITORon it, re-encrypting again after the editor is closed.
Ansible Vault provides functionality for encrypting both entire files and strings stored within files; however, keeping track of the password(s) required for decryption is not handled by this module.
Instead one must specify a password file when running the playbook.
Ansible example for password file:
my_secret_password.txt.gpg
ansible-playbook --vault-password-file my_secret_password.txt site.yml
Alternatively, one can specify this in the
ANSIBLE_VAULT_PASSWORD_FILEenvironment variable.
Entire files, such as SSL certs and private keys, are treated just like regular files. You decrypt them any time you push a new release to the puppet master.
Puppet example for an encrypted file:
secret_file.key.gpg
file { '/etc/my_little_secret.key': ensure => 'file', owner => 'root', group => 'puppet', mode => '0760', source => "puppet:///modules/${module_name}/secret_file.key", }
Small strings, such as passwords and API keys, are stored in a hiera yaml file, which you encrypt with
blackbox_register_new_file. For example, we use a file called
blackbox.yaml. You can access them using the hiera() function.
Setup: Configure
hiera.yamlby adding "blackbox" to the search hierarchy:
:hierarchy: - ... - blackbox - ...
In blackbox.yaml specify:
--- module::test_password: "my secret password"
In your Puppet Code, access the password as you would any hiera data:
$the_password = hiera('module::test_password', 'fail')file {'/tmp/debug-blackbox.txt': content => $the_password, owner => 'root', group => 'root', mode => '0600', }
The variable
$the_passwordwill contain "my secret password" and can be used anywhere strings are used.
eval $(gpg-agent --daemon)
blackbox_register_new_file path/to/file.name.key
Multiple file names can be specified on the command line:
Example 1: Register 2 files:
blackbox_register_new_file file1.txt file2.txt
Example 2: Register all the files in
$DIR:
find $DIR -type f -not -name '*.gpg' -print0 | xargs -0 blackbox_register_new_file
This happens quite rarely, but we've got it covered:
blackbox_deregister_file path/to/file.name.key
FYI: Your repo may use
keyrings/liveinstead of
.blackbox. See "Where is the configuration stored?"
.blackbox/blackbox-admins.txtis a file that lists which users are able to decrypt files. (More pedantically, it is a list of the GnuPG key names that the file is encrypted for.)
To join the list of people that can edit the file requires three steps; You create a GPG key and add it to the key ring. Then, someone that already has access adds you to the system. Lastly, you should test your access.
If you don't already have a GPG key, here's how to generate one:
gpg --gen-key
WARNING: New versions of GPG generate keys which are not understood by old versions of GPG. If you generate a key with a new version of GPG, this will cause problems for users of older versions of GPG. Therefore it is recommended that you either assure that everyone using Blackbox have the exact same version of GPG, or generate GPG keys using a version of GPG as old as the oldest version of GPG used by everyone using Blackbox.
Pick defaults for encryption settings, 0 expiration. Pick a VERY GOOD passphrase. Store a backup of the private key someplace secure. For example, keep the backup copy on a USB drive that is locked in safe. Or, at least put it on a secure machine with little or no internet access, full-disk-encryption, etc. Your employer probably has rules about how to store such things.
FYI: If generating the key is slow, this is usually because the system isn't generating enough entropy. Tip: Open another window on that machine and run this command:
ls -R /
Now that you have a GPG key, add yourself as an admin:
blackbox_addadmin KEYNAME
...where "KEYNAME" is the email address listed in the gpg key you created previously. For example:
blackbox_addadmin [email protected]
When the command completes successfully, instructions on how to commit these changes will be output. Run the command as given to commit the changes. It will look like this:
git commit -m'NEW ADMIN: [email protected]' .blackbox/pubring.gpg .blackbox/trustdb.gpg .blackbox/blackbox-admins.txt
Then push it to the repo:
git pushor
ht push
(or whatever is appropriate)
NOTE: Creating a Role Account? If you are adding the pubring.gpg of a role account, you can specify the directory where the pubring.gpg file can be found as a 2nd parameter:
blackbox_addadmin [email protected] /path/to/the/dir
Ask someone that already has access to re-encrypt the data files. This gives you access. They simply decrypt and re-encrypt the data without making any changes.
Pre-check: Verify the new keys look good.
git pull # Or whatever is required for your system gpg --homedir=.blackbox --list-keys
For example, examine the key name (email address) to make sure it conforms to corporate standards.
Import the keychain into your personal keychain and reencrypt:
gpg --import .blackbox/pubring.gpg blackbox_update_all_files
Push the re-encrypted files:
git commit -a git pushor
hg commit hg push
Make sure you can decrypt a file. (Suggestion: Keep a dummy file in VCS just for new people to practice on.)
Simply run
blackbox_removeadminwith their keyname then re-encrypt:
Example:
blackbox_removeadmin [email protected] blackbox_update_all_files
When the command completes, you will be given a reminder to check in the change and push it.
Note that their keys will still be in the key ring, but they will go unused. If you'd like to clean up the keyring, use the normal GPG commands and check in the file.
FYI: Your repo may use
keyrings/liveinstead of
.blackbox. See "Where is the configuration stored?"
gpg --homedir=.blackbox --list-keys gpg --homedir=.blackbox --delete-key [email protected] git commit -m'Cleaned [email protected] from keyring' .blackbox/*
FYI: Your repo may use
keyrings/liveinstead of
.blackbox. See "Where is the configuration stored?"
The key ring only has public keys. There are no secret keys to delete.
Remember that this person did have access to all the secrets at one time. They could have made a copy. Therefore, to be completely secure, you should change all passwords, generate new SSL keys, and so on just like when anyone that had privileged access leaves an organization.
Blackbox stores its configuration data in the
.blackboxsubdirectory. Older repos use
keyrings/live. For backwards compatibility either will work.
All documentation refers to
.blackbox.
You can convert an old repo by simply renaming the directory:
mv keyrings/live .blackbox rmdir keyrings
There is no technical reason to convert old repos except that it is less confusing to users.
This change was made in commit 60e782a0, release v1.20180615.
The details:
$BLACKBOXDATA. If this environment variable is set, this is the directory that will be used. If it lists a directory that does not exist, Blackbox will print an error and exit.
$BLACKBOXDATAis not set: (which is the typical use case)
keyrings/liveand use it if it exists.
.blackboxwill be used. If
.blackboxdoes not exist, Blackbox will print an error and exit.
Overview:
To add "blackbox" to a git or mercurial repo, you'll need to do the following:
FYI: Your repo may use
keyrings/liveinstead of
.blackbox. See "Where is the configuration stored?"
You'll want to include blackbox's "bin" directory in your PATH:
export PATH=$PATH:/the/path/to/blackbox/bin blackbox_initialize
If you're using antigen, adding
antigen bundle StackExchange/blackboxto your .zshrc will download this repository and add it to your $PATH.
Follow the instructions for "How to indoctrinate a new user into the system?". Only do Step 1.
Once that is done, is a good idea to test the system by making sure a file can be added to the system (see "How to enroll a new file into the system?"), and a different user can decrypt the file.
Make a new file and register it:
rm -f foo.txt.gpg foo.txt echo This is a test. >foo.txt blackbox_register_new_file foo.txt
Decrypt it:
blackbox_edit_start foo.txt.gpg cat foo.txt echo This is the new file contents. >foo.txt
Re-encrypt it:
blackbox_edit_end foo.txt.gpg ls -l foo.txt*
You should only see
foo.txt.gpgas
foo.txtshould be gone.
The next step is to commit
foo.txt.gpgand make sure another user can check out, view, and change the contents of the file. That is left as an exercise for the reader. If you are feel like taking a risk, don't commit
foo.txt.gpgand delete it instead.
i.e. This is how a Puppet Master can have access to the unencrypted data.
FYI: Your repo may use
keyrings/liveinstead of
.blackbox. See "Where is the configuration stored?"
An automated user (a "role account") is one that that must be able to decrypt without a passphrase. In general you'll want to do this for the user that pulls the files from the repo to the master. This may be automated with Jenkins CI or other CI system.
GPG keys have to have a passphrase. However, passphrases are optional on subkeys. Therefore, we will create a key with a passphrase then create a subkey without a passphrase. Since the subkey is very powerful, it should be created on a very secure machine.
There's another catch. The role account probably can't check files into Git/Mercurial. It probably only has read-only access to the repo. That's a good security policy. This means that the role account can't be used to upload the subkey public bits into the repo.
Therefore, we will create the key/subkey on a secure machine as yourself. From there we can commit the public portions into the repo. Also from this account we will export the parts that the role account needs, copy them to where the role account can access them, and import them as the role account.
ProTip: If asked to generate entropy, consider running this on the same machine in another window:
sudo dd if=/dev/sda of=/dev/null
For the rest of this doc, you'll need to make the following substitutions:
NOTE: This should be more automated/scripted. Patches welcome.
On SECUREHOST, create the puppet master's keys:
$ mkdir /tmp/NEWMASTER $ cd /tmp/NEWMASTER $ gpg --homedir . --gen-key Your selection? (1) RSA and RSA (default) What keysize do you want? (2048) DEFAULT Key is valid for? (0) DEFAULTReal name: Puppet CI Deploy Account
Email address: [email protected]
NOTE: Rather than a real email address, use the [email protected] of the host the key will be used on. If you use this role account on many machines, each should have its own key. By using the FQDN of the host, you will be able to know which key is which. In this doc, we'll refer to [email protected] as $KEYNAME
Save the passphrase somewhere safe!
Create a sub-key that has no password:
$ gpg --homedir . --edit-key svc_deployacct gpg> addkey (enter passphrase) Please select what kind of key you want: (3) DSA (sign only) (4) RSA (sign only) (5) Elgamal (encrypt only) (6) RSA (encrypt only) Your selection? 6 What keysize do you want? (2048) Key is valid for? (0) Command> key 2 (the new subkey has a "*" next to it) Command> passwd (enter the main key's passphrase) (enter an empty passphrase for the subkey... confirm you want to do this) Command> save
Now securely export this directory to NEWMASTER:
gpg --homedir . --export -a svc_sadeploy >/tmp/NEWMASTER/pubkey.txt tar cvf /tmp/keys.tar . rsync -avP /tmp/keys.tar NEWMASTER:/tmp/.
On NEWMASTER, receive the new GnuPG config:
sudo -u svc_deployacct bash mkdir -m 0700 -p ~/.gnupg cd ~/.gnupg && tar xpvf /tmp/keys.tar