store and retrieve files from hashicorp vault

Question

I can't figure out how to store files in hashicorp vault. Our use case for a PoC is to store a SSL cert at a certain path and then download it via the HTTP API.

I tried using the kv secrets engine which seems the most appropriate.

Answer 1

So Vault's Default offering doesn't have this baked in, but there's a Desktop GUI program that add's this functionality in a user friendly way.
https://github.com/adobe/cryptr
I did run into a bit of confusion when using it:
If you have a KVv2, the HC Web UI, and the Cryptr Desktop GUI will use different conventions.
When writing Vault policies you'd use /KVv2/data/path/
When using Cryptr you'd use /KVv2/data/path/
When using HC WebUI you'd use /kvv2/path/

Fact: You can utilize base64 encoding to store raw binary files in any KV store.
Thus you can use the technique to store in Hashicorp Vault as well.

So base64 encoding is a reversible function that allows you to take any binary file, convert it to a 1 line string, then take the generated 1 line string and convert it back to any binary file. And since you can store a 1 line string in any KV store, you can store arbitrary binary files in any KV store! :) (*)

Here's some code to do what you're asking:

CMD:\> vault server -dev
WindowsSubsystemForLinuxBash:/mnt/c# curl -L https://releases.hashicorp.com/vault/1.0.2/vault_1.0.2_linux_amd64.zip > vault.zip
Bash# apt-get update
Bash# apt-get install unzip
Bash# unzip vault.zip -d /bin
Bash# chmod +x /bin/vault
Bash# export VAULT_ADDR=http://127.0.0.1:8200
Bash# vault login s.aO8ustaAV4Ot1OxzBe94vi3J
Bash# cat excelfile.xlsx | md5sum
fb6b4eaa2be1c8c410645a5f0819539e  -
Bash# cat excelfile.xlsx | base64 | base64 --decode > x.xlsx
Bash# cat x.xlsx | md5sum
fb6b4eaa2be1c8c410645a5f0819539e  -
Bash:/mnt/c# cat excelfile.xlsx | base64 | vault kv put secret/excelfile.xlsx base64dfile=-
(=- means assign value from standard in, which in this case is the piped output of the cat file command)
Chrome: localhost:8200
(login with dev root token, and you'll see the value is characters in a 1 line string)
Bash# rm excelfile.xlsx
Bash# vault kv get -field=base64dfile secret/excelfile.xlsx | tr -d '\n' | base64 --decode > excelfile.xlsx
(or)
Bash# vault kv get -field=base64dfile secret/excelfile.xlsx | sed 's/\r//' | base64 --decode > excelfile.xlsx
Bash# cat excelfile.xlsx | md5sum
fb6b4eaa2be1c8c410645a5f0819539e  -

(*Note Vault and other KV stores often have file size limits, Vault with Consul backend would have a secret file size limit of around ~375kb since base64 encoding will bloat the file size by 4/3rds bringing the size to 500kb and Consul has a Key Value pair limit of 0.5mb ish.)
(Note for perspective that's plenty of space as cert files can be ~8KB/if it's larger than 375kb it's probably not a secret.)

Lets say down the road you need to store bigger secrets:
(Such as Kubernetes etcd snapshot)
Since Vault went 1.0, there's built in functionality to migrate your storage backend, so you could switch from "Consul Storage Backend" to "Hybrid Storage Backend of AWS S3 Storage with Consul" (Consuls still needed for HA consistency locking in multi server setups)" to have a bigger limit. Picking a different storage backend will give you a bigger KV size limit. Note Vault probably imposes a sensible limit like 10mb though. Because even if you had a Vault Backend that supported 1TB Key Value sizes, you definitely would want to think twice about storing large files in vault because the base64 process will add computing overhead as well as bloat the files by 4/3rds so a 300mb file would take up 400mb of space once base64'd. (That being said it could make since for the sake of consistency, consistency is good for automation and maintainability, and compute/storage resources.)

Here's how I'd use Vault if I needed to support large secrets:
I'd write a wrapper python script to get and fetch secrets from vault, and I'd have 3 scenarios, 2 reserved keywords, and the following naming convention/logic:

1. For secrets > 375kb
   secret/filename bigfile:json containing a symmetric encryption key and the location of an encrypted file stored in a spot designed to store large files.
   Wrapper script would recognize "bigfile" as a reserved keyword, and execute logic to parse json, download encrypted file from file store (Torrent/TFP server/CephFS Path/Azure Blob/AWS S3/GCP Cloud Storage), and decrypt the file a for me to my current context.
2. For secret binary files < 375kb
   secret/filename base64dfile:1 line string of characters representing the base64 encoded version of the binary file
   Wrapper script would recognize "base64dfile" as a reserved keyword, and execute logic to unbase64 it and convert to a file upon fetching it.
3. For text files (.json with secrets, .yamls with secrets, .pem certs, etc) < 375
   secret/filename filename:filecontents as multiline strings are allowed