Automating gsutil commands

Question

I'm trying to automate some gsutils commands, but struggling to see where the authentication files are kept and how to re-use (if thats what happens).

I've gone through the gcloud init process in bash...

curl https://sdk.cloud.google.com | bash
gcloud init

All works well when I run

  'gsutil ls'

Now I'm trying to automate the process, so this would work on a new server adding into a crontab on it (rather than creating a new config each time).

I saw a mention of setting env variable GOOGLE_APPLICATION_CREDENTIALS, so I copied my credentials from web login to a file and tried it, eg trying as a different user to test

export GOOGLE_APPLICATION_CREDENTIALS=/home/user/.gsutil/mycreds

and then gsutil ls, but fails.

So I assume I've got the whole credentials thing a bit wrong. I'm assuming there is a file somewhere that was originally created by gcloud which I could use, but I can't see it anywhere ?

I've looked at the answer here but doesn't seem up to date now, as per last comment.

Edit: I have followed Zacharys steps, gcloud auth activate-service-account --key-file=myfilelocation

However, with 'gsutil ls' I now get..

You are attempting to perform an operation that requires a project id, with none configured. Please re-run gsutil config and make sure to follow the instructions for finding and entering your default project id.

So my next question would be, where is it looking for the project id ? If I run gsutil config, it seems to create a new set of auth which then creates another error, so have removed that.

Answer 1

You should be able to do this without diving in too deep to the implementation of authentication for gsutil.

If you're using standalone gsutil (if you installed via this method), the instructions in the linked question are still valid (as Travis points out).

If you'd like to continue using the gsutil supplied via the Cloud SDK, you should use service accounts. Service accounts are the preferred method of authenticating on headless machines or in non-interactive contexts.

Your flow would look something like the following:

1. Create a service account via the Google Cloud Developers Console.
2. On the remote machine, install the Cloud SDK and gsutil. If you're not installing interactively, it's better to skip the curl ... | bash method. Instead, download this install archive, extract it, and run the install.sh script. This script has options (visible with --help); if you specify choices to all of these options, it won't prompt you.
3. Copy the service account to the remote machine. Run gcloud auth activate-service-account --key-file=/path/to/service-account.json.
4. Run gsutil. You should be appropriately authenticated.