Let's be honest – every application will have issues. And once you deploy to the cloud, it'll likely have even more issues. To better debug applications targeting Google Cloud, it would be nice to be able to run them locally as if they were running in the cloud, so you can deep dive into certain areas of the code without all of the feedback loops of a deploy.
Some of the potential issues you'll run into will result from security context or configurations. In order to reproduce them, you have to be able to test with the exact same context locally as would exist in a production cloud environment. This begs the question:
How can you securely run an app locally with the exact same context as on Google Cloud?
That means 2 things:
- The app must run with the same security context – permissions and principal.
- The app requires no code change/update – or doesn’t implement a condition that changes the app behavior if it runs locally or in the cloud.
The Google Cloud authentication mode
Before exploring the different solution, let’s have a look at the Google Cloud environment security mechanism.
The Google Cloud client libraries that you use automatically detect the runtime environment to obtain the credentials. The mechanism is named ADC: Application Default Credential.
On Google Cloud, every service has a metadata server which exposes endpoints, and for some, generates short lived tokens and without exposing any secrets.
By default, the default service account of the service is used to generate those tokens. However, many services allow you to customize the service account to use. It’s a good practice to use a custom service account on that service, and to grant only the required permissions on it for a dedicated workload.
If you have several workloads that use the same service, you can have 1 custom service account per workload and narrow the granted permissions.
It’s the least privilege principle.
The default service accounts have too wide permissions like Project Editor, and it’s dangerous to allow such powerful permissions by default everywhere!
If you use a service account key file on Google Cloud environment, I recommend you upgrade your code and application to remove it. The service account key files are only files and it’s dangerous and difficult, to manage them safely. I already wrote an article on that topic.
The most secure way to keep secrets secret, is not to have secrets!
Service account use is a bad practice – even if you watch videos, follow tutorials or read documentation EVEN IF COMING FROM GOOGLE CLOUD!
The 3 legacy ways to achieve it
To reproduce the ADC locally, and running the application in the exact same security condition as on Google Cloud, there are 3 existing (but not satisfying solutions).
Let’s discover what they are, and why they aren’t good.
Method 1: Use service account key files
It’s the most common way to achieve it in the Google Cloud security environment. Lot of articles and videos use (and abuse) it. Mainly because it’s easy.
The principle is :
- Download a service account key file locally and save it as JSON
- Define an environment variable
GOOGLE_APPLICATION_CREDENTIALthat points to the absolution path of the key file on your workstation.
- Run your application.
The ADC mechanism inside the Google Cloud client libraries automatically detects the environment variable and uses it as credential source without the need to change the code.
The main problem of that solution is the service account key file itself. It’s a secret. Not many in the development teams are aware of that, and the impact can lead to a leakage.
When you download a secret on your workstation, is it well secured? Will you delete the file after your tests? Will you do that with your production environment? Are you sure?
In addition, because you directly use the service account identity, it’s impossible to differentiate who has performed the request: the application on your workstation or on Google Cloud?
In the audit logs, there is no difference, it’s the same identity. That’s a serious problem in the case of internal security issues. It’s impossible to trace the source of the issue/attack/data thief.
This solution isn’t really acceptable from a security perspective.
Method 2: Grant your user account the same permissions
To prevent the service account key file download, you can use your own credentials, and grant the same permissions as the Google Cloud service account in your own user account.
The ADC mechanism inside the Google Cloud client libraries detects your user account credentials stored in a “well-known” locations (usual local directories), and loads them as source credentials. No need to change the code.
You have the same permission as the service account on the project, but YOU aren’t the service account. Your principal isn’t the same.
In addition, if the service account email has been granted on another resource, in another project (on a BigQuery dataset hosted in another project), your user account won’t, and you won’t be able to reproduce the exact same workload.
For a small app limited to 1 project, the solution can work. But we can’t rely on it for all apps.
Method 3: Impersonate a service account in your code
To have the exact same permission in all the projects, and to have the same credential as the Google Cloud service account on your local environment:
- You must use the service account credential.
- You must not use a service account key file.
Here comes the service account impersonation feature.
The impersonation is the ability to use your user credentials to generate OAuth tokens on behalf of the service account.
It’s a powerful feature that allows reuse of service account permissions without downloading a secret key.
In addition, your principal is logged in the impersonation delegate chain to know the originator of the request. You have a clear view of WHO is using the service account.
The problem comes from the integration with the Google Cloud client libraries. The feature is supported but you need to add a piece of code to activate the impersonation.
When you want to test locally your app because you have security issues, and the first thing that you do is to change the security logic, it’s obviously absurd!
A new and better impersonation method
So, if we could combine all the pros without the cons, we will have the perfect solution: impersonate a service account, but without changing the code and letting the Google Cloud client library manage that for us.
That feature exists!
It’s a brand new feature and few know it. The idea is to configure the account impersonation outside of the app code, with the
**gcloud** CLI. For that, you can use that command
1gcloud auth application-default login \ 2--impersonate-service-account=<service account email>
Pretty simple, no?!
Then run your app, and it will run with the same service account credentials as your Google Cloud environment by impersonation! The Google Cloud client libraries are “awesome”!
Of course you must be a Service Account User on the service account to be able to impersonate it.
Awesome? quite. Simple? Yes. Does it work? Not sure!!
Indeed, I initially tested it with a Golang app and it didn’t work. It was too new and the feature wasn’t implemented in the Google Cloud Golang client libraries. I implemented it and it was merged in October.
So now, it works for Golang (perform a
go get golang.org/x/oauth2 to download the latest version on your local environment). It also works in Java.
From my latest tests, it wasn’t the case in Python (feel free to contribute if you need that feature!)
Update security context by configuration.
The reproducibility on your local environment is really critical to quickly understand and resolve the problems, but security is certainly more important when you work with sensitive applications in production.
This table summarizes the 4 solutions
Solutions are multiple, some are legacy, others are quicker. However, keep security in mind at any stage of the development, and even after the development!
The Service Account impersonation with
**gcloud** CLI hasn’t any red flag but is not yet perfectly supported by the client libraries of all the languages. Anyway, try it first!!
Google Cloud security products are gaining more and more features that help you to avoid bad practices and to keep your environment clean and safe!