Dear Keys, are you still alive ?

2 years ago I had a dream ! Back then my company was asking me for detailed information about Google Cloud service account key usage and I was having …nothing to work with at that time.

While this has now changed, remember most importantly that you should :

“Only use service accounts key files where appropriate”

Ok, you know service accounts have to be used only where appropriate, and this is more true for service account keys, check the official Google Blog :

https://cloud.google.com/blog/products/identity-security/how-to-authenticate-service-accounts-to-help-keep-applications-secure

The problem with a key is its lifecycle — we know that it’s recommended to rotate them, to use them only when mandatory, and to delete them when not use …

In my previous company, we had 2000 service accounts and 5000+ keys and it was impossible to clean up the keys as we didn’t know which one were already in use.

So now my ambition is to be able to know for each key that I have created :

• is it still alive ?
• how many times has it been used ?

That will help me to clean the unused keys I hope.

Monitoring usage for service accounts and keys

Documentation says I can show info about my keys with Cloud Operations Metrics

While that’s quite helpful, it’s not enough for me :

• Because I have many keys from different project, and the metrics page is too small for that
• Because the metrics page only shows the Key_Id, not the service account or the project
• Because I can’t share the metrics page with my boss
• Because it’s just too technical

So, here is what I’m doing :

Service Account Keys monitoring

Operations have been split across different Cloud Functions to keep things as simple as possible, and to avoid timeout or quota problems. Scalability is important.

** I will focus on the API Call only, not on global architecture. You can find on Google documentation how to trigger, call Pub/Sub etc …**

A daily Cloud Scheduler job will trigger a Cloud Function to parse all my GCP projects by calling Ressource Manager API v1 with the Discovery method :

import google.auth
from googleapiclient import discovery  

credentials, project = google.auth.default(
scopes=['https://www.googleapis.com/auth/cloud-platform'])

ressourceManagerClient = discovery.build('cloudresourcemanager', 'v1', credentials=credentials)
projects_list = ressourceManagerClient.projects().list().execute()
for project in projects_list['projects']
    project_id = project['projectId']

Now for each project a Cloud Function is called with the IAM API v1 to get the service account list of the project and for each one, the list of the created keys. The Google documentation say :

The projects.serviceAccounts.keys.list method lists all of the service account keys for a service account.

Before using any of the request data below, pass the following arguments:

• PROJECT_ID: Your Google Cloud project ID. Project IDs are alphanumeric strings, like myproject.
• SA_NAME: The name of the service account whose keys you want to list.
• KEY_TYPES: Optional. A comma-separated list of key types that you want to include in the response. The key type indicates whether a key is user-managed (USER_MANAGED) or system-managed (SYSTEM_MANAGED). If left blank, all keys are returned. ** USER_MANAGED are keys that you have created, SYSTEM are Google generated keys

import google.auth
from googleapiclient import discovery
credentials, project = google.auth.default(
scopes=['https://www.googleapis.com/auth/cloud-platform'])

iamClient = discovery.build('iam', 'v1', credentials=credentials)
accounts_list = iamClient.projects().serviceAccounts().list(name="projects/myproject").execute()
for service_account in accounts_list['accounts']:
key =   iamClient.projects().serviceAccounts().keys().list(name=service_account['name']).execute()
    for sa_key in key['keys']:
      if sa_key['keyType'] == “USER_MANAGED”:
         Key_ID = (key['name']).split("keys/"))[1]
         Service_Account = accounts_list['name']
         Service_Account_ID = accounts_list['uniqueId']

Now for each Service Account Key I have to ask the Time Series API v3 for all the events where the Key have been used during a specific time period :

Check out the Google documentation about this TimeSeries call :

The Cloud Monitoring API’s timeSeries.list method , when used with specific filters, allows you to get usage metrics for a single service account key. You can then use those metrics to determine when the key was last used.

Before using any of the request data below, use the following arguments:

• PROJECT_ID: Your Google Cloud project ID. Project IDs are alphanumeric strings, like myproject.
• KEY_ID: The unique ID of your service account key.
• END_TIME: The end of the time interval that you want to check, in percent-encoded RFC 3339 format. For example, 2020-06-12T00%3A00%3A00.00Z.
• START_TIME: The start of the time interval that you want to check, in percent-encoded RFC 3339 format. For example, 2020-04-12T00%3A00%3A00.00Z.

import google.auth
import json
from google.auth.transport.requests import AuthorizedSession
credentials, project = google.auth.default(
scopes=['https://www.googleapis.com/auth/cloud-platform'])
TIMESERIES_URL = "https://monitoring.googleapis.com/v3/projects/myproject/timeSeries?filter=metric.type%3D%22iam.googleapis.com%2Fservice_account%2Fkey%2Fauthn_events_count%22%20AND%20metric.labels.key_id%3D%22"+str(KEY_ID)+"%22&interval.endTime=2021-04-26T15:01:23Z&interval.startTime=2021-04-20T15:01:23Z"
AUTHORIZED_SCOPE = ["https://www.googleapis.com/auth/cloud-platform"]
credential = google.auth.default(scopes=AUTHORIZED_SCOPE)[0]
authed_session = AuthorizedSession(credential)
response = authed_session.request(
method="GET", url=TIMESERIES_URL)
key_usage = response.json()
test =  "timeSeries" in key_usage
if test is True:
   timeSeries = key_usage['timeSeries'][0]['points']
   for window in timeSeries:
     print(window['interval']['startTime'])
     print(window['interval']['endTime'])
     print(window['value']['int64Value'])

I will get the amount of calls made for each key with a 10-min time window (between start and end time)

The last step is to send all this data to BigQuery :

from google.cloud import bigquery
bigqueryClient = bigquery.Client()
row = [
    {
      u"project_id": var_projectid,
      u"sa_email": var_saemail,
      u"sa_id": var_said,
      u"key_id": var_keyid,
      u"start_time": var_start,
      u"end_time": var_end,
      u"value": var_value
    }
]
bigqueryClient.insert_rows_json("dataset.table", row, row_ids=[None] * len(row))

Now I have a row per project / per service account / per key / per 10-min window :

Remember that if my key is not used in the specific window nothing is logged, so nothing appears, according to the documentation :

Service accounts and service account keys appear in these metrics if they are used to call any Google API, including APIs that are not part of Google Cloud.

The metrics include both successful and failed API calls.

For example, if an API call fails because the caller is not authorized to call that API, or because the request referred to a resource that does not exist, the service account or key that was used for that API call appears in the metrics.

Now let’s see what we can do and discover with DataStudio :

Oh yeah! I can see that my Service Account Key is used a lot. Otherwise, if I can’t find it in my Dashboard, that means I can probably delete it.

Check out these useful BigQuery Tips to analyze the data :

Query the project where there is the biggest usage of Keys :
SELECT
   SUM(value)as totalCount, projectId
FROM
   'my-project.mydataset.mytable'
GROUP BY projectId
ORDER BY totalCount DESC
Query the Keys that have not been used since the last 30 days:
SELECT
   keyid,
   LastUse
   FROM (
      SELECT
         keyid,
         MAX(start) AS LastUse
      FROM
         'my-project.mydataset.mytable'
      GROUP BY keyid
      ORDER BY LastUse DESC )
WHERE LastUse < TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 day)

Conclusion

I can now have visibility about what’s going on with my GCP Service Account Keys, I can share this easily, and take actions !

Check out the repo for some sample code here.