Setting up access to Terra staging area

Ingest submits HCA data to Terra staging area which is a bucket in Google Cloud Platform (GCP). The GCP buckets locations are configured in <env>.yaml files in ingest-kube-deployment/apps

e.g.

For dev environment, find the value of terraBucketName and terraBucketPrefix in the dev.yaml

The Terra staging area GCP bucket location should be in: 

gs://<terraBucketName>/<terraBucketPrefix>

so the actual bucket location is:

gs://broad-dsp-monster-hca-dev-ebi-staging/dev

Using your Google account

  1. Ensure you are in the dcp-ingest-team google group
    • Everyone in the dcp-ingest-team google group should have read access to the staging bucket
    • New members can be added to this group by group owners, currently:
    • Amnon, Tony, Claire, Oihane

It is the group owners’ responsibility to ensure this list is kept up to date and that no one who shouldn’t have access is in the group. If you notice someone who should no longer have access, please let the group owners know

  1. Install gsutil and login using your google account. You could follow instructions

You can also access the bucket via a browser interface at:

Using Ingest Exporter’s GCP service account

  1. Install gsutil

  2. Download the Ingest Exporter’s GCP service account credentials from AWS Secrets. Currently, only Ingest Developers have access to this secret. 
    aws secretsmanager get-secret-value \
 --profile=embl-ebi \
 --region us-east-1 \
 --secret-id ingest/dev/secrets \
 --query SecretString \
 --output text | jq -jr '.ingest_exporter_terra_svc_account' > any-secured-directory/terra-gcp-credentials-dev.json

    Currently, only Ingest developers have access to this secret.

  3. Configure gsutil to use the GCP credentials 
    gcloud auth activate-service-account --key-file KEY_FILE
  4. Access the GCP bucket for TDR staging area 
    $ gsutil ls  gs://broad-dsp-monster-hca-dev-ebi-staging/dev