Ingest Developer Getting Started

Ingest Architecture

How to update the diagram ?

This diagram is generated using diagrams.net. The .drawio file is found in AIT Shared drive AIT > HCA Ingest > Documentation > Ingest Service_v2.drawio To embed, go to File > Embed > HTML... create and copy the html and paste in this .md file.

Follow config when creating the html to embed:

  • Links: Select Open in New Window
  • Lightbox > Select Edit: Custom - link to actual draw.io file

Table of contents

  1. Ingest Architecture
  2. How to update the diagram ?
  3. Access
  4. Licenses
    1. JetBrains Licenses
  5. Software
    1. MongoDB
    2. RabbitMQ
    3. Repositories
      1. Ingest-Core (InteliJ)
        1. Run Dependencies in Docker
        2. Run Locally
      2. Ingest-Broker (PyCharm)
        1. Run in Docker
        2. Run Locally
      3. Ingest-UI
        1. Ingest-Client
      4. DevOps
    4. Some main docs
    5. Bookmarks

Access

  • create an ebi account
  • github
  • Slack
  • EBI GitLab
  • login to EBI AWS
    1. need to be in Google Group grp-aws-ait-team-power-users. If you are not a member, talk to your supervisor.
    2. You can click on ‘Management console’ to access console or use ‘Command line or programmatic access’ to access aws CLI. Add to your profile (~/.aws/config, ~/.aws/credentials):
      • aws_access_key_id=
      • aws_secret_access_key=
      • aws_session_token=
    3. note: The CLI Keys along with browser login session are short lived and has a timeout of 4hr.
    4. Please reach to Cloud-consultants for any connection problems.
  • Quay.io - for built docker image storage
  • Confluence
  • Zenhub
  • Google Apps (Google Drive Shared HCA folder & Calendar) - use EBI Single sign-on
  • Register at Snyk. If Snyk authentication fails during your CI/CD pipeline, follow these steps to diagnose and resolve the issue:
    • Authenticate: Run snyk auth in your terminal and follow prompts.
    • Retrieve Token: Check the token stored locally: 
        cat ~/.config/configstore/snyk.json
    • Add Variable: Set SNYK_TOKEN variable to Settings > CI/CD in GitLab with your token value.
    • If you need to integrate Snyk into the Dockerfile, modify your Dockerfile to pass the Snyk token during build.
  • mailing lists:
  • Calendar

Licenses

JetBrains Licenses

  • IntelliJ IDEA Ultimate - can be used for java/python, node
  • Alternatively, the language packs can be used: PyCharm Professional, WebStorm
  1. follow this EBI procedure:
  2. Ask for to be included in the “JetBrains All Products Pack”

Other software uses free/community licensing.

Software

Caveat: All practical descriptions for Mac/Linux.

  1. Homebrew
  2. git and git-secrets
  3. Docker Desktop
  4. Kubernetes tools: follow see ingest-kube-deployment repo
  5. docker-machine-driver-hyperkit 
    brew install docker-machine-driver-hyperkit
sudo chown root:wheel /usr/local/opt/docker-machine-driver-hyperkit/bin/docker-machine-driver-hyperkit
sudo chmod u+s /usr/local/opt/docker-machine-driver-hyperkit/bin/docker-machine-driver-hyperkit

MongoDB

Follow the MongoDB installation guide for your platform.

RabbitMQ

Follow the RabbitMQ installation guide for your platform.

Repositories

Ingest-Core (InteliJ)

see readme

Run Dependencies in Docker 

docker-compose up mongo rabbitmq

Run Locally

  1. Add Plugin for Project Lombok
    • usually not necessary, come by default with idea
  2. Edit Preferences: Preferences Build, Execution, Deployment Build Tools Gradle
    1. Build and run using: Gradle
    2. Run tests using: IntelliJ IDEA
  3. run Spring Boot with profile local (see src/main/resources/application-local.properties)

Ingest-Broker (PyCharm)

Run in Docker 

docker build -t humancellatlas/ingest-broker .
docker-compose up

Run Locally

Install Python 3.9 or higher

brew install pyenv
pyenv install --list
pyenv install 3.7.4

Will return something like:

Installed Python-3.7.4 to /Users/<username>/.pyenv/versions/3.7.4

Preferences | Project: ingest-broker | Project Interpreter [[/images/image1.png]]

Ingest-UI

  1. Install NVM 
    curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.34.0/install.sh | bash
  2. Add to ~/.bash_profile, Restart Terminal 
    source ~/.bashrc
  3. Install Node 
    nvm install node@12.16.3
  4. Install Yarn 
    brew install yarn@1.22.4
  5. Add to ~/.bashrc, Restart Terminal 
    export PATH="$PATH:/usr/local/Cellar/yarn/1.22.4/bin"
  6. Install global dependencies (ingest-ui/client): 
    npm install -g typescript@"<3.3.0"
npm install -g node-sass@4.12.0
  7. Install Local Packages (ingest-ui/client): 
    yarn install
  8. Install Local Packages (ingest-ui): 
    yarn install
  9. Set up WebStorm for starting Angular
    1. Edit Preferences: Run/Debug configurations Add new configuration (npm)
    2. Create for dev and local environments that run dev and local npm scripts
    3. optional create for other environments too (listed in package.json scripts)

Ingest-Client

Preferences | Project: ingest-client | Project Interpreter [[/images/image2.png]]

DevOps

  1. Verify you can login to aws using your EBI credentials.
  2. Setup our Infrastructure-as-Code tools following the readme in ingest-kube-deployment
  3. If necessary, read up on Docker and containerisation, get familiar with Kubernetes basics, and Helm for application management in Kubernetes.

Some main docs

In Ingestion Service/Dev folder

Bookmarks