Skip to content

Development

Jump to bottom
Farley Lai edited this page Apr 30, 2021 · 1 revision

General practices follow the 12 Factor App.

12 Factor App

III. Config

Prep.

  1. AWS credentials from admin
  2. For development machines, store credentials in ~/.aws that no others can access
  3. For EC2/IoT/DeepLens, assign roles to access KMS and S3 to begin with
  4. Create Customer Master Key

Practices

  1. Populate secrets as .env
    • login credentials
      • git/https: user/passwd
      • git/ssh: .pem key
    • API keys
    • Tokens
  2. Encrypt with CMK and upload to S3
  3. Grant users and EC2/IoT instances access to S3
  4. Users and EC2/IoT instances acquires key to decrypt secretes from S3 and export to the runtime environment
  5. The program acquires secrets from the environment variables

References

Coding Style

  • PyTorch source code and documentation
  • Black code formatter for Python

Coding Guidelines

Required

  • modular
    • PyTorch package organization
  • multi-GPU and checkpointing

Professional

Version Control

  • GitLab and Git
  • Git LFS for binaries if not shared
  • /zdata for shared data

Package Management

Use conda as the default package dependency manager whenever possible to avoid unexpected complexities with pip. The convention is maintain at least the two distributions for ease of development and testing:

  • make conda-build: build the distribution for end users to install in the system path
    • pip install --no-deps --ignore-installed --no-use-wheel -vv .
  • make dev-setup: local pip installation for modifications in place
    • pip install --no-deps --ignore-installed --no-use-wheel -vv -e .

Minimal conda packages

Cluster over Slurm

Agile Development Process

Continuous Integration and Delivery

Clone this wiki locally