Skip to content

deployment variable in a (secure) environment variable #349

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
peterjc wants to merge 6 commits into
base: master
Choose a base branch
from
Draft

deployment variable in a (secure) environment variable #349

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
eda2e82
Define new command line API for key in environment variable
peterjc Jun 26, 2019
303d346
First pass at the configure command
peterjc Jun 26, 2019
b8eb11b
Polishing config with --dkenv
peterjc Jun 26, 2019
5c9f2d8
Cope with git or https clone url
peterjc Jun 27, 2019
65e31de
Refactor decrypt_file to introduce write_private_key
peterjc Jun 27, 2019
6b30470
Write the --dkenv private key to disk
peterjc Jun 27, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 60 additions & 20 deletions doctr/__main__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,8 @@

on your local machine. This will prompt for your GitHub credentials and the
name of the repo you want to deploy docs for. This will generate a secure key,
which you should insert into your .travis.yml.
which you should insert into your .travis.yml (or set as a secure environment
variable in your TravisCI repository settings if using the --dkenv option).

Then, on Travis, for the build where you build your docs, add::

Expand Down Expand Up @@ -134,6 +135,10 @@ def get_parser(config=None):
if we do not appear to be on Travis.""")
deploy_parser_add_argument('deploy_directory', type=str, nargs='?',
help="""Directory to deploy the html documentation to on gh-pages.""")
deploy_parser_add_argument('--dkenv', type=str, metavar="ENVVAR",
Copy link
Member

@asmeurer asmeurer Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should be able to detect this automatically.

Copy link
Author

@peterjc peterjc Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, personally I think the interface is much simpler and more flexible if the user picks the environment variable name(s) themselves.

Copy link
Member

@asmeurer asmeurer Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can make a flag if the user wants to change it, but there should be a default. It should be possible to just use the variable names we already use, and detect based on the content if it is pointing to a file or if it is a private key. Most users don't really care how doctr works. They just do whatever it says to do and if it pushes their stuff up to gh-pages they are happy.

help="""Push to GitHub using a deployment key stored in the named
environment variable via your TravisCI repository settings.
Use this if you used 'doctr configure --dkenv ENVVAR'.""")
deploy_parser_add_argument('--token', action='store_true', default=False,
help="""Push to GitHub using a personal access token. Use this if you
used 'doctr configure --token'.""")
Expand Down Expand Up @@ -193,6 +198,11 @@ def get_parser(config=None):
configure_parser.set_defaults(func=configure)
configure_parser.add_argument('--force', action='store_true', help="""Run the configure command even
if we appear to be on Travis.""")
configure_parser.add_argument('--dkenv', type=str, metavar="ENVVAR",
Copy link
Member

@asmeurer asmeurer Jul 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use the existing environment variable names. If this works we should use it by default and have the old way behind a flag (or just remove it).

Copy link
Author

@peterjc peterjc Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If you would be happy making the deploy key in an environment variable the default, then maybe you are right - re-using the old environment variable names would make sense.

Copy link
Member

@asmeurer asmeurer Jul 2, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Assuming there are no issues with it we should. We will have to continue to support the old way at least for deploy for existing repos. It will need a lot of testing to make sure, once we get it working.

help="""Generate a deployment key to push to GitHub. The public key
will be added to your GitHub repository settings. The private key
should be added to you TravisCI repository settings as a protected
environment variable.""")
configure_parser.add_argument('--token', action="store_true", default=False,
help="""Generate a personal access token to push to GitHub. The default is to use a
deploy key. WARNING: This will grant read/write access to all the
Expand Down Expand Up @@ -275,6 +285,9 @@ def deploy(args, parser):

config = get_config()

if args.token and args.dkenv:
parser.error("The --token and --dkenv settings are incompatible.")

if args.tmp_dir:
parser.error("The --tmp-dir flag has been removed (doctr no longer uses a temporary directory when deploying).")

Expand Down Expand Up @@ -310,10 +323,10 @@ def deploy(args, parser):

canpush = setup_GitHub_push(deploy_repo, deploy_branch=deploy_branch,
auth_type='token' if args.token else 'deploy_key',
full_key_path=keypath,
full_key_path=None if args.dkenv else keypath,
branch_whitelist=branch_whitelist,
build_tags=args.build_tags,
env_name=env_name)
env_name=args.dkenv if args.dkenv else env_name)

if args.sync:
built_docs = args.built_docs or find_sphinx_build_dir()
Expand Down Expand Up @@ -383,6 +396,13 @@ def configure(args, parser):
parser.error(red("doctr appears to be running on Travis. Use "
"doctr configure --force to run anyway."))

if args.token and args.dkenv:
parser.error("The --token and --dkenv settings are incompatible.")

if len(args.dkenv.split()) != 1:
# Not going to repeat this sanity test in the deploy command:
parser.error("The --dkenv setting should be one word only, e.g. DOC_KEY.")

if not args.authenticate:
args.upload_key = False

Expand Down Expand Up @@ -484,11 +504,17 @@ def configure(args, parser):
deploy_key_repo, env_name, keypath = get_deploy_key_repo(deploy_repo, args.key_path)

private_ssh_key, public_ssh_key = generate_ssh_key()
key = encrypt_to_file(private_ssh_key, keypath + '.enc')
del private_ssh_key # Prevent accidental use below
if args.dkenv:
key = None # don't need it on disk
encrypted_variable = None # not applicable
private_ssh_key = private_ssh_key.decode('ASCII') # Will print this later!
else:
key = encrypt_to_file(private_ssh_key, keypath + '.enc')
encrypted_variable = encrypt_variable(env_name.encode('utf-8') + b"=" + key,
build_repo=build_repo, tld=tld,
travis_token=travis_token, **login_kwargs)
private_ssh_key = None # Prevent accidental use below
public_ssh_key = public_ssh_key.decode('ASCII')
encrypted_variable = encrypt_variable(env_name.encode('utf-8') + b"=" + key,
build_repo=build_repo, tld=tld, travis_token=travis_token, **login_kwargs)

deploy_keys_url = 'https://github.com/{deploy_repo}/settings/keys'.format(deploy_repo=deploy_key_repo)

Expand All @@ -509,16 +535,17 @@ def configure(args, parser):
and add the following as a new key:{RESET}

{ssh_key}

{BOLD_MAGENTA}Be sure to allow write access for the key.{RESET}
""".format(ssh_key=public_ssh_key, deploy_keys_url=deploy_keys_url, N=N,
BOLD_MAGENTA=BOLD_MAGENTA, RESET=RESET)))

if not args.dkenv:
print(dedent("""\
{N}. {BOLD_MAGENTA}Add the file {keypath}.enc to be staged for commit:{RESET}

print(dedent("""\
{N}. {BOLD_MAGENTA}Add the file {keypath}.enc to be staged for commit:{RESET}

git add {keypath}.enc
""".format(keypath=keypath, N=N, BOLD_MAGENTA=BOLD_MAGENTA, RESET=RESET)))
git add {keypath}.enc
""".format(keypath=keypath, N=N, BOLD_MAGENTA=BOLD_MAGENTA, RESET=RESET)))

options = '--built-docs ' + bold_black('<path/to/built/html/>')
if args.key_path:
Expand All @@ -531,23 +558,36 @@ def configure(args, parser):
options += ' --token'
key_type = "personal access token"

if args.dkenv:
options += ' --dkenv ' + args.dkenv
print(dedent("""\
{N}. {BOLD_MAGENTA}Add the following private deployment key to your TravisCI
repository settings as environment variable {env_name}:{RESET}
""".format(N=N, BOLD_MAGENTA=BOLD_MAGENTA, RESET=RESET,
env_name=args.dkenv, private_ssh_key=private_ssh_key)))
print(private_ssh_key)

print(dedent("""\
{N}. {BOLD_MAGENTA}Add these lines to your `.travis.yml` file:{RESET}
""".format(N=N, BOLD_MAGENTA=BOLD_MAGENTA, RESET=RESET)))

env:
global:
# Doctr {key_type} for {deploy_repo}
- secure: "{encrypted_variable}"
if not args.dkenv:
print(dedent("""\
env:
global:
# Doctr {key_type} for {deploy_repo}
- secure: "{encrypted_variable}"
""".format(key_type=key_type,
encrypted_variable=encrypted_variable.decode('utf-8'))))

print(dedent("""\
script:
- set -e
- {BOLD_BLACK}<Command to build your docs>{RESET}
- pip install doctr
- doctr deploy {options} {BOLD_BLACK}<target-directory>{RESET}
""".format(options=options, N=N, key_type=key_type,
encrypted_variable=encrypted_variable.decode('utf-8'),
deploy_repo=deploy_repo, BOLD_MAGENTA=BOLD_MAGENTA,
BOLD_BLACK=BOLD_BLACK, RESET=RESET)))
""".format(options=options, deploy_repo=deploy_repo,
BOLD_BLACK=BOLD_BLACK, RESET=RESET)))

print(dedent("""\
Replace the text in {BOLD_BLACK}<angle brackets>{RESET} with the relevant
Expand Down
71 changes: 59 additions & 12 deletions doctr/travis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,20 @@
from .common import red, blue, yellow
DOCTR_WORKING_BRANCH = '__doctr_working_branch'

def write_private_key(filename, private_key_bytes):
"""
Helper function to record (decrpted) private key to as file for ssh.
"""
with open(filename, 'wb') as f:
f.write(filename)
os.chmod(filename, 0o600)

def write_key_from_env_var(filename, env_var):
"""
Write private key from environment variable named in --dkenv to disk.
"""
write_private_key(filename, os.environ[env_var])

def decrypt_file(file, key):
"""
Decrypts the file ``file``.
Expand All @@ -41,10 +55,7 @@ def decrypt_file(file, key):
with open(file, 'rb') as f:
decrypted_file = fer.decrypt(f.read())

with open(file[:-4], 'wb') as f:
f.write(decrypted_file)

os.chmod(file[:-4], 0o600)
write_private_key(file[:-4], decrypted_file)

def setup_deploy_key(keypath='github_deploy_key', key_ext='.enc', env_name='DOCTR_DEPLOY_ENCRYPTION_KEY'):
"""
Expand Down Expand Up @@ -165,8 +176,13 @@ def get_current_repo():
'remote.origin.url']).decode('utf-8')

# Travis uses the https clone url
_, org, git_repo = remote_url.rsplit('.git', 1)[0].rsplit('/', 2)
return (org + '/' + git_repo)
# e.g. https://github.com/<owner>/<repo>.git
# If run outside Travis using --force, might have:
# e.g. git@github.com:<owner>/<repo>.git
if remote_url.endswith(".git"):
remote_url = remote_url[:-4]
_, owner, repo = remote_url.replace(":", "/").rsplit("/", 2)
return owner + '/' + repo

def get_travis_branch():
"""Get the name of the branch that the PR is from.
Expand All @@ -189,12 +205,22 @@ def setup_GitHub_push(deploy_repo, *, auth_type='deploy_key',
"""
Setup the remote to push to GitHub (to be run on Travis).

``auth_type`` should be either ``'deploy_key'`` or ``'token'``.
``auth_type`` should be ``'deploy_key'`` (encrpted deplyoment key
on disk), ``'dkenv'`` (deployment key in environment variable), or
``'token'``.

For ``auth_type='token'``, this sets up the remote with the token and
checks out the gh-pages branch. The token to push to GitHub is assumed to be in the ``GH_TOKEN`` environment
For ``auth_type='deploy_key'``, this sets up the remote with ssh access.
assuming the private deploement key is in the ``full_key_path`` file
and can be decrypted with the ``env_name`` environment variable.

For ``auth_type='dkenv'``, this sets up the remote with ssh access
assuming the private deployment key is in the ``env_name`` environment
variable.

For ``auth_type='token'``, this sets up the remote with the token and
checks out the gh-pages branch. The token to push to GitHub is assumed
to be in the ``GH_TOKEN`` environment variable.

For ``auth_type='deploy_key'``, this sets up the remote with ssh access.
"""
# Set to the name of the tag for tag builds
Expand All @@ -210,8 +236,8 @@ def setup_GitHub_push(deploy_repo, *, auth_type='deploy_key',
stacklevel=2)
branch_whitelist.add('master')

if auth_type not in ['deploy_key', 'token']:
raise ValueError("auth_type must be 'deploy_key' or 'token'")
if auth_type not in ['deploy_key', 'dkenv', 'token']:
raise ValueError("auth_type must be 'deploy_key', 'dkenv', or 'token'")

TRAVIS_BRANCH = os.environ.get("TRAVIS_BRANCH", "")
TRAVIS_PULL_REQUEST = os.environ.get("TRAVIS_PULL_REQUEST", "")
Expand All @@ -220,7 +246,15 @@ def setup_GitHub_push(deploy_repo, *, auth_type='deploy_key',
TRAVIS_REPO_SLUG = os.environ["TRAVIS_REPO_SLUG"]
REPO_URL = 'https://api.github.com/repos/{slug}'
r = requests.get(REPO_URL.format(slug=TRAVIS_REPO_SLUG))
fork = r.json().get('fork', False)

if auth_type == 'dkenv':
# Here we don't care if we are on a fork or not - one of the reasons
# for putting the key in a TravisCI secure environment variable is
# to allow things like setting up test source and deployment repos
# under a personal fork.
fork = False
else:
fork = r.json().get('fork', False)

canpush = determine_push_rights(
branch_whitelist=branch_whitelist,
Expand All @@ -230,6 +264,14 @@ def setup_GitHub_push(deploy_repo, *, auth_type='deploy_key',
TRAVIS_TAG=TRAVIS_TAG,
build_tags=build_tags)

if auth_type == 'dkenv':
if args.dkenv not in os.environ:
print("WARNING: Environment variable {dkenv} not set".format(dpenv=args.dkenv))
canpush = False
elif not os.environ[args.dkenv]:
print("WARNING: Environment variable {dkenv} empty".format(dpenv=args.dkenv))
canpush = False

print("Setting git attributes")
set_git_user_email()

Expand All @@ -244,6 +286,11 @@ def setup_GitHub_push(deploy_repo, *, auth_type='deploy_key',
run(['git', 'remote', 'add', 'doctr_remote',
'https://{token}@github.com/{deploy_repo}.git'.format(token=token.decode('utf-8'),
deploy_repo=deploy_repo)])
elif auth_type == 'dkenv':
# TODO - setup the key
write_key_from_env_var(full_key_path.rsplit('.', 1)[0], args.dkenv)
run(['git', 'remote', 'add', 'doctr_remote',
'git@github.com:{deploy_repo}.git'.format(deploy_repo=deploy_repo)])
else:
keypath, key_ext = full_key_path.rsplit('.', 1)
key_ext = '.' + key_ext
Expand Down