Skip to content

CustomizingALGoForGitHub.md

Latest commit

 

History

History
348 lines (249 loc) · 21.5 KB

CustomizingALGoForGitHub.md

File metadata and controls

348 lines (249 loc) · 21.5 KB

Customizing AL-Go for GitHub

AL-Go for GitHub is a plug-and-play DevOps solution, intended to support 100% of the functionality needed by 90% of the people developing applications for Microsoft Dynamics 365 Business Central out-of-the-box.

If AL-Go functionality out-of-the-box doesn't match your needs, you should always start by creating a feature suggestion here and see whether your needs are met by other mechanisms or required by other partners and should be part of AL-Go for GitHub.

If your feature should be part of AL-Go for GitHub, you can select to contribute to AL-Go for GitHub yourself or wait for Microsoft or other partners to pickup your feature suggestion.

If your feature suggestion isn't accepted, you really have three options:

  1. Customize AL-Go for GitHub to fit your needs
  2. Select another managed DevOps solution
  3. Create your own DevOps solution from scratch (not recommended)

Creating your own DevOps solution from scratch requires dedicated resources to develop and maintain workflows, processes etc. This is not a small task. There are many moving parts in a DevOps solution, which might require you to make changes to workflows and scripts over time and stay secure and having to maintain many repositories is tedious and time consuming, even when using templates and other advanced features.

Microsoft will continuously develop and maintain AL-Go for GitHub and ensure that we always use the latest versions of GitHub actions, which are under our control. Microsoft will never add dependencies to any third party GitHub action, which are not under our control.

Keeping your repositories up-to-date can be done manually or on a schedule (like Windows update really). You will be notified when an update is available and we recommend that you keep your repositories up-to-date at all time. If you make modifications to the AL-Go System Files (scripts and workflows) in your repository, in other ways than described in this document, these changes will be removed with the next AL-Go update.

Tip

If for some reason the updated version of AL-Go for GitHub doesn't work for you, we recommend that you file an issue here with a detailed description of the problem and full logs of the failing workflows. You can then revert back to the prior version of AL-Go for GitHub until the issue is resolved.

It is important to get back to the mainstream version of AL-Go for GitHub as soon as the issue is resolved.

There are three ways you can customize AL-Go for GitHub to fit your needs. You can

  1. customize the repository with custom scripts, workflows or jobs following the guidelines below
  2. create a customized repository and use this as your custom template repository
  3. fork the AL-Go for GitHub and create your "own" version (not recommended)

Caution

The more you customize AL-Go for GitHub, the more likely you are to be broken by future updates to AL-Go for GitHub, meaning that you will have to update your customizations to match the changes in AL-Go for GitHub.

Customizing your repository

There are several ways you can customize your AL-Go repository and ensure that the changes you make, will survive an update of AL-Go for GitHub.

Hide/Remove unused workflows

By adding a setting called unusedALGoSystemFiles in your repo settings, you can tell AL-Go for GitHub that these system files are not used. Example:

  "unusedALGoSystemFiles": [
    "AddExistingAppOrTestApp.yaml",
    "CreateApp.yaml",
    "CreatePerformanceTestApp.yaml",
    "CreateTestApp.yaml",
    "cloudDevEnv.ps1"
  ]

This setting will cause AL-Go for GitHub to remove these files during the next update. Note that if you remove files like _BuildALGoProject.yaml, AL-Go will obviously stop working as intended - so please use with care.

Custom delivery

You can setup custom delivery in order to deliver your apps to locations not supported by AL-Go for GitHub out-of-the-box, by adding a custom delivery powershell script (named .github/DeliverTo<DeliveryTarget>.ps1) and a context secret (called <DeliveryTarget>Context) formatted as compressed json, you can define the delivery functionality as you like. Example:

Param([Hashtable] $parameters)

Get-ChildItem -Path $parameters.appsFolder | Out-Host
$context = $parameters.context | ConvertFrom-Json
Write-Host "Token Length: $($context.Token.Length)"

In this example the context secret is assumed to contain a Token property. Read this for more information.

Custom deployment

You can setup custom deployment to environment types not supported by AL-Go for GitHub out-of-the-box. You can also override deployment functionality to environment Type SaaS if you like. You can add an environment called <EnvironmentName> and a DeployTo<EnvironmentName> setting, defining which environment Type should be used. Example:

  "Environments": [
    "<EnvironmentName>"
  ],
  "DeployTo<EnvironmentName>": {
    "EnvironmentType": "<EnvironmentType>"
  }

You also need to create an AuthContext secret (called <EnvironmentName>_AuthContext) and a powershell script (named .github/DeployTo<EnvironmentType>.ps1), which defines the deployment functionality. Example:

Param([Hashtable] $parameters)

$parameters | ConvertTo-Json -Depth 99 | Out-Host
$tempPath = Join-Path ([System.IO.Path]::GetTempPath()) ([GUID]::NewGuid().ToString())
New-Item -ItemType Directory -Path $tempPath | Out-Null
Copy-AppFilesToFolder -appFiles $parameters.apps -folder $tempPath | Out-Null
Get-ChildItem -Path $tempPath -Filter *.app | Out-Host
$authContext = $parameters.authContext | ConvertFrom-Json
Write-Host "Token Length: $($authContext.Token.Length)"

In this example the AuthContext secret is assumed to contain a Token property. Read this for more information.

Adding custom workflows

If you add new workflows to the .github/workflows folder, which is unknown to AL-Go for GitHub, AL-Go will leave them untouched. These workflows need to follow standard GitHub Actions schema (yaml) and can be triggered as any other workflows. Example:

name: 'Create Build Tag'

on:
  workflow_run:
    workflows: [' CI/CD','CI/CD']
    types: [completed]
    branches: [ 'main' ]

run-name: "[${{ github.ref_name }}] Create build tag"

permissions: read-all

jobs:
  CreateTag:
    if: github.event.workflow_run.conclusion == 'success'
    runs-on: windows-latest
    steps:
      - name: mystep
        run: |
          Write-Host "Create tag"

It is recommended to prefix your workflows with my, our, your name or your organization name in order to avoid that the workflow suddenly gets overridden by a new workflow in AL-Go for GitHub. The above workflow is a real example from here.

Caution

This workflow gets triggered when the CI/CD workflow has completed. Note that the name of the CI/CD workflow currently is prefixed with a space, this space will very likely be removed in the future, which is why we specify both names in this example. Obviously this workflow would break if we decide to rename the CI/CD workflow to something different.

Adding custom scripts

You can add custom powershell scripts under the .github folder for repository scoped scripts or in the .AL-Go folder for project scoped scripts. Specially named scripts in the .AL-Go folder can override standard functionality in AL-Go for GitHub workflows. A list of these script overrides can be found here. Scripts under the .github folder can be used in custom workflows instead of using inline scripts inside the workflow.

One example of a script override is the NewBcContainer override used in the System Application project in BCApps (can be found here). This override looks like:

Param([Hashtable] $parameters)

$script = Join-Path $PSScriptRoot "../../../scripts/NewBcContainer.ps1" -Resolve
. $script -parameters $parameters

Which basically launches a script located in the script folder in the repository for creating the build container needed for building and testing the System Application. That script can be found here.

Caution

Script overrides will almost certainly be broken in the future. The current script overrides is very much tied to the current implementation of the Run-AlPipeline function in BcContainerHelper. In the future, we will move this functionality to GitHub actions and no longer depend on BcContainerHelper and Run-AlPipeline. At that time, these script overrides will have to be changed to follow the new implementation.

Adding custom jobs

You can also add custom jobs to any of the existing AL-Go for GitHub workflows. Custom jobs can depend on other jobs and other jobs can be made to depend on custom jobs. Custom jobs need to be named CustomJob<something>, but can specify another name to be shown in the UI. Example:

  CustomJob-CreateBuildTag:
    name: Create Build Tag
    needs: [ Initialization, Build ]
    if: (!cancelled()) && (needs.Build.result == 'success')
    runs-on: [ ubuntu-latest ]
    steps:
      - name: Create Tag
        run: |
          Write-Host "Create Tag"

  PostProcess:
    needs: [ Initialization, Build2, Build1, Build, Deploy, Deliver, DeployALDoc, CustomJob-CreateBuildTag ]
    if: (!cancelled())
    runs-on: [ windows-latest ]
    steps:
      ...

Adding a custom job like this, will cause this job to run simultaneously with the deploy and the deliver jobs.

Note

All custom jobs will be moved to the tail of the yaml file when running Update AL-Go System Files, but dependencies to/from the custom jobs will be maintained.

Caution

Custom jobs might be broken if the customized AL-Go for GitHub workflow has been refactored and the referenced jobs have been renamed. Therefore, please make sure to review the changes in AL-Go workflows when running Update AL-Go System Files.

Custom job permissions

If any of your custom jobs require permissions, which exceeds the permissions already assigned in the workflow, then these permissions can be specified directly on the custom job.

Using custom template repositories

If you have have customizations you want to apply to multiple repositories, you might want to consider using a custom template. A custom template is really just an AL-Go repository (which can be customized), which you use as a template repository for your repositories. This way, you can control your scripts, jobs or steps in a central location, potentially for specific purposes.

Note

Custom templates can be public or private. If you are using a private custom template repository, AL-Go for GitHub will use the GhTokenWorkflow secret for downloading the template during Update AL-Go System Files and check for updates.

Tip

The recommended way to create a new repository based on your custom AL-Go template is to create a new repository based on AL-Go-PTE or AL-Go-AppSource, create a GhTokenWorkflow secret and then run the Update AL-Go System Files workflow with your custom template specified.

Note

If you use the custom template as a GitHub template for creating the repository, by clicking use this template in your custom template - then you need to re-specify the custom Template the first time you run Update AL-Go System Files as the repository will be a copy of the template repository and by default point to the template repository of the custom template as it's template repository.

Repositories based on your custom template will notify you that changes are available for your AL-Go System Files when you update the custom template only. You will not be notified when new versions of AL-Go for GitHub is released in every repository - only in the custom template repository.

Warning

You should ensure that your custom template repository is kept up-to-date with the latest changes in AL-Go for GitHub.

Tip

You can setup the Update AL-Go System Files workflow to run on a schedule to uptake new releases of AL-Go for GitHub regularly.

Using custom template files

When updating AL-Go for GitHub, only specific system files from the template repository are synced to your end repository by default. Files such as README.md, .gitignore, and other documentation or non-system files are not updated by AL-Go for GitHub. By default, AL-Go syncs workflow files in .github/workflows, PowerShell scripts in .github and .AL-Go, and configuration files required for AL-Go operations. When using custom template repositories, you may need to add additional files related to AL-Go for GitHub, such as script overrides, complementary workflows, or centrally managed files not part of the official AL-Go templates.

In order to instruct AL-Go which files to look for at the template repository, you need to define the customALGoFiles setting. The setting is an object that can contain two properties: filesToInclude and filesToExclude.

filesToInclude, as the name suggests, is an array of file configurations that will instruct AL-Go which files to include (create/update). Every item in the array may contain the following properties:

  • sourceFolder: A path to a folder, relative to the template, where to look for files. If not specified the root folder is implied. * characters are not supported. Example: src/scripts.
  • filter: A string to use for filtering in the specified source path. It can contain * and ? wildcards. Example: *.ps1 or fileToUpdate.ps1.
  • destinationFolder: A path to a folder, relative to repository that is being updated, where the files should be placed. If not specified, defaults to the same as the source file folder. Example: src/templateScripts.
  • perProject: A boolean that indicates whether the matched files should be propagated for all available AL-Go projects. In that case, destinationFolder is relative to the project folder. Example: .AL-Go/scripts.

Note

filesToInclude is used to define all the template files that will be used by AL-Go for GitHub. If a template file is not matched, it will be ignored. Please pay attention, when changing the file configurations: there might be template files that were previously propagated to your repositories. In case these files are no longer matched via filesToInclude, AL-Go for GitHub will ignore them and you might have to remove them manually.

filesToExclude is an array of file configurations that will instruct AL-Go which files to exclude (remove) from filesToInclude. Every item in the array may contain the following properties:

  • sourceFolder: A path to a folder, relative to the template, where to look for files. If not specified the root folder is implied. Example: src/scripts.
  • filter: A string to use for filtering in the specified source path. It can contain * and ? wildcards. Example: notRelevantScript.ps1 or *-internal.ps1

[!NOTE] filesToExclude is an array of file configurations already included in filesToInclude. These files are specifically marked to be excluded from the update process. This mechanism allows for fine-grained control over which files are propagated to the end repository and which should be explicitly removed, ensuring that unwanted files are not carried forward during updates.

The following table summarizes how AL-Go for GitHub manages file updates and exclusions when using custom template files. Say, there is a file (e.g. file.ps1) in the template repository.

File is present in end repo File is matched by filesToInclude File is matched by filesToExclude Result
Yes/No Yes No The file is updated/created in the end repo
Yes Yes Yes The file is removed from the end repo, as it's matched for exclusion
Yes No Yes The files is not removed as it was not matched as update
No Yes/No Yes The file is not created in the end repo, as it's matched for exclusion

Examples of using custom template files

Below are examples of how to use the filesToInclude and filesToExclude settings in your AL-Go configuration.

Example 1: Updating specific scripts for all projects

"customALGoFiles": {
  "filesToInclude": [
    {
      "sourceFolder": ".github/customScripts",
      "filter": "*.ps1",
      "destinationFolder": ".AL-Go/scripts",
      "perProject": true
    }
  ]
}

This configuration will copy all PowerShell scripts from .github/customScripts in your template repository to the .AL-Go/scripts folder in each project of your target repository.

Example 2: Excluding a specific script from updates

"customALGoFiles": {
  "filesToInclude": [
    {
      "sourceFolder": ".github/customScripts",
      "filter": "*.ps1",
      "destinationFolder": ".AL-Go/scripts",
      "perProject": true
    }
  ],
  "filesToExclude": [
    {
      "sourceFolder": ".github/customScripts",
      "filter": "DoNotPropagate.ps1"
    }
  ]
}

This will update all .ps1 scripts except DoNotPropagate.ps1, which will be excluded from the update process.

Example 3: Updating workflow files and excluding one

"customALGoFiles": {
  "filesToInclude": [],
  "filesToExclude": [
    {
      "sourceFolder": ".github/workflows",
      "filter": "experimental-workflow.yaml"
    }
  ]
}

All workflow YAML files will be updated except experimental-workflow.yaml, which will be removed from the target repository if present. Note that AL-Go for GitHub already syncs all workflow files under .github/workflows by default, so you don't need to specify filesToInclude; however, any files matched by filesToExclude will be excluded from this default sync.

Example 4: Multiple update and exclude rules

"customALGoFiles": {
  "filesToInclude": [
    {
      "sourceFolder": "shared/config",
      "filter": "*.json",
      "destinationFolder": "config"
    },
    {
      "sourceFolder": ".github/scripts",
      "filter": "*.ps1",
      "destinationFolder": ".github/scripts"
    }
  ],
  "filesToExclude": [
    {
      "sourceFolder": "shared/config",
      "filter": "legacy-config.json"
    }
  ]
}

This configuration updates all JSON files from shared/config and all PowerShell scripts from .github/scripts, but excludes legacy-config.json from being updated or created.

These examples demonstrate how you can fine-tune which files are propagated from your template repository and which are excluded, giving you granular control over your AL-Go customization process.

Forking AL-Go for GitHub and making your "own" public version

Using a fork of AL-Go for GitHub to have your "own" public version of AL-Go for GitHub gives you the maximum customization capabilities. It does however also come with the most work.

Note

When customizing AL-Go for GitHub using a fork, your customizations are public and will be visible to everyone. For more information, read this.

There are two ways of forking AL-Go for GitHub. You can fork the main AL-Go for GitHub repository and develop AL-Go for GitHub like we do in Microsoft, or you can fork the template repositories AL-Go PTE and/or AL-Go-AppSource.

While we don't recommend forking the template repositories, we realize that it is possible for simple changes to the templates. You can fork the template repositories and make the changes directly in your fork. Note that we do not accept any pull requests to the template repositories as they are deployed from the main AL-Go repository. We do not actually develop anything in the template repositories ourself. In the template repositories you will find a branch for every version of AL-Go we have shipped. The main branch is the latest version and the preview branch is the next version. You can customize the preview branch and/or the main branch and then use your fork as the template repository when running Update AL-Go System Files from your app repositories.

Tip

When forking the template repositories, you should include all branches in order to be able to use either the latest version of AL-Go or the preview version of AL-Go.

When forking the main AL-Go for GitHub repository, you are basically developing AL-Go in the same way as we are doing in Microsoft. Please follow the guidelines here on how to develop. This gives you maximum customization capabilities, but if your changes are not being contributed to AL-Go, then you will have to merge our changes all the time.

Caution

We strongly suggest that you keep your changes to a minimum and that you keep your fork up-to-date with the latest changes of AL-Go for GitHub at all time.

back