Tokens Replacement Service [Pilot] #447

ahmadabdalla · 2021-11-09T22:36:07Z

Change #295

First release for the Tokens Replacement Functionality

Supports 3 Token Types

Default Tokens [Only modifiable via Environment Variables and Actions]: Comes from environment variables. Defined in the Actions
Local Tokens: [Optional, Stored in Source Control]. Stored in the Settings JSON
Remote Tokens: [Optional, Stored in Key Vault]. Only activated when the PLATFORM_KEYVAULT secret is defined in GitHub.
3a. ParameterFileTokens: Normal key/value pairs.
3b. SecureParameterFileTokens: Are always set as 'Secure Strings' and only swapped at the last phase. Should be used for Secure String Parameters.

Please read the following documentations

NOTE: This is not a breaking change and is only applied transparently with no impact to existing resources. Only Resources that want to consume the tokens, can now do it by performing two things:

Set 2 Environment Variables in the Workflow. DEPLOYMENT_SP_ID and PLATFORM_KEYVAULT
Reference Tokens in the Parameter Files as needed (i.e. <<deploymentSpId>>, <<managementGroupId>>, <<platformKeyVault>>.

Testing

Pilot Modules

Dependency Pipeline Changes:

Added the two new environment variables (Platform Key Vault, and Deployment SP ID)
Added Platform Key Vault Task (Conditional to the Env. Variable)
Added Store Parameter File Tokens in Key Vault Task (Conditional to the Env. Variable)

There are failures in the current dependency pipeline due to a change in the child resources for event hub. Not related to this change

Other Notes

Global Modules test references test cases for tenant, management group, principal ID checks. Which have been developed but commented out for this release until we enforce the change on all pipelines.

Type of Change

New feature (non-breaking change which adds functionality)
This change requires a documentation update (Wiki)

Checklist

I'm sure there are no other open Pull Requests for the same update/change
My corresponding pipelines / checks run clean and green without any errors or warnings
My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation (readme)
I did format my code

Porting from Fork to Origin/Branch

github-actions · 2021-11-09T22:52:09Z

Unit Test Results

  1 files ±0   1 suites ±0 19s ⏱️ +4s
31 tests ±0 31 ✔️ ±0 0 💤 ±0 0 ❌ ±0
34 runs +1 31 ✔️ ±0 3 💤 +1 0 ❌ ±0

Results for commit 5c9e3e3. ± Comparison against base commit 949f42b.

This pull request removes 31 and adds 31 tests. Note that renamed tests count towards both.

/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] All apiVersion properties should be set to a static, hard-coded value
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] All parameters in parameters files exist in template file (deploy.json)
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] All required parameters in template file (deploy.json) should exist in parameters files
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] All resources that have a Location property should refer to the Location parameter &#x27;parameters(&#x27;Location&#x27;)&#x27;
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] CUA ID deployment should be present in the template
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] If delete lock is implemented, the template should have a lock parameter with the default value of [NotSpecified]
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] Output names should be camel-cased (no dashes or underscores and must start with lower-case letter)
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] Parameter files should not contain subscriptionId original value and but a token string
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] Parameter names should be camel-cased (no dashes or underscores and must start with lower-case letter)
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Insights/metricAlerts] Standard outputs should be provided (e.g. resourceName, resourceId, resouceGroupName)
…

/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] All apiVersion properties should be set to a static, hard-coded value
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] All parameters in parameters files exist in template file (deploy.json)
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] All required parameters in template file (deploy.json) should exist in parameters files
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] All resources that have a Location property should refer to the Location parameter &#x27;parameters(&#x27;Location&#x27;)&#x27;
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] CUA ID deployment should be present in the template
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] If delete lock is implemented, the template should have a lock parameter with the default value of [NotSpecified]
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] Output names should be camel-cased (no dashes or underscores and must start with lower-case letter)
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] Parameter files should not contain the subscriptionId guid
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] Parameter names should be camel-cased (no dashes or underscores and must start with lower-case letter)
/home/runner/work/ResourceModules/ResourceModules/arm/.global/global.module.tests.ps1 ‑ Deployment template tests.Deployment template tests.[Microsoft.Network/applicationSecurityGroups] Standard outputs should be provided (e.g. resourceName, resourceId, resouceGroupName)
…

♻️ This comment has been updated with latest results.

segraef

lgtm, we have to make sure to include the doco into the Wiki for 0.3/0.4 from an overall perspective. Chatted with @ahmadabdalla regarding that as well already.

.github/workflows/platform.dependencies.yml

* Update Dependency pipeline after child restructure (#448) * Implemented a first version to allow for custom removal scripts & retry (#431)

.github/workflows/platform.dependencies.yml

MariusStorhaug

Really good work. Some minor things here and there.

.github/actions/templates/validateModuleDeploy/action.yml

settings.json

utilities/tokensReplacement/helper/Get-RemoteCustomParameterFileTokens.ps1

docs/wiki/PipelinesDesign.md

docs/wiki/TestingUsage.md

docs/wiki/ParameterFileTokens.md

MariusStorhaug

Small thing.

.github/actions/templates/validateModuleDeploy/action.yml

utilities/tokensReplacement/helper/Get-RemoteCustomParameterFileTokens.ps1

arm/.global/global.module.tests.ps1

arm/Microsoft.Storage/storageAccounts/.parameters/parameters.json

docs/wiki/GettingStarted.md

docs/wiki/ParameterFileTokens.md

docs/wiki/PipelinesDesign.md

docs/wiki/TestingUsage.md

utilities/tokensReplacement/Convert-TokensInParameterFile.ps1

Co-authored-by: Marius <[email protected]>

…rceModules into users/ahmad/tokensV3

MariusStorhaug

All comments i had have been sorted. For me this is good to go. Approved

docs/wiki/GettingStarted.md

docs/wiki/TestingUsage.md

docs/wiki/GettingStarted.md

docs/wiki/TestingUsage.md

docs/wiki/ParameterFileTokens.md

eriqua

🆗

ahmadabdalla added 4 commits November 10, 2021 08:24

Tokens v3 (#446)

1ade1c2

Porting from Fork to Origin/Branch

updated global tests and typos in test usage

ba8c461

Token for Principal ID

8be0c83

Applied Pilot Resources

59b5231

ahmadabdalla self-assigned this Nov 9, 2021

ahmadabdalla added this to the v 0.3 milestone Nov 9, 2021

ahmadabdalla added [cat] pipelines category: pipelines [prio] high importance of the issue: high priority documentation Improvements or additions to documentation labels Nov 9, 2021

commented global tests for mgmt group id, tenant

d7de5af

segraef reviewed Nov 10, 2021

View reviewed changes

MariusStorhaug self-requested a review November 10, 2021 08:04

eriqua reviewed Nov 10, 2021

View reviewed changes

.github/workflows/platform.dependencies.yml Outdated Show resolved Hide resolved

ahmadabdalla added 4 commits November 11, 2021 09:12

Syncing latest changes from fork to tokens branch (#458)

cc3e64c

* Update Dependency pipeline after child restructure (#448) * Implemented a first version to allow for custom removal scripts & retry (#431)

syncing changes based on latest main config

5a3c240

Merge remote-tracking branch 'origin/main' into users/ahmad/tokensV3

6e39045

Merge branch 'main' into users/ahmad/tokensV3

7b34fcf

ahmadabdalla commented Nov 11, 2021

View reviewed changes

.github/workflows/platform.dependencies.yml Outdated Show resolved Hide resolved

MariusStorhaug suggested changes Nov 11, 2021

View reviewed changes

ahmadabdalla added 5 commits November 12, 2021 15:39

Feedback - Part 1

14fdc0c

changed link to reference public

1ae41ae

updated dependency pipeline order

4e73798

Merge remote-tracking branch 'origin/main' into users/ahmad/tokensV3

951469c

Simplified Variable Names

b328b09

MariusStorhaug suggested changes Nov 14, 2021

View reviewed changes

.github/actions/templates/validateModuleDeploy/action.yml Show resolved Hide resolved

utilities/tokensReplacement/helper/Get-RemoteCustomParameterFileTokens.ps1 Outdated Show resolved Hide resolved

ahmadabdalla requested a review from MariusStorhaug November 14, 2021 19:23

MariusStorhaug suggested changes Nov 14, 2021

View reviewed changes

ahmadabdalla and others added 2 commits November 15, 2021 08:05

Apply suggestions from code review

2dcf698

Co-authored-by: Marius <[email protected]>

feedback

7244992

ahmadabdalla added 2 commits November 15, 2021 08:06

Merge branch 'users/ahmad/tokensV3' of https://github.com/Azure/Resou…

783334a

…rceModules into users/ahmad/tokensV3

Updated based on feedback

5c9e3e3

ahmadabdalla requested a review from MariusStorhaug November 14, 2021 22:13

MariusStorhaug previously approved these changes Nov 15, 2021

View reviewed changes