Skip to content

Commit b663b86

Browse files
aj-ennsaj-enns
committed
Initial public release of adoqr (Azure DevOps Quick Review)
0 parents  commit b663b86

42 files changed

Lines changed: 13302 additions & 0 deletions

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.github/copilot-instructions.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
## Versioning
2+
3+
This repository uses Semantic Versioning.
4+
5+
For deployable or user-visible changes:
6+
- Update the root `VERSION` file using `X.Y.Z` format only.
7+
- Update `CHANGELOG.md` under `## [Unreleased]` using Keep a Changelog categories: `Added`, `Changed`, `Deprecated`, `Removed`, `Fixed`, `Security`.
8+
- Choose the bump level as follows:
9+
- `MAJOR` for breaking changes or required migration.
10+
- `MINOR` for backward-compatible features or enhancements.
11+
- `PATCH` for bug fixes, documentation updates, refactors, and non-breaking maintenance.
12+
- If the correct bump level is ambiguous, ask before changing `VERSION`.
13+
- Do not describe or rely on versioning behaviors that are not implemented in this repository.
14+
15+
When making version-related edits, keep changelog entries user-facing rather than implementation-focused.

‎.github/skills/ado-assessment/SKILL.md‎

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
---
2+
name: ado-assessment
3+
description: 'Review an Azure DevOps organization or project for adherence to Azure DevOps best practices. Use when: running an ADO Quick Review, evaluating org or project configuration, checking adoption of recommended settings, identifying improvement opportunities, ADO best practices, org review, project review.'
4+
---
5+
6+
# Azure DevOps Quick Review
7+
8+
Review an Azure DevOps organization or project for adherence to **Azure DevOps best practices** by collecting settings via Azure CLI / REST API and evaluating them against documented Microsoft recommendations.
9+
10+
**This skill does NOT call AzSK.ADO.** It brings best-practice knowledge derived from those controls and uses native Azure CLI and REST API calls to collect and evaluate settings.
11+
12+
## Workflow
13+
14+
### Phase 1 — Collect Settings
15+
16+
Gather the current configuration of an ADO organization or project using `az devops` CLI commands and REST API calls. The reference files below contain the exact commands and API endpoints for each resource type.
17+
18+
### Phase 2 — Assess Against Best Practices
19+
20+
Compare collected settings against the best-practice controls documented in the reference files. Each control has a severity (High/Medium/Low), a rationale, and remediation guidance. Generate a report showing which best practices have been adopted and which remain as improvement opportunities.
21+
22+
## Prerequisites
23+
24+
```powershell
25+
# Azure CLI 2.81.0+
26+
az version
27+
28+
# Azure DevOps extension
29+
az extension add --name azure-devops
30+
31+
# Login
32+
az login
33+
34+
# Set defaults
35+
az devops configure --defaults organization=https://dev.azure.com/<YourOrg> project=<YourProject>
36+
```
37+
38+
## Authentication
39+
40+
```powershell
41+
# Interactive login
42+
az login
43+
44+
# PAT-based login (for automation)
45+
$env:AZURE_DEVOPS_EXT_PAT = '<your-pat>'
46+
az devops login
47+
48+
# Get REST API bearer token
49+
$token = az account get-access-token --resource "499b84ac-1321-427f-aa17-267ca6975798" --query accessToken -o tsv
50+
```
51+
52+
## Resource Types Covered
53+
54+
| Resource | Controls | Scope |
55+
|---|---|---|
56+
| Organization | 35+ controls | Auth, users, extensions, pipelines, feeds, audit |
57+
| Project | 25+ controls | Visibility, admins, pipelines, permissions, credential scanning |
58+
| Build Pipelines | 20+ controls | Secrets, permissions, forks, branches, task groups |
59+
| Release Pipelines | 12+ controls | Secrets, approvals, permissions, task groups |
60+
| Service Connections | 15+ controls | Auth, scope, access, permissions, branches |
61+
| Agent Pools | 10+ controls | Permissions, auto-provision, patching, secrets |
62+
| Repos / Feeds / Secure Files / Environments | 20+ controls | Access, permissions, branches, approvals |
63+
| Variable Groups | 10+ controls | Secrets, permissions, access, branches |
64+
| Users / PATs | 6+ controls | PAT scope, expiry, alternate credentials |
65+
66+
Each resource is evaluated against documented Microsoft best practices.
67+
68+
## Reference Files
69+
70+
Read the relevant reference file based on the user's assessment scope.
71+
72+
| File | Keywords | Covers |
73+
|---|---|---|
74+
| [references/org-settings.md](./references/org-settings.md) | organization, org, AAD, external users, public projects, extensions, audit, OAuth, SSH, feeds, conditional access, admin | Organization-level settings — collection commands and best-practice controls |
75+
| [references/project-settings.md](./references/project-settings.md) | project, visibility, admin, credential scanner, permissions, pipeline scoping, inactive | Project-level settings — collection commands and best-practice controls |
76+
| [references/build-controls.md](./references/build-controls.md) | build, pipeline, CI, secrets, fork, task group, variable, branch, YAML | Build pipeline best-practice controls — collection commands and assessments |
77+
| [references/release-controls.md](./references/release-controls.md) | release, deployment, approval, CD, production, stage | Release pipeline best-practice controls — collection commands and assessments |
78+
| [references/service-connection-controls.md](./references/service-connection-controls.md) | service connection, endpoint, SPN, certificate, subscription, ARM | Service connection best-practice controls — collection commands and assessments |
79+
| [references/agent-pool-controls.md](./references/agent-pool-controls.md) | agent, pool, self-hosted, auto-provision, auto-update | Agent pool best-practice controls — collection commands and assessments |
80+
| [references/repo-feed-environment-controls.md](./references/repo-feed-environment-controls.md) | repository, repo, feed, artifact, secure file, environment, branch protection | Repos, feeds, secure files, and environment best-practice controls |
81+
| [references/variable-group-controls.md](./references/variable-group-controls.md) | variable group, secret, key vault, linked | Variable group best-practice controls |
82+
| [references/user-pat-controls.md](./references/user-pat-controls.md) | user, PAT, personal access token, alternate credentials, inactive | User and PAT hygiene best-practice controls |
83+
| [references/errors.md](./references/errors.md) | error, 401, 403, permission denied, troubleshoot | Common errors and troubleshooting |
84+
85+
## Assessment Output Format
86+
87+
When reporting results, use this format for each control:
88+
89+
```
90+
| Status | Severity | Control | Finding |
91+
|--------|----------|---------|---------|
92+
| PASS | High | AAD authentication enabled | Organization uses AAD-backed auth |
93+
| FAIL | High | External user access | 3 external users found with access |
94+
| FAIL | Medium | Audit streaming | Audit streaming is not configured |
95+
```
96+
97+
Summarize at the end with counts: `X PASS | Y FAIL | Z NOT CHECKED`
98+
99+
## Saving Results
100+
101+
After the assessment completes, save the results as Markdown files **in the workspace root**. Create separate files for the organization and each project:
102+
103+
| File | Contents |
104+
|---|---|
105+
| `{org-name}-org-assessment.md` | Organization-level controls (auth, users, extensions, pipelines, feeds, audit, agent pools, PATs) |
106+
| `{org-name}-{project-name}-assessment.md` | Project-level controls (visibility, admins, pipeline settings, permissions) **plus** all project-scoped resources: build pipelines, release pipelines, service connections, repos, feeds, environments, variable groups, secure files |
107+
108+
### File naming
109+
110+
- Use the organization short name and project name, lowercased and with spaces replaced by hyphens.
111+
- Examples for org `pbi-demo` with projects `eShop Web Demo` and `CL-DEMO`:
112+
- `pbi-demo-org-assessment.md`
113+
- `pbi-demo-eshop-web-demo-assessment.md`
114+
- `pbi-demo-cl-demo-assessment.md`
115+
116+
### File structure
117+
118+
Each file must include:
119+
120+
1. **Header** — assessment date, organization/project name, assessor
121+
2. **Control results table** — the standard `Status | Severity | Control | Finding` table
122+
3. **Summary counts** — `X PASS | Y FAIL | Z NOT CHECKED`
123+
4. **Critical findings** — list of FAIL items sorted by severity (High first) with remediation steps
124+
125+
### When to write
126+
127+
Write the files at the end of Phase 2 (after all controls are assessed and results are shown to the user). Always create the files — do not ask for confirmation.

‎.github/skills/ado-assessment/references/agent-pool-controls.md‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
# Agent Pool Best-Practice Controls
2+
3+
Collect and assess agent pool settings against Azure DevOps best practices.
4+
5+
## How to Collect Agent Pool Settings
6+
7+
```powershell
8+
$org = "https://dev.azure.com/{organization}"
9+
$project = "{project}"
10+
$header = @{ Authorization = "Bearer $token" }
11+
12+
# List all agent pools (org level)
13+
Invoke-RestMethod -Uri "$org/_apis/distributedtask/pools?api-version=7.1" -Headers $header
14+
15+
# Get a specific agent pool
16+
Invoke-RestMethod -Uri "$org/_apis/distributedtask/pools/{poolId}?api-version=7.1" -Headers $header
17+
18+
# List agents in a pool
19+
Invoke-RestMethod -Uri "$org/_apis/distributedtask/pools/{poolId}/agents?api-version=7.1" -Headers $header
20+
21+
# Get agent details (includes capabilities, status, last completed request)
22+
Invoke-RestMethod -Uri "$org/_apis/distributedtask/pools/{poolId}/agents/{agentId}?includeCapabilities=true&includeLastCompletedRequest=true&api-version=7.1" -Headers $header
23+
24+
# Check agent pool permissions
25+
Invoke-RestMethod -Uri "$org/_apis/distributedtask/pools/{poolId}/roles?api-version=7.1" -Headers $header
26+
27+
# Check pipeline permissions for agent pool
28+
Invoke-RestMethod -Uri "$org/$project/_apis/pipelines/pipelinePermissions/queue/{queueId}?api-version=7.1-preview.1" -Headers $header
29+
30+
# List project-level queues (agent pools visible to a project)
31+
Invoke-RestMethod -Uri "$org/$project/_apis/distributedtask/queues?api-version=7.1-preview.1" -Headers $header
32+
```
33+
34+
---
35+
36+
## Best-Practice Controls
37+
38+
### AP-01: Security Patches on Self-Hosted VMs
39+
| Field | Value |
40+
|-------|-------|
41+
| **Severity** | High |
42+
| **Automated** | No (manual review) |
43+
| **Check** | Non-hosted agent virtual machines must have all required security patches installed. |
44+
| **Rationale** | Unpatched VMs are easy targets for compromise from various malware/trojan attacks that exploit known vulnerabilities in operating systems and related software. |
45+
| **Collection** | Identify self-hosted agent pools (`isHosted: false`). Check agent OS version and patch level via agent capabilities. |
46+
| **Remediation** | Implement automated patching for self-hosted agent VMs. Use Azure Update Manager or WSUS. |
47+
48+
### AP-02: Hardened OS Image
49+
| Field | Value |
50+
|-------|-------|
51+
| **Severity** | Medium |
52+
| **Automated** | No (manual review) |
53+
| **Check** | Use a security hardened, locked down OS image for self-hosted VMs in agent pool. |
54+
| **Rationale** | The agent machine is serving as a 'gateway' into the corporate environment. Using a locked-down, secure baseline configuration ensures this machine is not leveraged as an entry point. |
55+
| **Collection** | Review self-hosted agent OS images for CIS benchmark compliance. |
56+
| **Remediation** | Use CIS-hardened images for agent VMs. Remove unnecessary software and services. |
57+
58+
### AP-03: Inherited Permissions Disabled
59+
| Field | Value |
60+
|-------|-------|
61+
| **Severity** | High |
62+
| **Automated** | Yes |
63+
| **Check** | Do not allow inherited permission on agent pool. |
64+
| **Rationale** | Disabling inherited permissions lets you finely control access at the agent level. This ensures the principle of least privilege and provides access only to persons that require it. |
65+
| **Collection** | Check agent pool permissions for inheritance settings. |
66+
| **Remediation** | Disable inheritance on individual agent pools and set explicit permissions. |
67+
68+
### AP-04: Auto-Provisioning Disabled
69+
| Field | Value |
70+
|-------|-------|
71+
| **Severity** | High |
72+
| **Automated** | Yes |
73+
| **Check** | Do not enable auto-provisioning for agent pools. |
74+
| **Rationale** | By enabling auto-provisioning, the organization agent pool is imported in all new team projects and is accessible immediately. A vulnerability in components used by one project can be leveraged to attack other projects. |
75+
| **Collection** | `Invoke-RestMethod -Uri "$org/_apis/distributedtask/pools/{poolId}?api-version=7.1"` — check `autoProvision` property. |
76+
| **Remediation** | Set `autoProvision: false` on agent pools. Manually grant access per-project. |
77+
78+
### AP-05: Not Accessible to All YAML Pipelines
79+
| Field | Value |
80+
|-------|-------|
81+
| **Severity** | High |
82+
| **Automated** | Yes |
83+
| **Check** | Do not make agent pool accessible to all (YAML) pipelines in the project. |
84+
| **Rationale** | To support security of pipeline operations, agent pools must not be granted access to all YAML pipelines. A vulnerability in one pipeline can be leveraged to attack other pipelines with access to critical resources. |
85+
| **Collection** | REST: `_apis/pipelines/pipelinePermissions/queue/{queueId}` — check if `allPipelines.authorized` is `true`. |
86+
| **Remediation** | Disable "Grant access permission to all pipelines". Add individual pipeline authorizations. |
87+
88+
### AP-06: Inactive Agent Pools
89+
| Field | Value |
90+
|-------|-------|
91+
| **Severity** | Medium |
92+
| **Automated** | Yes |
93+
| **Check** | Inactive agent pools must be removed if no more required. |
94+
| **Rationale** | Agent pools may contain potentially sensitive information (such as code, secrets, pre-release information and logs) from previously run pipelines. Each inactive agent pool can increase exposure. |
95+
| **Collection** | Check agent last completed request date. Flag pools where all agents have no activity in 90+ days. |
96+
| **Remediation** | Delete inactive agent pools after confirming they are no longer needed. |
97+
98+
### AP-07: Auto-Update Enabled
99+
| Field | Value |
100+
|-------|-------|
101+
| **Severity** | High |
102+
| **Automated** | Yes |
103+
| **Check** | Enable auto-update of agents in the pool. |
104+
| **Rationale** | Unpatched agents are easy targets for compromise. Being on the latest OS version significantly reduces risks from security design issues and bugs present in older versions. |
105+
| **Collection** | `Invoke-RestMethod -Uri "$org/_apis/distributedtask/pools/{poolId}?api-version=7.1"` — check `autoUpdate` property. Also check individual agent `version` against latest available. |
106+
| **Remediation** | Enable auto-update on agent pools. Verify agents are running the latest version. |
107+
108+
### AP-08: No Plain Text Secrets in Capabilities
109+
| Field | Value |
110+
|-------|-------|
111+
| **Severity** | High |
112+
| **Automated** | Yes |
113+
| **Check** | Secrets and keys must not be stored as plain text in agent capabilities. |
114+
| **Rationale** | Keeping secrets as plain text in agent capabilities can expose credentials. Any user who can deploy a pipeline to run on such agents can access these secrets and compromise the security of resources involving the secrets. |
115+
| **Collection** | REST: `_apis/distributedtask/pools/{poolId}/agents/{agentId}?includeCapabilities=true` — inspect `userCapabilities` for values containing keywords like "password", "secret", "key", "token", "connectionstring". |
116+
| **Remediation** | Remove sensitive data from agent user capabilities. Use pipeline variables or Key Vault instead. |
117+
118+
### AP-09: Broader Group Excessive Permissions
119+
| Field | Value |
120+
|-------|-------|
121+
| **Severity** | High |
122+
| **Automated** | Yes |
123+
| **Check** | Broader groups should not have excessive permissions on agent pool. |
124+
| **Rationale** | If broader groups (e.g., Contributors) have excessive permissions (Admin/User) on an agent pool, integrity can be compromised by a malicious user. Removing unnecessary access minimizes exposure in case of compromise. |
125+
| **Collection** | Check agent pool role assignments for broad groups (Contributors, Project Valid Users). |
126+
| **Remediation** | Remove excessive role assignments for broad groups. Grant Reader role only where needed. |

0 commit comments

Comments
 (0)