Skill v1.0.0
currentTrusted Publisher100/100version: "1.0.0" name: ado-assessment description: 'Review an Azure DevOps organization or project for adherence to Azure DevOps best practices. Use when: running an ADO Quick Review, evaluating org or project configuration, checking adoption of recommended settings, identifying improvement opportunities, ADO best practices, org review, project review.'
Azure DevOps Quick Review
Review an Azure DevOps organization or project for adherence to Azure DevOps best practices by collecting settings via Azure CLI / REST API and evaluating them against documented Microsoft recommendations.
This skill does NOT call AzSK.ADO. It brings best-practice knowledge derived from those controls and uses native Azure CLI and REST API calls to collect and evaluate settings.
Workflow
Phase 1 — Collect Settings
Gather the current configuration of an ADO organization or project using az devops CLI commands and REST API calls. The reference files below contain the exact commands and API endpoints for each resource type.
Phase 2 — Assess Against Best Practices
Compare collected settings against the best-practice controls documented in the reference files. Each control has a severity (High/Medium/Low), a rationale, and remediation guidance. Generate a report showing which best practices have been adopted and which remain as improvement opportunities.
Prerequisites
# Azure CLI 2.81.0+az version# Azure DevOps extensionaz extension add --name azure-devops# Loginaz login# Set defaultsaz devops configure --defaults organization=https://dev.azure.com/<YourOrg> project=<YourProject>
Authentication
# Interactive loginaz login# PAT-based login (for automation)$env:AZURE_DEVOPS_EXT_PAT = '<your-pat>'az devops login# Get REST API bearer token$token = az account get-access-token --resource "499b84ac-1321-427f-aa17-267ca6975798" --query accessToken -o tsv
Resource Types Covered
| Resource | Controls | Scope | |
|---|---|---|---|
| Organization | 35+ controls | Auth, users, extensions, pipelines, feeds, audit | |
| Project | 25+ controls | Visibility, admins, pipelines, permissions, credential scanning | |
| Build Pipelines | 20+ controls | Secrets, permissions, forks, branches, task groups | |
| Release Pipelines | 12+ controls | Secrets, approvals, permissions, task groups | |
| Service Connections | 15+ controls | Auth, scope, access, permissions, branches | |
| Agent Pools | 10+ controls | Permissions, auto-provision, patching, secrets | |
| Repos / Feeds / Secure Files / Environments | 20+ controls | Access, permissions, branches, approvals | |
| Variable Groups | 10+ controls | Secrets, permissions, access, branches | |
| Users / PATs | 6+ controls | PAT scope, expiry, alternate credentials |
Each resource is evaluated against documented Microsoft best practices.
Reference Files
Read the relevant reference file based on the user's assessment scope.
| File | Keywords | Covers | |
|---|---|---|---|
| references/org-settings.md | organization, org, AAD, external users, public projects, extensions, audit, OAuth, SSH, feeds, conditional access, admin | Organization-level settings — collection commands and best-practice controls | |
| references/project-settings.md | project, visibility, admin, credential scanner, permissions, pipeline scoping, inactive | Project-level settings — collection commands and best-practice controls | |
| references/build-controls.md | build, pipeline, CI, secrets, fork, task group, variable, branch, YAML | Build pipeline best-practice controls — collection commands and assessments | |
| references/release-controls.md | release, deployment, approval, CD, production, stage | Release pipeline best-practice controls — collection commands and assessments | |
| references/service-connection-controls.md | service connection, endpoint, SPN, certificate, subscription, ARM | Service connection best-practice controls — collection commands and assessments | |
| references/agent-pool-controls.md | agent, pool, self-hosted, auto-provision, auto-update | Agent pool best-practice controls — collection commands and assessments | |
| references/repo-feed-environment-controls.md | repository, repo, feed, artifact, secure file, environment, branch protection | Repos, feeds, secure files, and environment best-practice controls | |
| references/variable-group-controls.md | variable group, secret, key vault, linked | Variable group best-practice controls | |
| references/user-pat-controls.md | user, PAT, personal access token, alternate credentials, inactive | User and PAT hygiene best-practice controls | |
| references/errors.md | error, 401, 403, permission denied, troubleshoot | Common errors and troubleshooting |
Assessment Output Format
When reporting results, use this format for each control:
| Status | Severity | Control | Finding ||--------|----------|---------|---------|| PASS | High | AAD authentication enabled | Organization uses AAD-backed auth || FAIL | High | External user access | 3 external users found with access || FAIL | Medium | Audit streaming | Audit streaming is not configured |
Summarize at the end with counts: X PASS | Y FAIL | Z NOT CHECKED
Saving Results
After the assessment completes, save the results as Markdown files in the workspace root. Create separate files for the organization and each project:
| File | Contents | |
|---|---|---|
{org-name}-org-assessment.md | Organization-level controls (auth, users, extensions, pipelines, feeds, audit, agent pools, PATs) | |
{org-name}-{project-name}-assessment.md | Project-level controls (visibility, admins, pipeline settings, permissions) plus all project-scoped resources: build pipelines, release pipelines, service connections, repos, feeds, environments, variable groups, secure files |
File naming
- Use the organization short name and project name, lowercased and with spaces replaced by hyphens.
- Examples for org
pbi-demowith projectseShop Web DemoandCL-DEMO: pbi-demo-org-assessment.mdpbi-demo-eshop-web-demo-assessment.mdpbi-demo-cl-demo-assessment.md
File structure
Each file must include:
- Header — assessment date, organization/project name, assessor
- Control results table — the standard
Status | Severity | Control | Findingtable - Summary counts —
X PASS | Y FAIL | Z NOT CHECKED - Critical findings — list of FAIL items sorted by severity (High first) with remediation steps
When to write
Write the files at the end of Phase 2 (after all controls are assessed and results are shown to the user). Always create the files — do not ask for confirmation.