Demo Session: How to measure the effectiveness of GitHub Copilot

[KittyChiu]

Developers using GitHub Copilot know its value, but how do you show that value to get continued buy-in from the business? How do you know if you're measuring what matters most, and how do you use those metrics to craft a story for executives?

@mickeygousset and @KittyChiu will be presenting this in the coming Universe'24.
We would love to hear your thought here in this thread - so how do you measure your Copilot effectiveness?

Did you missed it? No worries, the recording is here.

UPDATED 2024-10-30

Here're the link to the community's github/copilot-survey-engine which you can automate and trigger at every PR closure. For longer version of the developers survey, here's the link.

Sample code snippet to `calculate_org_deployment_frequency.sh`

#!/bin/bash
# 1. Log in via `gh auth login`
# 2. chmod +x calculate_org_deployment_frequency.sh
# 3. Run ./calculate_org_deployment_frequency.sh

# Variables
ORG="<YOUR_ORG>"

# Fetch repositories
repos=$(gh repo list $ORG --limit 1000 --json name -q '.[].name')

calculate_frequency() {
  local repo=$1
  deployments=$(gh api repos/$ORG/$repo/deployments)
  current_time=$(date +%s)
  last_period=$(($current_time - 15768000)) # 6 months in seconds

  successful_deployments=0

  for deployment in $(echo $deployments | jq -r '.[] | select(.created_at | fromdateiso8601 > '$last_period') | .id'); do
    statuses=$(gh api repos/$ORG/$repo/deployments/$deployment/statuses | jq -r '.[].state')
    for status in $statuses; do
      if [ "$status" == "success" ]; then
        successful_deployments=$((successful_deployments + 1))
        break
      fi
    done
  done

  echo "$repo: Total successful deployments in the last 6 months: $successful_deployments"
}

# Iterate over repositories and calculate deployment frequency
for repo in $repos; do
  calculate_frequency $repo
done

Sample code snippet to `calculate_avg_pr_per_developer.sh`

#!/bin/bash
# 1. Log in via `gh auth login`
# 2. chmod +x calculate_avg_pr_per_developer.sh
# 3. Run ./calculate_avg_pr_per_developer.sh

# Variables
ORG="<YOUR_ORG>"

# Fetch repositories
repos=$(gh repo list $ORG --limit 1000 --json name -q '.[].name')

calculate_average_pr_merged_per_user() {
  local repo=$1
  current_time=$(date +%s)
  start_date=$(date -u -v -1m +"%Y-%m-%dT%H:%M:%SZ") # 1 month

  prs=$(gh pr list --repo $ORG/$repo --limit 1000 --state merged --search "closed:>$start_date sort:updated-desc" --json mergedAt,mergedBy)

  declare user_pr_count
  user_pr_count=()
  for pr in $(echo $prs | jq -r '.[] | select(.mergedAt) | .mergedBy.login'); do
    user_pr_count[$pr]=$((user_pr_count[$pr] + 1))
  done

  total_users=${#user_pr_count[@]}
  total_prs=0

  for count in "${user_pr_count[@]}"; do
    total_prs=$((total_prs + count))
  done

  mean_pr_merged_per_user=0

  if [ $total_users -gt 0 ]; then
    mean_pr_merged_per_user=$(echo "scale=2; $total_prs / $total_users" | bc)
  fi

  echo "$repo: Average PRs merged per user in the last month period: $mean_pr_merged_per_user"
}

# Iterate over repositories and calculate average PR merged per developer
for repo in $repos; do
  calculate_average_pr_merged_per_user $repo
done

Lastly, we could not communicate to you earlier before the official announcement, the GA release of the Copilot Metrics API is now available. There are new payload schema and API endpoint to this GA release, do read this changelog to find out more.

A sneak peek of noticeable differences between beta and GA payload schema,

UPDATED 2024-11-27

Here's the PowerBI AppSource link to fast track visualizing your Copilot activities. Check out this gist on the setup instruction. Kudos to @ayodejiayodele and his team on making this available to the community.

[KittyChiu]

☝🏻 Check out the above update on the PowerBI dashboard.

[josecleto]

@KittyChiu
Thanks for sharing. Does this metrics works for the Azure DevOps platform as well? If not, when will it be?

[KittyChiu]

@josecleto The metrics mentioned in the content are related to GitHub Copilot and GitHub APIs.