Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

🐛 save file names during gitfile handler setup to avoid data race #4522

Merged
merged 1 commit into from
Feb 16, 2025

Conversation

spencerschrock
Copy link
Member

What kind of change does this PR introduce?

bug fix from #4474

What is the current behavior?

Each call to ListFiles would walk the HEAD commit's tree, but go-git is not thread-safe.
So the old method would lead to race conditions when multiple checks called ListFiles at the same time. This slipped by my development but was reproducible when built with -race:

go build -race && ./scorecard --repo ossf-tests/scorecard-check-license-e2e --file-mode git
...
fatal error: concurrent map read and map write
...
==================
WARNING: DATA RACE
Write at 0x00c000222f30 by goroutine 67:
  runtime.mapassign()
      ~/sdk/go1.24.0/src/internal/runtime/maps/runtime_swiss.go:191 +0x0
< omitted > 
  github.com/go-git/go-git/v5.(*Repository).Head()
      ~/go/pkg/mod/github.com/go-git/go-git/[email protected]/repository.go:1497 +0x1ee
  github.com/ossf/scorecard/v5/internal/gitfile.(*Handler).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/internal/gitfile/gitfile.go:103 +0xc6
  github.com/ossf/scorecard/v5/clients/githubrepo.(*Client).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/clients/githubrepo/client.go:187 +0x87
  github.com/ossf/scorecard/v5/checks/fileparser.OnAllFilesDo()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/fileparser/listing.go:160 +0x6b
  github.com/ossf/scorecard/v5/checks/raw.DependencyUpdateTool()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/raw/dependency_update_tool.go:35 +0x110
  github.com/ossf/scorecard/v5/checks.DependencyUpdateTool()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/dependency_update_tool.go:42 +0x7a
  github.com/ossf/scorecard/v5/checker.(*Runner).Run()
      ~/go/src/github.com/spencerschrock/scorecard.git/checker/check_runner.go:118 +0xe48
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks.func1()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:67 +0x257

Previous read at 0x00c000222f30 by goroutine 58:
  runtime.mapaccess1()
      ~/sdk/go1.24.0/src/internal/runtime/maps/runtime_swiss.go:43 +0x0
< omitted > 
  github.com/go-git/go-git/v5.(*Repository).Head()
      ~/go/pkg/mod/github.com/go-git/go-git/[email protected]/repository.go:1497 +0x1ee
  github.com/ossf/scorecard/v5/internal/gitfile.(*Handler).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/internal/gitfile/gitfile.go:103 +0xc6
  github.com/ossf/scorecard/v5/clients/githubrepo.(*Client).ListFiles()
      ~/go/src/github.com/spencerschrock/scorecard.git/clients/githubrepo/client.go:187 +0x87
  github.com/ossf/scorecard/v5/checks/raw/github.Packaging()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/raw/github/packaging.go:34 +0x69
  github.com/ossf/scorecard/v5/checks.Packaging()
      ~/go/src/github.com/spencerschrock/scorecard.git/checks/packaging.go:64 +0x2aa
  github.com/ossf/scorecard/v5/checker.(*Runner).Run()
      ~/go/src/github.com/spencerschrock/scorecard.git/checker/check_runner.go:118 +0xe48
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks.func1()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:67 +0x257

Goroutine 67 (running) created at:
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:59 +0xda
  github.com/ossf/scorecard/v5/pkg/scorecard.runScorecard.gowrap2()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:179 +0x8f

Goroutine 58 (running) created at:
  github.com/ossf/scorecard/v5/pkg/scorecard.runEnabledChecks()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:59 +0xda
  github.com/ossf/scorecard/v5/pkg/scorecard.runScorecard.gowrap2()
      ~/go/src/github.com/spencerschrock/scorecard.git/pkg/scorecard/scorecard.go:179 +0x8f
==================

What is the new behavior (if this is a feature change)?**

The data race no longer occurs because the files are enumerated from the git tree inside a sync.Once, so only one thread.
These files are put in a slice where they can concurrently be read by multiple goroutines later.

go build -race && ./scorecard --repo ossf-tests/scorecard-check-license-e2e --file-mode git
<normal output>
  • Tests for the changes have been added (for bug fixes/features)

Which issue(s) this PR fixes

NONE

Special notes for your reviewer

Does this PR introduce a user-facing change?

For user-facing changes, please add a concise, human-readable release note to
the release-note

(In particular, describe what changes users might need to make in their
application as a result of this pull request.)

Fixed a data race when analyzing repositories with `--file-mode git`

go-git is not thread-safe, so the old method would lead to race
conditions when multiple checks called ListFiles at the same time. This
was reproducible when built with `-race`, and now no longer occurs
because it's done inside a sync.Once.

Signed-off-by: Spencer Schrock <[email protected]>
@spencerschrock spencerschrock requested a review from a team as a code owner February 13, 2025 07:03
@spencerschrock spencerschrock requested review from justaugustus and raghavkaul and removed request for a team February 13, 2025 07:03
Copy link

codecov bot commented Feb 13, 2025

Codecov Report

Attention: Patch coverage is 66.66667% with 7 lines in your changes missing coverage. Please review.

Project coverage is 68.19%. Comparing base (353ed60) to head (e4146dc).
Report is 114 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #4522      +/-   ##
==========================================
+ Coverage   66.80%   68.19%   +1.39%     
==========================================
  Files         230      247      +17     
  Lines       16602    18635    +2033     
==========================================
+ Hits        11091    12709    +1618     
- Misses       4808     5083     +275     
- Partials      703      843     +140     

@spencerschrock spencerschrock changed the title 🐛 cache files during gitfile handler setup to avoid data race 🐛 save files during gitfile handler setup to avoid data race Feb 13, 2025
@spencerschrock spencerschrock changed the title 🐛 save files during gitfile handler setup to avoid data race 🐛 save file names during gitfile handler setup to avoid data race Feb 13, 2025
Copy link
Member

@justaugustus justaugustus left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @spencerschrock!

@justaugustus justaugustus merged commit cd152cb into ossf:main Feb 16, 2025
43 checks passed
@spencerschrock spencerschrock deleted the git-list branch February 17, 2025 16:32
balteravishay pushed a commit to PrinceAsiedu/scorecard that referenced this pull request Feb 17, 2025
go-git is not thread-safe, so the old method would lead to race
conditions when multiple checks called ListFiles at the same time. This
was reproducible when built with `-race`, and now no longer occurs
because it's done inside a sync.Once.

Signed-off-by: Spencer Schrock <[email protected]>
Signed-off-by: balteravishay <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Done
Development

Successfully merging this pull request may close these issues.

2 participants