-
-
Notifications
You must be signed in to change notification settings - Fork 208
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Use static versions for interdependencies #1623
Merged
Merged
Changes from all commits
Commits
Show all changes
5 commits
Select commit
Hold shift + click to select a range
8dc7199
Use static versions for interdependencies
mcmire c8f94e6
Merge branch 'main' into make-yarn-link-work-again
mcmire 69ed88f
Merge branch 'main' into make-yarn-link-work-again
mcmire 811e5ea
Rewrite section on testing changes to include local
mcmire cf79ca4
Merge branch 'main' into make-yarn-link-work-again
mcmire File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -51,30 +51,72 @@ When submitting a pull request for this repo, take some a bit of extra time to f | |
- What are the anticipated effects to whichever platform might want to make use of these changes? | ||
- If there are breaking changes to the API, what do consumers need to do in order to adapt to those changes upon upgrading to them? | ||
|
||
## Testing changes to packages against another project | ||
## Testing changes to packages in another project | ||
|
||
If you have a project that depends on packages from this monorepo, you may wish to load changes to those packages into the project and test them locally or in CI before releasing them. To solve that problem, this repository provides a mechanism to publish "preview" versions of packages. These versions can then be used in the project like any other versions. | ||
If you have a project that depends on a package in this monorepo, you may want to load those changes into the project without having to create a whole new monorepo release. How you do this depends on your use case. | ||
|
||
### Publishing preview builds as a MetaMask contributor | ||
### Testing changes to packages locally | ||
|
||
If you're a member of the MetaMask organization, you can create preview builds for a pull request by posting a comment with the text `@metamaskbot publish-preview`. This starts the `publish-preview` GitHub action; after a few minutes, it should complete and you will see a new comment that lists the newly published packages. You can then [use these versions in your project](#using-preview-builds). | ||
If you're developing your project locally and want to test changes to a package, you can follow these steps: | ||
|
||
Note two things about each package: | ||
1. First, you must build the monorepo. It's recommend to run `yarn build:watch` so that changes to the package you want to change are reflected in your project automatically. | ||
2. Next, you need to connect the package to your project by overriding the resolution logic in your package manager to replace the published version of the package with the local version. | ||
|
||
- The name is scoped to `@metamask-previews` instead of `@metamask`. | ||
- The ID of the last commit in the branch is appended to the version, e.g. `1.2.3-e2df9b4` instead of `1.2.3`. | ||
1. Open `package.json` in the project and locate the dependency entry for the package. | ||
2. Locate the section responsible for resolution overrides (or create it if it doesn't exist). If you're using Yarn, this is `resolutions`; if you're using NPM or any other package manager, this is `overrides`. | ||
3. Add a line to this section that mirrors the dependency entry on the left-hand side and points to the local path on the right-hand side: | ||
|
||
If you make more changes and wish to publish another set of preview builds, post another comment on the pull request with `@metamaskbot publish-preview`. | ||
``` | ||
"@metamask/<PACKAGE_NAME>@<PUBLISHED_VERSION_RANGE>": "file:<PATH_TO_CORE>/packages/<PACKAGE_NAME>" | ||
``` | ||
### Publishing preview builds as an independent contributor | ||
> **Example:** | ||
> | ||
> - If you're a member of MetaMask, your project uses Yarn, `@metamask/controller-utils` is listed in dependencies at `^1.1.4`, and your clone of the `core` repo is at the same level as your project, add the following to `resolutions`: | ||
> | ||
> ``` | ||
> "@metamask/controller-utils@^1.1.4": "file:../core/packages/controller-utils" | ||
> ``` | ||
> | ||
> - If you are an individual contributor, your project uses NPM, `@metamask/assets-controllers` is listed in dependencies at `^3.4.7`, and your fork of the `core` repo is at the same level as your project, add the following to `overrides`: | ||
> | ||
> ``` | ||
> "@metamask/assets-controllers@^3.4.7": "file:../core/packages/assets-controllers" | ||
> ``` | ||
If you've forked this repository, you can create preview versions for a branch by following these steps: | ||
4. Run `yarn install`. | ||
1. First, since preview build releases are published under an NPM organization, you'll need access to one. If this is not the case, you can either [create a new organization](https://www.npmjs.com/org/create) or [convert your existing username into an organization](https://www.npmjs.com/org/upgrade). | ||
3. Due to the use of Yarn's `file:` protocol, if you update the package in the monorepo, then you'll need to run `yarn install` in the project again. | ||
2. Once you've done this, open the `package.json` for each package that you want to publish and change the scope in the name from `@metamask` to `@<NPM_ORG>`, replacing `NPM_ORG` appropriately. | ||
### Testing changes to packages with preview builds | ||
3. Next, run the following commands to generate preview versions for all packages based on the current branch and publish them (again, replacing `NPM_ORG`): | ||
If you want to test changes to a package where it would be unwieldy or impossible to use a local version, such as on CI, you can publish a preview build and configure your project to use it. | ||
#### Publishing preview builds as a MetaMask contributor | ||
If you're a member of the MetaMask organization, you can create preview builds based on a pull request by following these steps: | ||
1. Post a comment on the pull request with the text `@metamaskbot publish-preview`. This starts the `publish-preview` GitHub action, which will create preview builds for all packages in the monorepo. | ||
2. After a few minutes, the action should complete and you will see a new comment that lists the newly published packages along with their versions. | ||
Note two things about each package: | ||
- The name is scoped to `@metamask-previews` instead of `@metamask`. | ||
- The ID of the last commit in the branch is appended to the version, e.g. `1.2.3-e2df9b4` instead of `1.2.3`. | ||
Now you can [use these preview builds in your project](#using-preview-builds). | ||
If you make more changes to a package, follow step 2 again, making sure to update the reference to the package in your project's `package.json` to use the newly published preview version. | ||
#### Publishing preview builds as an independent contributor | ||
If you've forked this repository, you can create preview builds based on a branch by following these steps: | ||
1. First, since an NPM scope is used to host preview build releases, you'll need access to one. If you do not, you can either [create a new organization](https://www.npmjs.com/org/create) or [convert your existing username into an organization](https://www.npmjs.com/org/upgrade). | ||
2. Once you've done this, open the `package.json` for each package that you want to publish and change the scope in the name from `@metamask` to `@<NPM_ORG>`, replacing `NPM_ORG` with your NPM organization. | ||
3. Next, run the following command to create and publish the preview builds (again, replacing `NPM_ORG` as appropriate): | ||
``` | ||
yarn prepare-preview-builds "@<NPM_ORG>" "$(git rev-parse --short HEAD)" && yarn build && yarn publish-previews | ||
|
@@ -85,11 +127,11 @@ If you've forked this repository, you can create preview versions for a branch b | |
- The name is scoped to the NPM organization you entered instead of `@metamask`. | ||
- The ID of the last commit in the branch is appended to the version, e.g. `1.2.3-e2df9b4` instead of `1.2.3`. | ||
4. Once this command is complete, all preview builds will be published, and you can then [use them in your project](#using-preview-builds). | ||
Now you can [use these preview builds in your project](#using-preview-builds). | ||
If you make more changes and wish to publish another set of preview builds, run steps 3 and 4 again. | ||
If you make more changes to a package, follow step 3 again, making sure to update the reference to the package in your project's `package.json` to use the newly published preview version. | ||
### Using preview builds | ||
#### Using preview builds | ||
To use a preview build for a package within a project, you need to override the resolution logic for your package manager so that the "production" version of that package is replaced with the preview version. Here's how you do that: | ||
|
@@ -98,24 +140,24 @@ To use a preview build for a package within a project, you need to override the | |
3. Add a line to this section that mirrors the dependency entry on the left-hand side and points to the preview version on the right-hand side: | ||
``` | ||
"@metamask/<PACKAGE_NAME>@<PRODUCTION_VERSION_SPECIFIER>": "npm:@<NPM_ORG>/<PACKAGE_NAME>@<PREVIEW_VERSION>" | ||
"@metamask/<PACKAGE_NAME>@<PRODUCTION_VERSION_RANGE>": "npm:@<NPM_ORG>/<PACKAGE_NAME>@<PREVIEW_VERSION>" | ||
``` | ||
4. Run `yarn install`. | ||
|
||
For example: | ||
> **Example:** | ||
> | ||
> - If you're a member of MetaMask, your project uses Yarn, `@metamask/controller-utils` is listed in dependencies at `^1.1.4`, and you want to use the preview version `1.2.3-e2df9b4`, add the following to `resolutions`: | ||
> | ||
> ``` | ||
> "@metamask/controller-utils@^1.1.4": "npm:@metamask-previews/[email protected]" | ||
> ``` | ||
> | ||
> - If you are an individual contributor, your project uses NPM, `@metamask/assets-controllers` is listed in dependencies at `^3.4.7`, and you want to use the preview version `4.5.6-bc2a997` published under `@foo`, add the following to `overrides`: | ||
> | ||
> ``` | ||
> "@metamask/assets-controllers@^3.4.7": "npm:@foo/[email protected]" | ||
> ``` | ||
- If you're a member of MetaMask, your project uses Yarn, `@metamask/controller-utils` is listed in dependencies at `^1.1.4`, and you want to use the preview version `1.2.3-e2df9b4`, add the following to `resolutions`: | ||
|
||
``` | ||
"@metamask/controller-utils@^1.1.4": "npm:@metamask-previews/[email protected]" | ||
``` | ||
|
||
- If you are an individual contributor, your project uses NPM, `@metamask/assets-controllers` is listed in dependencies at `^3.4.7`, and you want to use the preview version `4.5.6-bc2a997` published under `@foo`, add the following to `overrides`: | ||
|
||
``` | ||
"@metamask/assets-controllers@^3.4.7": "npm:@foo/[email protected]" | ||
``` | ||
4. Run `yarn install`. | ||
## Releasing | ||
|
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Has it been considered to use unpublished placeholder versions here? Like
^3.2.0-next
. The published versions would of course have theirpackage.json
aligned with published versions. This would remove the risk of accidentally resolving from registry while keeping the ease of override in this PR, I think?It's a pattern I recall seeing from time to time.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just so I understand, you're saying that if we use static versions there is a risk that a dependency on an internal package could point to a published version instead of the internal version, but if we use an unpublished version then we never have that risk?
If so, that's a good point. One immediate thought I have is that this shouldn't be an issue, because you need to use Yarn whenever installing dependencies in this repo, and Yarn seems to be smart about understanding that if package A depends on package B and package B is a workspace package, and the version range of the dependency matches the version of package B in its manifest, it won't attempt to download the published version of package B. This can be observed via the lockfile: https://github.com/MetaMask/core/blob/make-yarn-link-work-again/yarn.lock#L1330
Another thing to think about is that if we added a suffix, we'd have to make sure to strip it when we release. This isn't hard to do in the context of this repo — we can do it in a
prepack
step — but we'd also need to do it in our GitHub Actions and increate-release-branch
, where there are various places that we read the current version of a package by looking directly at itspackage.json
. This would add some complexity to these pieces that could be tricky to maintain, so I'd want to make sure we'd need them.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yup. And there's some history for this happening. Especially since this is a monorepo which includes external dependencies in the same namespace as the monorepo itself (
@metamask/
), it's an easy enough mistake to make in larger changes, even for those who actually do read scan through each lockfile change manually. I've seen it happen multiple times in other monorepos using this scheme and I suspect it's one reason behind theworkspace:^
scheme in the first place.And yes, it would require some updates to release/publish scripts. But there is already an implicit requirement to bump version precisely once per successful publish.
What is the benefit of having checked-in versions always match the last published version, as opposed to unpublished versions? It seems that you never actually want them to match, and they will start to drift at the first commit after each release despite version numbers matching. It seems to me that this will mean that version-bumping should also change to be done at the time of commit of changes (e.g. bump package to a major at the point of merging a breaking change) as opposed to like now, at point of release.
I may be missing the reasoning behind why these versions specifically are desired, though.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maye @haltman-at / @eggplantzzz have a useful perspective to share given that Truffle seems to already be using the same scheme proposed here.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The constraints mitigate this risk pretty well I'd think. They automatically keep the ranges synchronized with the current versions of each package. Contributors should never have to remember to update them at all.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We developed the release process for the monorepo this way not so much because there was a clear benefit to doing so, but because it evolved from the standard release process that we (and the JavaScript community at large) follow for polyrepos.
I hear what you're saying. I think this might just be a different perspective, however. The checked-in version of a package doesn't mean "the last published version"; rather, it means "the next version to publish". On top of this, our release automation assumes that a package should be published if its version has changed, and that it shouldn't otherwise. This is the reason why we don't bump the version of a package until we want to create a new release. (We are talking about changing this so that contributors can queue version bumps along with changes that necessitate them, but even in that case, I don't think we would actually bump the version of the package in the manifest until release time.)
Returning to your first paragraph, I don't want to dismiss this and I definitely appreciate your perspective. That said, do you think that Yarn constraints would adequately protect against what you've seen?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Appreciate the explanation. I did miss the yarn constraints initially, which I agree should address the much of the potential practicalities, yes.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Merging since we've got 2 'yes' votes on this and we want to unblock another team we're closely working with, but I don't want to stop the conversation here.