Monorepo multi-package release practice based on Rush

Preface

In May, I shared the application-level Monorepo optimization plan , which mainly explained the problems and solutions of the previous monorepo (Yarn + Lerna), but in this sharing, it did not involve the pacakge release related content (in that period mainly It is mainly based on application development), occasionally pacakge development is also a scenario with relatively simple dependencies (single package development/release), npm publish can be done by using 0619791d22ca90.

With subsequent development (mainly the migration of another warehouse within the team), package development scenarios accounted for a considerable proportion (the number of warehouse code lines reached one million, and the number of projects exceeded 100), but the multi-package release experience did not Not very good, mainly in the following 3 aspects:

1. The publishing method is quite different from that of Lerna, and the Rush related command documents are relatively simple (too simple, the parameters have been tried many times), and it is impossible to get started quickly;
2. The release process is not standardized enough, basically relying on hand-knocking the command line;
3. Lack of standard development workflow.

This sharing is to solve the above problems and find out the best practice of Monorepo multi-package release scenario in practice.

Workspace protocol (workspace:)

Before discussing, we must first understand Workspace protocol (workspace:) , here is pnpm as an example, the following example is taken from Workspace | pnpm

By default, if the package version available in the workspace matches the declared range, pnpm will link the package from the workspace. For example, there is foo@1.0.0 , and another project bar in "foo: ^1.0.0" depends on 0619791d22cbbb, then bar will use foo in the workspace. If bar depends on "foo: 2.0.0" , then pnpm will download foo@2.0.0 for use by bar, which introduces some Uncertainty.

When using the workspace protocol, pnpm will refuse to resolve to anything other than the local workspace package. Therefore, if you set "foo": "workspace:2.0.0" , the installation will fail this time because "foo@2.0.0" does not exist in the workspace.

Multi-package release

Basic operation

Compared with the traditional single warehouse and single package to manually release one by one, one of the advantages of monorepo is that it can easily release multiple packages.

rush change

In Rush monorepo, rush change is the starting point of the contracting process, and its product <branchname>-<timestamp>.json (substituting changefile.json later) will be rush version by 0619791d22cc3c and rush publish .

The changefile.json generation process is as follows:

1. Detect the difference between the current branch and the target branch (usually master), and filter out the changed items (based on the git diff command);
2. Inquire some information (such as the version update strategy and a brief description of the update content) through interactive command lines for each project selected;
3. Based on the above information, generate the changefile.json corresponding to the package common/changes

Note: The change type (type field) in the screenshot is none, not any of major/minor/patch. None means "roll these changes to the next patch, minor or major Major", so in theory, if there are only change files of type "none" for a project, it will neither consume files nor upgrade the version.

type: none allows us to integrate the has been developed but does not need to follow the next release cycle of into the master in advance, until the changefile.json whose type is not none appears in the pacakge.

rush version and rush publish

rush version or rush publish --apply will update the version number based on the generated changefile.json (that is, bump version, following the semver specification, the version number of the upper package of the released package may be updated, which will be described in detail in the next section).

rush publish --publish will release the corresponding package based on changefile.json.

The release process of Rush is basically the same as that of Changesets, another popular Monorepo scene delivery tool. If you encounter a simple PNPM Monorepo, you may be able to reuse the solution in this article based on Changesets🥳.

• 🦋 A way to manage your versioning and changelogs with a focus on monorepos
• Changesets: The popular Monorepo scenario distribution tool

Cascade release

As mentioned earlier, when updating the version number, in addition to updating the version number of the package that currently needs to be released, the version number of the upper-level package may also be updated, depending on how the upper-level package references the current package in package.json.

Shown below, @modern-js/plugin-tailwindcss (upper layer package) by "workspace:^1.0.0" form introduced @modern-js/utils (bottom package).

package.json(@modern-js/plugin-tailwindcss)

{
  "name": "@modern-js/plugin-tailwindcss",
  "version": "1.0.0",
  "dependencies": {
    "@modern-js/utils": "workspace:^1.0.0"
  }
}

package.json(@modern-js/utils)

{
  "name": "@modern-js/utils",
  "version": "1.0.0"
}

• If @modern-js/utils updated to 1.0.1 , Rush will not update the version number of @modern-js/plugin-tailwindcss Because ^1.0.0 compatible with 1.0.1 , from a semantic point of view, @modern-js/plugin-tailwindcss does not need to update the version number. You can get @modern-js/utils@1.0.1 by @modern-js/plugin-tailwindcss@1.0.0
• If @modern-js/utils update to 2.0.0 , Rush when updating the version number will be updated @modern-js/plugin-tailwindcss version number to 1.0.1 . Because ^1.0.0 not compatible with 2.0.0 , update the @modern-js/plugin-tailwindcss version to 1.0.1 cite the latest @modern-js/utils@2.0.0 . At this time, the content of the package.json of @modern-js/plugin-tailwindcss

{
  "name": "@modern-js/plugin-tailwindcss",
  "version": "1.0.1",
  "dependencies": {
   // 引用版本号也发生了变化 
    "@modern-js/utils": "workspace:^2.0.0"
  }
}

The version number is updated, and it needs to be published to npm. At this time, you need to add the --include-all rush publish . After configuring this parameter, rush publish checks that shouldPublish: true package in the warehouse is newer than the npm version, the package will be released.

This completes the semantic-based cascading release.

Unexpected release

At the beginning of the transformation project based on Rush, the projects in the monorepo always used "workspace: *" each other, that is, the latest version in the monorepo was used.

This leads to two problems:

1. When the app is online, it may bring the package in the development process to go online (based on the Trunk Based Development development branch model, master is the trunk branch)
2. Unexpected releases will be brought when the package is sent. Because "workspace: *" , the bottom package is updated and the upper package must be released (in order to ensure the semantics of *

Therefore, references between projects in monorepo need to follow the following specifications.

Reference norm

1. Determine whether you need to use workspace: reference the latest version in monorepo
2. If you need to use workspace: , please use "workspace: ^x.x.x" instead of "workspace: *" to avoid meaningless releases (of course, you also need to consider actual dependencies. If packageA and packageB always need to be released together, you should use "workspace: *" )

With the increasing number of projects in monorepo, if all workspace: are used between projects, then when a package is updated, all its internal access parties are required to passively perform regression testing

It is of course no problem to improve the master branch admission standard through CI, but the business scenarios are often too complicated, and the intervention of test students is still required, unless the code quality and test quality of team members are extremely high.

Or use the feature flag mechanism for control, but manual control is often costly and requires the cooperation of mature infrastructure solutions.

For projects that are not closely related to the business, they only exist in the same monorepo at the physical level, and do not need to pay attention to the latest version of the other party, and do not need to use workspace: .

For example, it is reasonable to put babel/react/modernjs and other packages in the same monorepo management. It is workspace: inside each package to enjoy the advantages of monorepo engineering, and it is more appropriate to directly use the stable version of npm for dependencies between projects.

Of course, the business boundaries of open source projects are very obvious, and specific to our business warehouse (one warehouse for a team), there may be many modules that can't be workspace: . At this time, using 0619791d22d218 is asking for trouble.

So how to judge whether you need to use workspace: ?

For example, suppose I am the owner of package bar, and now I want to reference a package foo, I need to pass the following judgments:

foo update, bar will definitely be updated and tested and the iterative online rhythm is the same, then use workspace: , otherwise use the remote version of npm.

Workflow

have to be aware of is:

1. When the function points of the development phase are merged into the trunk branch (master branch) type: none of 0619791d22d2a6 is generated. This is to avoid other packages being released with packages that are in the development process.
2. Because it is necessary to generate the type: major/minor/patch of 0619791d22d2ca to release the test package in the test branch, so the test phase will not be merged, and the official version will be merged after the acceptance is completed.

Detailed assembly line

test version

1. Get the package to be released this time based on changefile.json

2. Install the dependencies of the target package on demand

   • rush install -t package1 -t package2

3. Build the target package on demand

   • rush build -t package1 -t package2

4. rush publish reads changefile.json to update the version number

   • rush publish --prerelease-name [canary.x] --apply

5. rush publish publishes the package with the version number changed

   • rush publish --publish --tag canary --include-all --set-access-level public
6. Sync the posted information to the relevant notification group through the robot

official version

1. Get the package to be released this time based on changefile.json

2. Install the dependencies of the target package on demand

   • rush install -t package1 -t package2

3. Build the target package on demand

   • rush build -t package1 -t package2
4. Pull a target branch to carry the commits generated in the release process (this branch can be understood as a release branch)

5. rush version Consume changefile.json on the target branch pulled in the previous step to update the version number and generate CHANGELOG.md

   • rush version --bump --target-branch [source-branch] --ignore-git-hooks
6. Execute rush update on the target branch to update the lockfile to avoid inconsistency between package.json and lockfile

7. rush publish publish package to npm

   • rush publish --apply --publish --include-all --target-branch [source-branch] --add-commit-details --set-access-level public

8. Generate a Merge Request that merges the target branch to the master branch

   1. Deleting change files and updating change logs for package updates.
   2. Applying package updates.
   3. rush update.
9. Synchronize the release information to the relevant notification group through the robot (including Merge Request information, which needs to be merged in time)

Release acceleration

As you can see, the first three steps in the publishing process are the same:

1. Get the package to be released this time based on changefile.json

2. Install the dependencies of the target package on demand

   • rush install -t package1 -t package2

3. Build the target package on demand

   • rush build -t package1 -t package2

But when this solution just landed, it used the simple and rude way of "installing all monorepo dependencies and building all packages".

What monorepo needs to solve is the scale problem: the project is getting bigger and bigger, the dependency installation is getting slower and slower, the build is getting slower and slower, and the running test case is getting slower and slower.

"On-demand" has become a key word. pnpm is already very good as a package manager, and it can even install dependencies on demand, but it is still lacking in the capabilities required for large monorepo, so we introduced Rush to solve the engineering under monorepo problem.

So the goal is clear: as the monorepo grows larger and larger, the complexity of the entire project has always been maintained at a stable level. —— Application-level Monorepo optimization plan

Before optimization, one release is close to 12min, even if only one package is released, and there is only one sentence console.log("hello world") this package, and as the number of projects increases, 12min may be just the starting point. So "on demand" is back to our sight.

Rush will change the version number of the project that needs to be released during the release process. As long as this process is advanced and the project with the changed version number is obtained in advance, the target parameters of the install and build commands can be obtained.

So by @microsoft/rush-lib through the rush version , I got the following code:

function getVersionUpdatedPackages(params: {
  rushConfiguration: RushConfiguration;
  prereleaseName?: string;
}) {
  const { prereleaseName, rushConfiguration } = params;
  const changeManager: ChangeManager = new ChangeManager(rushConfiguration);

  if (prereleaseName) {
    const prereleaseToken = new PrereleaseToken(prereleaseName);
    changeManager.load(rushConfiguration.changesFolder, prereleaseToken);
  } else {
    changeManager.load(rushConfiguration.changesFolder);
  }
  // 改变 package.json 版本号（内存中，实际文件不做改动）
  changeManager.apply(false);
  return rushConfiguration.projects.reduce((accu, project) => {
    const packagePath: string = path.join(
      project.projectFolder,
      FileConstants.PackageJson,
    );
    // 实际 package.json 的版本号
    const oldVersion = (JsonFile.load(packagePath) as IPackageJson).version;
    // 内存中 package.json 的版本号
    const newVersion = project.packageJsonEditor.version;
    // 不一致则为我们的目标项目
    if (oldVersion !== newVersion) {
      accu.push({ name: project.packageName, oldVersion, newVersion });
    }
    return accu;
  }, [] as UpdatedPackage[]);
}

Auxiliary command

rush change-extra

It is caused by the trouble caused by the lockfile of the access party. This command can generate changefile.json for the unchanged package so that it can be released.

rush change command will compare the difference between the current branch and the master branch, find out the project that has changed, and let the developer generate the corresponding changefile.json file through the interactive command line.

As mentioned in the previous "cascading release", Rush can update the version of the relevant package according to the semver specification and release it. Under "workspace: ^x.x.x" , unless the underlying package undergoes a major update, the upper package will not be updated and released.

The problem lies in this. The upper-layer package is not released, and the lower-layer package is locked by the lockfile of the access party. We (forced) need a solution to release packages that do not actually need to be released (here @jupiter/block- tools), this is the reason why rush change-extra was born.

There is a need for a way to update the specified dependencies in depth, but there is currently no solution to the package manager dimension.

Concluding remarks

This article starts with Rush's basic outsourcing operations, introduces some of the problems encountered in the actual development process and gives the overall landing plan, and optimizes the online publishing speed based on the "on-demand" idea.