doc/development/ruby_upgrade.md
We strive to run GitLab using the latest Ruby MRI releases to benefit from performance and security updates and new Ruby APIs. When upgrading Ruby across GitLab, we should do so in a way that:
Before making changes to Ruby versions, read through this document carefully and entirely to get a high-level understanding of what changes may be necessary. It is likely that every Ruby upgrade is a little different than the one before it, so assess the order and necessity of the documented steps.
The first thing to consider when upgrading Ruby is scope. In general, we consider the following areas in which Ruby updates may have to occur:
We may not always have to touch all of these. For instance, a patch-level Ruby update is unlikely to require updates in third-party gems.
When assessing scope, the Ruby version level matters. For instance, it is harder and riskier to upgrade GitLab from Ruby 2.x to 3.x than it is to upgrade from Ruby 2.7.2 to 2.7.4, as patch releases are typically restricted to security or bug fixes. Be aware of this when preparing an upgrade and plan accordingly.
To help you estimate the scope of future upgrades, see the efforts required for the following upgrades:
Before any upgrade, consider all audiences and targets, ordered by how immediately they are affected by Ruby upgrades:
.ruby-version affects everyone using tooling that interprets these files.
The developers are affected as soon as they pull from the repository containing the merged changes..ruby-version.
Instead, they use the Ruby installed in the Docker container they execute in, which is defined in .gitlab-ci.yml.
The container images used in these jobs are maintained in the gitlab-build-images repository.
When we merge an update to an image, CI/CD jobs are affected as soon as the image is built..ruby-version is meaningless in this environment. Instead, those Docker images must be patched to upgrade Ruby.
GitLab.com is affected with the next deployment.Timing all steps in a Ruby upgrade correctly is critical. As a general guideline, consider the following:
Either way, we found that from past experience the following approach works well, with some steps likely only necessary for minor and major upgrades. Note that some of these steps can happen in parallel or may have their order reversed as described above.
Tracking this work in an epic is useful to get a sense of progress. For larger upgrades, include a timeline in the epic description so stakeholders know when the final switch is expected to go live. Include the designated performance testing template to help ensure performance standards on the upgrade.
Break changes to individual repositories into separate issues under this epic.
Especially for upgrades that introduce or deprecate features, communicate early that an upgrade is due, ideally with an associated timeline. Provide links to important or noteworthy changes, so developers can start to familiarize themselves with changes ahead of time.
GitLab team members should announce the intent in relevant Slack channels (#backend and #development at minimum)
and Engineering Week In Review (EWIR). Include a link to the upgrade epic in your
communication.
To build and run Ruby gems and the GitLab Rails application with a new Ruby, you must first prepare CI/CD and developer environments to include the new Ruby version. At this stage, you must not make it the default Ruby yet, but make it optional instead. This allows for a smoother transition by supporting both old and new Ruby versions for a period of time.
There are two places that require changes:
RUBY_VERSION.
All projects building against the same minor release automatically download the new patch release./patches directory to a new folder matching the Ruby version you upgrade to, or they aren't applied..tool-versions so asdf
users will benefit from this. Other users will have to install it manually
(example.)For larger version upgrades, consider working with Quality Engineering to identify and set up a test plan.
For patch releases this is unlikely to be necessary, but
for minor and major releases, there could be breaking changes or Bundler dependency issues when gems
pin Ruby to a particular version. A good way to find out is to create a merge request in gitlab-org/gitlab
and see what breaks.
This is typically necessary, since gems or Ruby applications that we maintain ourselves contain the build setup such as
.ruby-version, .tool-versions, or .gitlab-ci.yml files. While there isn't always a technical necessity to
update these repositories for the GitLab Rails application to work with a new Ruby,
it is good practice to keep Ruby versions in lock-step across all our repositories. For minor and major
upgrades, add new CI/CD jobs to these repositories using the new Ruby.
A build matrix definition can do this efficiently.
When upgrading Ruby, consider updating the repositories in the ruby/gems group as well.
For reference, here is a list of merge requests that have updated Ruby for some of these projects in the past:
To assess which of these repositories are critical to be updated alongside the main GitLab application consider:
Refer to the list of GitLab projects for a complete account of which repositories could be affected. For smaller version upgrades, it can be acceptable to delay updating libraries that are non-essential or where we are certain that the main application test suite would catch regressions under a new Ruby version.
[!note] Consult with the respective code owners whether it is acceptable to merge these changes ahead of updating the GitLab application. It might be best to get the necessary approvals but wait to merge the change until everything is ready.
With the dependencies updated and the new gem versions released, you can update the main Rails application with any necessary changes, similar to the gems and related systems. On top of that, update the documentation to reflect the version change in the installation and update instructions (example).
[!note] Be especially careful with timing this merge request, since as soon as it is merged, all GitLab contributors will be affected by it and the changes will be deployed. You must ensure that this MR remains open until everything else is ready, but it can be useful to get approval early to reduce lead time.
With the new Ruby made available as an option, and all merge requests either ready or merged,
there should be a grace period (1 week at minimum) during which developers can
install the new Ruby on their machines. For GDK and asdf users this should happen automatically
via gdk update.
This pause is a good time to assess the risk of this upgrade for GitLab.com. For Ruby upgrades that are high risk, such as major version upgrades, it is recommended to coordinate the changes with the infrastructure team through a change management request. Create this issue early to give everyone enough time to schedule and prepare changes.
If there are no known version compatibility issues left, and the grace period has passed, all affected repositories and developer tools should be updated to make the new Ruby default.
At this point, update the GitLab Compose Kit (GCK).
This is an alternative development environment for users that prefer to run GitLab in docker-compose.
This project relies on the same Docker images as our runners, so it should maintain parity with changes
in that repository. This change is only necessary when the minor or major version changes
(example.)
As mentioned above, if the impact of the Ruby upgrade on SaaS availability is uncertain, it is prudent to skip this step until you have verified that it runs smoothly in production via a staged rollout. In this case, go to the next step first, and then, after the verification period has passed, promote the new Ruby to be the new default.
The last step is to use the new Ruby in production. This requires updating Omnibus and production Docker images to use the new version.
To use the new Ruby in production, update the following projects:
Charts like the GitLab Helm Chart should also be updated if they use Ruby in some capacity, for example to run tests (see this example), though this may not strictly be necessary.
If you submit a change management request, coordinate the rollout with infrastructure engineers. When dealing with larger upgrades, involve Release Managers in the rollout plan.
If the upgrade was a patch release and contains important security fixes, it should be released as a GitLab patch release to self-managed customers. Consult our release managers for how to proceed.
There are several tools that ease the upgrade process.
We also log Ruby and Rails deprecation warnings to a dedicated log file, log/deprecation_json.log
(see GitLab Developers Guide to Logging for where to find GitLab log files),
which can provide clues when there is code that is not adequately covered by tests.
For GitLab.com, GitLab team members can inspect these log events in Kibana
(https://log.gprd.gitlab.net/goto/f7cebf1ff05038d901ba2c45925c7e01).
During the upgrade process, consider the following recommendations: