rfc/20240529-OpenTofu-Specific-Code-Override.md
Issue: https://github.com/opentofu/opentofu/issues/1275
Today, OpenTofu uses the .tf extension files as the main configuration files. This is the result of forking Terraform ~1.5.5. As time goes by, the OpenTofu code starts to diverge from the original Terraform codebase in different ways:
This divergence between OpenTofu and Terraform configuration syntax imposes challenges on the following personas:
In addition to supporting .tf extension files as we do today, OpenTofu will support .tofu extension files. When two files have the same name, one with the .tf extension and one with the .tofu extension, OpenTofu will load only the .tofu file and ignore the .tf file. Hence, creating a mechanism that overrides specific files. This way the .tofu extension files will be used for OpenTofu specific features and syntax and the .tf extension files can be used for Terraform compatible features and syntax. Allowing users and module maintainers to use OpenTofu and Terraform in the same project without conflicts.
For example, If I have two files:
// vars.tf
variable "region" {
type = string
}
// vars.tofu
variable "region" {
type = string
deprecated = "Reason this was deprecated"
}
Terraform would read all files including vars.tf and ignore vars.tofu.
OpenTofu would read all files, but replace vars.tf with vars.tofu and will include the additional deprecated attribute.
[!NOTE] This solution should not be confused with the Override Files feature that is already supported in OpenTofu. The Override Files feature is used to override specific portions of an existing file, while the proposed solution here will override the entire file. We chose to implement it this way because Terraform may introduce features in the future that OpenTofu cannot parse, making specific overrides insufficient.
As we know, .tf are not the only files that OpenTofu can load, in here we will drill down into all the supported scenarios:
.tofu file exists, we’ll load it instead of the .tf file..tofu.json file exists, we’ll load it instead of the tf.json file..tofutest.hcl/.tofutest.json files exist, we’ll load them instead of the .tftest.hcl/.tftest.json files._override.tofu/_override.tofu.json files exist, we’ll load them instead of the _override.tf/_override.tf.json files. To further clarify, if a main.tofu file exists with the override file main_override.tf, we’ll load them both (unless main_override.tofu exists, and then we'll load it instead of main_override.tf)..tofuvars file exists, we’ll load it instead of the .tfvars file..tofuvars.json file exists, we’ll load it instead of the .tfvars.json file.As part of this RFC we'll address the following user stories, and try to understand the user experience and how the suggested solution will impact our users:
Today:
As a module developer, I want to start supporting OpenTofu in my module, so my users can use it with both OpenTofu and Terraform. Today, my module automatically supports both Terraform and OpenTofu for features that were released until ~1.5.5 for both projects. The problem starts when I want to use new features that were released in versions > ~1.5.5:
required_version setting as I can't define two different minimum versions for OpenTofu and Terraform in the same module.With this solution implemented:
.tf file and a .tofu file with very similar content. The only difference is that the .tofu file will contain OpenTofu specific features that are not supported by Terraform. This way, I can use the new features without breaking my module's compatibility with Terraform..tf file and a .tofu file both containing:
terraform {
required_version = ">= x.x.x"
}
.tf file I'll put my minimum Terraform version, and in the .tofu file I'll put my minimum OpenTofu version. This way, I can set different minimum versions for OpenTofu and Terraform in the same module.Today:
Unfortunately, I don't have a simple way to do it today. I cannot limit my modules/configurations to be loaded only by OpenTofu.
With this solution implemented:
If I include only .tofu files in my module/configuration, I can be sure it'll be supported only by OpenTofu. This way, I can use OpenTofu-specific features without worrying that others might run the module/configuration by mistake with Terraform instead of OpenTofu.
Today:
I can migrate to OpenTofu very easily, and mostly keep my configuration the same way it is with minimal required code changes in some cases. But for using OpenTofu-specific configuration I still need to change my existing configuration files.
With this solution implemented:
I have another option for the migration process. Instead of changing my existing .tf configuration files to try OpenTofu-specific features, I can create a new .tofu file with the same content as the .tf file and start using OpenTofu-specific features in the .tofu file. This way, I can experiment with OpenTofu-specific features without changing my existing .tf files. If I decide to go back, I can simply delete the .tofu file and keep using the .tf file.
Today:
I can write OpenTofu code in my IDE, but I don't have complete support for new OpenTofu features. For example, my IDE will not recognize the syntax of State Encryption and will show a warning when I try to use it. This makes it harder for me to write OpenTofu code efficiently and without errors. In addition, I can't use some 3rd party tools that support Terraform, because they don't support OpenTofu.
With this solution implemented:
This problem will not be solved as part of the technical solution in this RFC, but in my opinion we should also address these issues as a part of this RFC. This is an opportunity to standardize the way IDEs and 3rd party tools support OpenTofu. If we want to support .tofu files, we should also make sure that IDE plugins and 3rd Party Tools can support the new files.
We want to give our users the best possible experience when they work with OpenTofu in their IDE. For a comfortable experience our users need features like code completion, syntax highlighting and validation, jump to definition, and others when they work with OpenTofu. If we add support for .tofu files, we need to make sure that the IDE plugins that support OpenTofu will also support the new file extensions.
Today, VSCode supports Terraform using Terraform Language Server and Terraform Extension. The VSCode Extension is using the Terraform Language Server as part of its implementation. Both projects were already forked by a member of the community - OpenTofu Language Server and OpenTofu VSCode Extension. By looking at the projects, it looks like some work has been done, but support is still lacking for features that were introduced as part of OpenTofu 1.7. We should talk to the maintainer and see how we can contribute and make sure the plugins are up-to-date for supporting the latest OpenTofu features. In addition, we should make sure that the plugins will support the new file extensions that we are going to introduce.
Currently, for Terraform, Jetbrains IDEs has the Terraform and HCL plugin that is not using the LSP protocol. The license of the plugin is Apache 2.0, based on the Marketplace link mentioned before, although I didn't find a clear license in the source code repository. We should talk to the maintainer. We might be able to contribute to the plugin to support OpenTofu, or more likely, create a new plugin that supports only OpenTofu (and the new file extensions suggested in this RFC). In case we want to create a new plugin, we might want to consider that on 2023, Jetbrains started supporting LSP for Plugin Developers and we can use the OpenTofu Language Server for the development of the new plugin. Although, it can only support paid IntelliJ-based IDEs, as explained here.
.tf and .tofu files based on user preferenceIn both Jetbrains IDEs and VSCode, plugins can be configured by the end user in the Settings menu. Possibly, we can add an optional setting to the plugin that allows it to recognize .tf files as OpenTofu files. This option can be very helpful for users switching to OpenTofu from Terraform, and want to get quick support for OpenTofu in the IDE without changing the file extension to .tofu. On the other hands, people who want to use both Terraform and OpenTofu in the same project can turn this setting off, and the plugin will not go over the .tf files. This setting should be per project and not for the whole IDE (which should be possible based on Jetbrains documentation and VSCode documentation).
The OpenTofu Language Server can be used with other IDEs that support the Language Server Protocol. There are different websites that provide a list of LSP implementations (Like, this list by Microsoft and Langserver.org) and we should enlist the OpenTofu Language Server, so users will be able to easily find it.
Today, some 3rd party tools officially support OpenTofu. Many of them are listed in the awesome-opentofu repository. Implementing the solution suggested in this RFC might affect 3rd party tools maintainers and they will have to create adjustment support the new .tofu extension files. As part of this RFC we should map those tools, understand if they are impacted by this change, and reach out to the maintainers and notify them about this change or contribute.
Furthermore, this change could potentially incentivize other tools to also support OpenTofu. With a clear strategy for managing code divergence in place, maintainers of those tools may gain a better understanding of how to provide support for OpenTofu. Additionally, they can begin by accommodating the new file extensions that we will be introducing in this RFC.
The technical change required to implement this RFC is fairly simple:
internal/configs package that loads .tf files to also load .tofu files (and all other file extensions mentioned in the Supported Scenarios section)..tf files when a .tofu file exists with the same name.fmt command - extend the relevant code in here.We have a working proof of concept and the implementation will be very similar to it. So the change in the code is very minimal and isolated to specific locations in the codebase.
.tofu files?A major drawback of this solution is that it leads to code duplication for users who need to stay compatible with both OpenTofu and Terraform (such as module maintainers). If These users want to use an OpenTofu-specific feature, they must create and manage two copies of the same file, one with the .tf extension and one with the .tofu extension. Potentially, in the future, they might have multiple duplicated files in their project. It can be confusing and error-prone, and might cause configuration drifts between the Terraform and OpenTofu configuration over time.
.tf files to .tofu files in the project. This way, users will have to put a very small effort in migrating to OpenTofu. A good example for a similar tool is the Java to Kotlin Converter..tofu files..tofu file will not override the .tf file completely, but will instead override specific blocks. This way, we can avoid code duplication. But this solution is not sufficient for the future, when Terraform might introduce features that OpenTofu cannot parse..otf, but we decided to use .tofu because the .otf extension is already used to represent OpenType font files.In my opinion, this is an opportunity to standardize the way IDEs and 3rd party tools support OpenTofu. So, we should not only pursue technical tasks, but also include peripheral tasks to establish a better experience for our users. Here are the tasks that should be done as part of this RFC: