docs/fir/k2_kmp.md
This document describes the implementation of the KMP support in the K2 Compiler along components.
Note: The current compilation model is temporary, and should be changed to final in KT-57327
A bunch of source files along with its depends-on relations with other source-sets.
See: https://kotlinlang.org/docs/multiplatform-discover-project.html#source-sets
Note: Binary dependencies are provided on the compilation level and not considered a part of the source-set definition by the compiler.
A compiler module, entity with source-code and dependencies.
In a platform compilation, multiple modules are created from the input source-sets.
A single invocation of the compiler entry-point
Hierarchical multi-platform projects
The process of replacing references to expect declarations with corresponding actual declarations. See IRActualizer
A relation between two declarations that form an expect-actual pair.
In general case, the relation is many-to-many. However, in correct code it is either one-to-one, or one-to-many in case of type-aliases.
See FIR: actual -> expect binding construction
A [source-set] that may contain expect declarations. As opposed to a platform source-sets that must only contain actual or regular declarations.
Common source-sets usually contain code that is shared across different targets, and by that is included in multiple platform compilations.
However, one may define HMPP hierarchy with only one target while still having common source-sets.
-Xrefines-pathEntry-point: org.jetbrains.kotlin.cli.metadata.K2MetadataCompiler
Also, see: org.jetbrains.kotlin.cli.metadata.FirMetadataSerializer
Inputs:
Outputs:
In this mode, we analyze only common sources. This compilation mode is very similar to a usual compilation pipeline with the exception that no IR is serialized to the output.
Actual-expect matching and checking are performed, since in HMPP we can have actuals in metadata compilation inputs.
Note: The full set of actual declarations can't be determined at that point. The only actual -> expect binding could be obtained and checked.
Note: In this compilation mode actual->expect binding may point to a deserialized element, since expect declarations from depends-on source-sets will be present as metadata binary dependencies.
Please consult with https://kotlinlang.org/docs/multiplatform-hierarchy.html for more info on source-set structure.
This is the main part where the pipeline diverges from simple compilation.
Ex:
// MODULE: common
expect fun a1(): String
expect fun a2(): String
// MODULE: intermediate()()(common)
actual fun a1(): String = ""
expect fun a4(): String
expect fun a5(): String
// MODULE: unrelated
expect fun a3(): String
expect fun a5(): String
// MODULE: platform()()(intermediate, unrelated)
actual fun a2(): String = ""
actual fun a3(): String = ""
actual fun a4(): String = ""
actual fun a5(): String = "" // ambiguous
In the given example, modules aka source-sets form the following dependency graph:
common
^
|
intermediate unrelated
^ ^
| /
| /
platform
During the platform compilation, the entire source-set graph is passed to the compiler CLI.
See: -Xfragments, -Xfragment-sources, -Xfragment-refines.
When combined with binary dependencies passed, it allows constructing graph of compiler modules.
Compiler module is created for each passed source-set.
The K2 uses shared platform binary dependencies for analysis of all source-sets in the platform compilation.
The key reason for that is that we are unable to provide full KLibs for common source-sets yet.
Observed behavior is the following:
// MODULE: common_dep
// library: dep
fun foo(a: Any) {} // (dep.1)
// MODULE: platform_dep
// library: dep
fun foo(a: String) {} // (dep.2)
// MODULE: common
// dependencies { implementation("dep") }
fun bar(a: Any) {} // (a.1)
fun test() {
bar("") // (a.1)
foo("") // (dep.2) !!! While platform compilation for this module
}
// MODULE: platform
// depends-on: common
fun bar(a: String) {} // (a.2)
Given that, source analysis order is defined as the following:
Analyze from the most common module to the platform modules.
So that all depends-on modules of the module are analyzed before the module itself.
In order to achieve it, modules are sorted topologically over depends-on relation graph.
Platform compilation pipeline consists of the following steps:
data and value classesFIR2IR is applied to the modules in the topological order.
Inputs:
Outputs:
The FIR2IR state, represented by Fir2IrCommonMemberStorage is shared across entire platform compilation.
See: org.jetbrains.kotlin.fir.backend.Fir2IrCommonMemberStorage
Since FIR2IR is invoked over each module over depends-on graph, we need to avoid creating IR for declarations in a common source-sets multiple times.
The same applies to the declarations from binary dependencies, which are shared among the entire compilation.
Therefore, FIR2IR uses shared storage for get-or-create operations for declarations.
FIR doesn't use fake-overrides in the frontend, and that's why we construct them during FIR2IR in order to match backend expectations
Also, the following example shows that it is impossible to build valid fake overrides during common module compilation.
Ex:
// MODULE: common
expect class A
expect class B
interface I {
fun foo(q: A)
}
interface J {
fun foo(q: B)
}
interface K : I, J {
// FIR2IR F/O fun foo(q: A) (1)
// FIR2IR F/O fun foo(q: B) (2)
}
// MODULE: platform()()(common)
actual typealias A = Int
actual typealias B = Int
class Impl : K {
override fun foo(q: Int) {}
}
In the given example during the actualization, common compilation would produce (1) and (2), which need to be combined into one fake override.
To achieve this, Fir2Ir creates a special symbol for fake overrides calls, and no declarations for fake overrides (except one in Lazy classes). A special symbol, represented by org.jetbrains.kotlin.ir.symbols.impl.IrFakeOverrideSymbolBase and its inheritors, is effectively a pair of real declaration symbol and dispatch receiver class. This symbol can't be bound, so backend can't work with it. To fix it, there is a phase, which replaces them with normal ones after actualization. The symbol is mapped to single one overriding corresponding real symbol within corresponding dispatch receiver class. It can be both a fake override and real declared symbol.
In theory, doing the same with other synthetic declarations (delegated members, data class generated members, etc) can lead to more consistent behaviour, but we don't do it know, as we are not aware of any problems
Lazy classes represent classes used from compiled sources, but not present in them. This can be classes from dependencies, java sources, or sources already compiled on previous round of incremental compilation.
Lazy classes are a special case for fake override building. They are now handled with frontend builder. It can be done correctly, as they can only exist in platform session.
There is a technical issue with them. It's possible to have a lazy class with normal class super-type, so it can refer to a fake override symbol in overridden symbols of some methods. But we can't fix them all in the same place, as normal ones, as it would trigger lazy computations, which should be avoided.
To fix this, the rebuild is delayed while possible, using org.jetbrains.kotlin.fir.backend.Fir2IrSymbolsMappingForLazyClasses, which stores delayed operations to happen, when some read lazy property.
The IR actualizer is a component performing actualization over IR
See: org.jetbrains.kotlin.backend.common.actualizer.IrActualizer
In the current model, IR actualizer is used during the platform compilation, to produce complete IR that is correct from the backend standpoint.
Inputs:
Outputs:
Constraints:
The following actions are preformed:
@OptionalExpectationThere are several restrictions, because of which this order of actions is required:
During the resolution of each module, a special set of measures is implemented to allow proper resolution.
Type refinement is a process of obtaining use-site view to the declaration. Since we re-use FIR of each module during the analysis of its dependants, we need to perform type refinement.
// MODULE: common
expect class Foo
val foo: Foo = TODO()
// declaration-site type: expect class Foo
// MODULE: platform()()(common)
actual class Foo {
fun bar() { }
}
fun test() {
foo.bar() // use-site type: actual class Foo
}
It is possible due to the following principles:
Refinement of a type happens in two stages:
ConeKotlinType.fullyExpandedType(useSiteSession: FirSession): ConeKotlinType
ConeClassifierLookupTag.toSymbol(useSiteSession: FirSession): FirClassifierSymbol<*>
WARNING: Type refinement is also needed for general dependency substitution algorithm, such as classpath order substitution.
It works for expect/actual classes only because actual->expect binding is defined as a matching of ClassId
We need to know a set of arguments that should be provided for a call to resolve it due to the Kotlin resolution and overloading rules.
// MODULE: common
expect fun foo(a: String = "")
expect class Bar {
fun buz(a: String = "")
}
// MODULE: intermediate()()(common)
actual fun foo(a: String) {}
actual class Bar {
fun buz(a: String) {}
}
fun test() {
foo() // use-site
Bar().buz() // use-site
}
In the given example, we need to know actual->expect binding in order to resolve use-site calls.
As during the resolution on the use-site, we will resolve to the actual declarations, we need to obtain its default argument positions from actuals.
We compute that binding by analyzing module with FirExpectActualMatcherTransformer.
See: org.jetbrains.kotlin.fir.resolve.transformers.mpp.FirExpectActualMatcherTransformer
The responsibility of this phase is to perform expect/actual matching.
This phase happens before body/implicit type resolution as the binding is required to perform call resolution in case of defaults
Binding is stored in the FirDeclaration.expectForActual attribute and allow to determine expects that corresponds to the
actual declaration during the analysis of intermediate module.
Hard-constraint:
FirExpectActualMatcherTransformer cannot use return types of declarations to bind actual with expect.
Since the return type of actual function might be not yet resolved before the implicit type body resolve phase.
// MODULE: common
expect fun foo(a: String = "")
// MODULE: platform()()(common)
actual fun foo(a: String) = run {
foo() // In order to resolve this call, we need to have actual -> expect binding
}
Explanation: While it is possible to compute the binding on-demand and remove this constraint, we chose to have it as a separate phase to avoid further complication of body resolve.
There are no overloads by return type in Kotlin, and it makes it possible to avoid return type matching as a part of actual->expect binding construction. However, we still check that return type matches afterward.
Expect-actual matching is performed in the AbstractExpectActualMatcher.
See: org.jetbrains.kotlin.resolve.calls.mpp.AbstractExpectActualMatcher
Matching is a process of finding expect-actual pairs. For classes, it is matching of class-ids. For callables, it is a complex rule, similar to overloading. If declarations don't match, no pair is formed and matching continues over other declarations.
All possible mismatches that can be reported by AbstractExpectActualMatcher are declared in
ExpectActualMatchingCompatibility
Expect-actual checking is performed in the AbstractExpectActualChecker.
See: org.jetbrains.kotlin.resolve.calls.mpp.AbstractExpectActualChecker
Checking is a process of checking that a pair of already matched declarations is correct w.r.t all compatibility requirements. If checking failed for a pair, error is reported for the pair. It will fail compilation.
All possible "checking" incompatibilities that can be reported by AbstractExpectActualChecker are declared in
ExpectActualCheckingCompatibility
Frontend checkers are executed in the context of each module (use-site), after its resolution.
Inputs:
Constraints:
The key limitation is the fact that frontend checkers are run in the context of declaration site.
Thus, it is impossible to observe member scopes of classes with respect to actualization since the corresponding actuals are contained in further modules.
Type checks can also be performed with knowledge available on expect declaration site.
In order to reduce the need to perform additional checks on the backend and in the IRActualizer it is advised to follow the LSP in the design.
By that, meaning that actual declaration must be as compatible with the corresponding expect declaration, as that it could replace the expect declaration, without the appearance of errors.
Note: There are compiler checks that violate LSP to the extent that it is reported on certain leaf types. It is expected that such checks won't trigger in case of actualization.
E.g:
// MODULE: common
expect class E()
fun foo() {
E() // It's OK, since it isn't deprecated in common
}
// MODULE: platform()()(common)
@Deprecated("Not OK")
actual class E actual constructor() {}