Compiler Reference Index

Compiler reference index service – is a service that allows to reduce the search scope of a declaration usages for compiled projects.

Basis: a large codebase with many declarations and usages with common names like getInstance
Action: find usages of some getInstance declaration
Problem: a lot of false positive candidate files caused by text search
Solution: extract usage information from the compiler to reduce the search scope

What should I do to get the maximum benefit of the index?

Compile project before searching.

How can I check the benefit of the index?

We have Show Compiler Index Status internal action to check the difference between Use Scope and Code Usage Scope.
Q: How to use it?
A: Put the caret at the declaration and call the action

Top level API

• PsiSearchHelper#getCodeUsageScope
  • KotlinScopeOptimizer and JavaCompilerReferencesCodeUsageScopeOptimizer under the hood

Structure questions

Who writes the index?

• Kotlin – the Kotlin compiler by IncrementalJvmCache
• Java – the JPS javac extension by JavaBackwardReferenceRegistrar

Who reads the index?

• Kotlin – KotlinCompilerReferenceIndexStorage
• Java – BackwardReferenceReader

Which module is compiled?

The producer on the JPS plugin side

• Kotlin – KotlinCompilerReferenceIndexBuilder
• Java – JavaBackwardReferenceIndexBuilder

The consumer on the IDE side

• Kotlin – DirtyScopeHolder by KotlinCompilerReferenceIndexService
• Java – DirtyScopeHolder by CompilerReferenceServiceImpl

Service lifecycle

Project startup

1. Preload CompilerReferenceServiceImpl and KotlinCompilerReferenceIndexService on the project startup
   1. Subscribe on the VFS events to track changed files
   2. Subscribe on BuildManagerListener to handle the start and the end of a build
   3. Subscribe on PortableCachesLoadListener to drop the index to avoid JPS cache corruption
2. Schedule CompilerManager#isUpToDate check by IsUpToDateCheckStartupActivity as backgroundPostStartupActivity
   1. The check will be performed only if Kotlin and/or Java can already have the index on the filesystem
   2. If the check returns true then the corresponding service will be marked as up to date (Kotlin, Java)

Build started

• Increase activeBuildCount
• Close the index storage if it is open

Build finished

• Decrease activeBuildCount
• Remove compiled modules from the dirty scope
• If activeBuildCount == 0 then open the index storage

Dirty scope

The dirty scope operates with modules. All modules are dirty be default (except for up to date case).
We should track changed files to provide the correct search scope. A change in the file causes the current and all dependent modules to be marked as "dirty".
The scope also contains modules excluded from compilation.

How to calculate the search scope?

The result search scope should contain files from:

1. the compiler index
2. the dirty scope
3. other languages
4. libraries if we should search in the libraries scope

The sources: Kotlin / Java.

What is next?

• KTIJ-19973 – the optimized scope should include the places of declaration definitions as well
  • Motivation: this is required to optimize the search of declaration definitions like overrides
• IDEA-281133 – Rewrite CompilerReferenceService and KotlinCompilerReferenceIndexService into one service
  • Problem: we have two related and similar services