docs/language-server/protocol-architecture.md
Enso is a sophisticated language, but in order to provide a great user experience to our users we also need the ability to provide great tooling. This tooling means a language server, but it also means a set of extra peripheral components that ensure we can run Enso in a way that the product requires.
These services are responsible for providing the whole-host of language- and project-level tooling to the IDE components, whether they're hosted in the cloud or locally on a user's machine.
This document contains the architectural and functional specification of the Enso protocol.
For a detailed specification of all of the messages that make up the protocol, please see the protocol message specifications.
<!-- MarkdownTOC levels="2,3,4" autolink="true" -->The divisions of responsibility between the backend engine services are dictated purely by necessity. As multi-client editing necessitates careful synchronisation and conflict resolution, between the actions of multiple clients. This section deals with the intended architecture for the Engine Services.
The engine services are divided into two main components:
Both components will be implemented as akka actors such that we can defer the decision as to run them in different processes until the requirements become more clear.
The project manager service is responsible for both allowing users to work with their projects but also the setup and teardown of the language server itself. Its responsibilities can be summarised as follows:
The language server is responsible for managing incoming connections and communicating with the clients, as well as resolving any potential conflicts between the clients. It is responsible for the following:
It is also responsible for actually servicing all of the incoming requests, which includes but isn't limited to:
It should be noted that the language server explicitly does not talk using LSP. The component is solely responsible for servicing requests, instead of dealing with the minutiae of connection negotiation and request handling.
Additionally, it is very important to note that the language server must not
depend directly on the runtime (as a compile-time dependency). This would
introduce significant coupling between the runtime implementation and the
language server. Instead, the LS should only depend on org.graalvm.polyglot to
interface with the runtime.
This section is partially out of date.
The protocol refers to the communication format that all of the above services speak between each other and to the GUI. This protocol is not specialised only to language server operations, as instead it needs to work for all of the various services in this set.
The protocol we are using intends to be fully compatible with the Microsoft LSP specification (version 3.15). In essence, we will operate as follows:
Aside from the language server protocol-based operations, we will definitely need a protocol extension to support Enso's custom language functionality.
Whatever protocol we decide on will need to have support for a couple of main communication patterns:
There are also certain messages that follow the request/response model but where the responses are trivial acknowledgements. For simplicity's sake these are currently subsumed by the generic request-response model.
As we have decided to remain compatible with LSP, we can use any communication pattern that we desire, either by employing existing LSP messages, or writing our own protocol extensions. Both of the above-listed patterns are supported by LSP.
We can support additional patterns through LSP's mechanisms:
The transport of the protocol refers to the underlying layer over which its messages (discussed in the protocol format below) are sent. As we are maintaining compatibility with LSP, the protocol transport format is already defined for us.
The actionables for this section are:
- Determine the details for the binary WebSocket, including how we want to encode messages and pointers into the stream, as well as how we set it up and tear it down.
Protocol messages are defined by LSP. Any extensions to the messages defined in the standard should use similar patterns such that they are not incongruous with LSP messages. The following notes apply:
This means that we have two pipes: one is the textual WebSocket defined by LSP, and the other is a binary WebSocket.
This entire section deals with the functional requirements placed upon the protocol used by the engine services. These requirements are overwhelmingly imposed by the IDE, but also include additional functionality for the future evolution of the language.
All of the following pieces of functionality that are explained in detail are those expected for the 2.0 release. Any additional functionality beyond this milestone is described in a dedicated section.
The engine services need to support robust handling of textual diffs. This is simply because it is the primary form of communication for synchronising source code between the IDE and the engine. It will need to support the following operations:
Both of these are supported natively within the LSP, and we will be using those messages to implement this.
It should be noted that we explicitly do not intend to handle updates to node metadata within the language server.
didChange message).We place the following requirements upon the implementation of this:
The implementation is as follows:
didOpen, didChange, willSaveWaitUntil,
didSave, didClose, and support for informing the runtime on each of these.Multiple-client support will be implemented while remaining compatible with the LSP specification.
It will work as follows:
didChange message without holding the write lock, it
should receive an applyEdit message that reverts the change, as well as a
notification of the error.applyEdit to synchronise views of the code.One of the most important functionalities for this service set is the ability to manage the state of a project in general. The project state refers to the whole set of the project files and metadata and needs to support the following functionalities:
All file-based operations in the project can be handled by the editor directly, or the language server (when doing refactoring operations), and as such need no support in this section of the protocol.
At the current time, the language server has a 1:1 correspondence with a project. In the future, however, we may want to add LSP support for multiple projects in a single engine, as this would allow users to work with multiple related projects in a single instance of the IDE.
The nature of LSP means that file management and storage is not handled by the language server, and is instead handled by the editor. The protocol makes a distinction between:
The language server must have direct access to the project directory on the machine where it is running (either the local machine or in the Enso cloud), and file operations between the IDE and that machine are handled indepdendently of the language server.
The language server process will need to be able to respond to requests for various kinds of execution of Enso code. Furthermore, it needs to be able to respond to requests to 'listen' to the execution of various portions of code. This implies that the following functionalities are needed:
stdout/stdin/stderr to and from the IDE.All of these functionalities will need to take the form of custom extensions to the LSP, as they do not fit well into any of the available extension points. To that end, these extensions should fit well with the LSP.
A subscription (execution listener) is applied to an arbitrary span of code at a given position in the call stack.
One of the most important elements of execution management for the language server is the ability to control and interact with the execution cache state in the runtime.
foo is used
by bar, then changing foo must recompute bar.The cache eviction strategy is one that will need to evolve. This comes down to the simple fact that we do not yet have the tools to implement sophisticated strategies, but we need to be correct.
b => a, then a change to a must invalidate the cache result of
b.In the future it will be desirable for long running computations to provide real-time progress information (e.g. for training a neural network it would be great to know which epoch is running).
LSP provides an inbuilt mechanism for reporting progress, but that will not work with visualizations. As a result that should be reserved for reporting progress of long-running operations within the language server rather than in user code.
The IDE needs the ability to request completions for some target point (cursor position) in the source code. In essence, this boils down to some kind of smart completion. The completion should provide the following:
import and hits <tab>. This
feature should suggest libraries that are available, along with provide their
top-level documentation to give users an idea of what they can be used for.Hints should be gathered by the runtime in an un-ranked fashion based upon the above criteria. This will involve combining knowledge from both the compiler and the interpreter to deliver a sensible set of hints.
5,
foo : 5 -> String scores higher than bar : Nat -> Dynamic, scores higher
than baz : Any -> Any. This should be done by heuristics initially, and
later by querying the typechecker for subsumption relationships (the notion of
specificity discussed in the types design document).tags section of the documentation should also
be used to rank candidates.From an implementation perspective, the following notes apply:
completion and completionResolve
messages provided by the LSP spec.We also want to be able to support a useful set of semantic analysis operations to help users navigate their code. As these rely on knowledge of the language semantics, they must be explicitly supported by the language server:
applyEdit message to ask
the IDE to insert an import for the symbol specified in the request. If the
file is closed, then the edit should be made directly, as the LSP specifies.In addition to the functionality discussed in detail above, there are further augmentations that could sensibly be made to the Engine services to support a much better editing and user-experience for Enso. These are listed briefly below and will be expanded upon as necessary in the future.
didChange and
applyEdit messages to reconcile all clients' views of the files. This is
also why willSaveWaitUntil is important, as it can ensure that no client
editor saves until it has the authority to do so (all changes are reconciled).The binary protocol refers to the auxiliary protocol used to transport raw binary data between the engine and the client. This functionality is entirely extraneous to the operation of the textual protocol, and is used for transferring large amounts of data between Enso components.
As the protocol is a binary transport, it is mediated and controlled by messages that exist as part of the textual protocol.
In order to deserialize a family of messages and correlate responses with requests, each request/response/notification is wrapped in an envelope structure. There is a separate envelope for incoming and outgoing messages:
namespace org.enso.languageserver.protocol.binary;
//A mapping between payload enum and inbound payload types.
union InboundPayload {
INIT_SESSION_CMD: InitSessionCommand,
WRITE_FILE_CMD: WriteFileCommand,
READ_FILE_CMD: ReadFileCommand
}
//An envelope for inbound requests and commands.
table InboundMessage {
//A unique id of the message sent to the server.
messageId: EnsoUUID (required);
//An optional correlation id used to correlate a response with a request.
correlationId: EnsoUUID;
//A message payload that carries requests sent by a client.
payload: InboundPayload (required);
}
namespace org.enso.languageserver.protocol.binary;
//A mapping between payload enum and outbound payload types.
union OutboundPayload {
ERROR: Error,
SUCCESS: Success,
VISUALIZATION_UPDATE: VisualizationUpdate,
FILE_CONTENTS_REPLY: FileContentsReply
}
//An envelope for outbound responses.
table OutboundMessage {
//A unique id of the message sent from the server.
messageId: EnsoUUID (required);
//An optional correlation id used to correlate a response with a request.
correlationId: EnsoUUID;
//A message payload that carries responses and notifications sent by a server
payload: OutboundPayload (required);
}
namespace org.enso.languageserver.protocol.binary;
//This message type is used to indicate failure of some operation performed.
table Error {
//A unique error code identifying error type.
code: int;
//An error message.
message: string;
}
//Indicates an operation has succeeded.
table Success {}
The binary protocol currently only supports a single type of communication pattern:
The binary protocol uses flatbuffers for the protocol transport format. This choice has been made for a few reasons:
The binary protocol exists in order to serve the high-bandwidth data transfer requirements of the engine and the GUI.
A major part of Enso Studio's functionality is the rich embedded visualizations that it supports. This means that the following functionality is necessary:
Visualizations in Enso are able to output arbitrary data for display in the GUI, which requires a mechanism for transferring arbitrary data between the engine and the GUI. These visualizations can output data in common formats, which will be serialised by the transport (e.g. text), but they can also write arbitrary binary data that can then be interpreted by the visualization component itself in any language that can be used from within the IDE.
From the implementation perspective:
As these services need to support multiple clients in future, there is some rigmarole around setting up the various connections needed by each client. The process for spawning and connecting to an engine instance is as follows:
session/initProtocolConnection
for more information.session/initDataConnection
below more information.As the engine performs sophisticated caching and persisting of data where possible, it is very important that the client informs the engine of the end of its session. In contrast to the initialisation flow above, this is not an involved process.
session/end to the server.