docs/site/Crafting-LoopBack-4.md
LoopBack is an open-source Node.js framework built for API developers. Its primary goal is to help create APIs as microservices from existing services/databases and expose them as endpoints for client applications, such as web, mobile, and IoT. LoopBack connects the dots between accepting API requests and interacting with backend resources. By facilitating developers to implement API logic with out of box integration capabilities, LoopBack establishes itself as the API composition layer to differentiate from other frameworks, such as Express, Hapi, and Sails.
Up to version 3.x, LoopBack built on the popular Express framework. In retrospect, basing LoopBack on Express was the right decision: Standing on the shoulders of Express enabled LoopBack to focus on adding value for API creation experience without reinventing the wheel. LoopBack also has benefitted from the Express ecosystem, especially ready-to-use middleware modules from npm as well as valuable knowledge and support by the community.
With LoopBack, developers can create and expose APIs just like following a recipe. LoopBack introduces a set of core concepts that represent the key aspects of API implementation. To create APIs out of existing databases or services, developers can simply scaffold a LoopBack application, then add necessary JSON declarations and Node.js code to get their APIs up and running in a few minutes.
LoopBack uses Express routing and middleware as the plumbing to a request/response pipeline for API use cases, such as authentication, authorization, and routing. Beyond inbound HTTP processing, LoopBack provides integration facilities such as models, datasources, and connectors to allow API logic to interact with various backend systems, including but not limited to, databases, REST APIs, SOAP web services and gRPC microservices. The ability to glue inbound communication and outbound integration makes LoopBack a very powerful framework for API developers. The diagram below illustrates how LoopBack fits into a typical end-to-end API processing flow.
LoopBack has grown significantly in features and users with many years of development and multiple releases. LoopBack has been well-received by the developer community. As an indication, the community has developed many extensions. The core team has also learned a lot from what we have done as well as great feedback from the community.
Like many projects, LoopBack has started to experience growing pains, especially as:
The code base becomes more complicated over time with more modules and more functionality. We would like to have more maintainers and contributors to help out. But the learning curve is getting steep. One of the things to blame is JavaScript itself, which is weakly-typed and lack of constructs such as interfaces to explicitly define contracts between code. There is quite a bit hidden knowledge that is not explicit or obvious for new comers.
Technical debt is accumulating, for example inconsistent designs across modules and feature flags for different behaviors. Here are a few examples:
It is becoming more difficult to add new features or fix bugs as some areas start to reach the limit of the current design.
loopback-datasource-juggler module is a kitchen sink for many
things, such as typing, data modeling, validation, aggregation,
persistence, and service integration.It's not easy to extend the framework without requesting the core team to make code changes in LoopBack modules. The current version of LoopBack has ad-hoc extensibility at various layers. Extension points are not consistently defined. For example,
More projects start to use LoopBack as the underlying platform. Such use cases require more knowledge of LoopBack internals and more flexibility and dynamicity to leverage LoopBack to manage and compose artifacts using a metadata driven approach. Some good examples are:
Since the release of 3.x, the team has been brainstorming about how to sustain and advance LoopBack. We did a lot of homework, triaged existing GitHub issues, reached out to community members and downstream products, and evaluated relevant frameworks and technologies to answer to the following questions:
LoopBack has gained traction among a spectrum of users beyond Node.js application developers, including:
The core team decided to make a bold move and rebuild LoopBack to meet the needs of all the above groups. The decision led to the inception of LoopBack 4, a new generation of API creation platform. For more information, read the blog post Announcing LoopBack.next, the Next Step to Make LoopBack Effortlessly Extensible.
LoopBack 4's goals are:
Catch up with latest and greatest technology advances.
Promote extensibility to grow the ecosystem.
Align with cloud native experience for microservices.
Remove the complexity and inconsistency across modules.
Separate concerns for better composability.
We decided not to take a "big-bang" approach to build LoopBack 4. Instead, we are doing it incrementally in multiple stages with smaller steps. This approach allows us to better engage the community from the beginning. We are following the principles below to pursue architectural simplicity and extensibility:
Imperative first, declarative later
Everything can be done by code via APIs. The LoopBack team or community
contributors can then create varieties of user experiences with such APIs.
For example, with APIs to define models, we allow applications to declare
models in JSON or YAML files so that they can be discovered and loaded. An
extension can parse other forms of model definition, such as JSON schemas,
ES6 classes with decorators, schemas in OpenAPI spec, or even XML schemas
into LoopBack model definitions.
We can also leverage programming constructs such as decorators allow developers to supply metadata in code. Furthermore, LoopBack artifacts can be declared in JSON or YAML files, which will be handy for users to generate and manipulate them by hand or tooling.
Build minimum features and add more later if necessary
Apply YAGNI (You Aint’t Gonna Need It). Design and build for what is needed now, not for what you think you may need in the future. There are many different perspectives in API creation and people ask for a lot of features. Starting with MVP allow us to reach the root of the issues without being derailed by noises and build the absolutely necessary features as the core building blocks.
Developer experience first
Always keep in mind that LoopBack is built for developers by developers. Our first priority is to make API developers' life easier. When we design APIs and user interfaces such as a CLI or GUI, we want to make sure they are intuitive to and natural to their thought process.
Here are the stages we are marching through toward the final version of LoopBack 4 as illustrated below.
Rebase and rewrite the core
Leverage TypeScript for better code quality and productivity.
Unify the asynchronous programming model/style.
Implement an IoC Container for better visibility and extensibility
Introduce Component as packaging model for extensions
Validate the core design by implementing an REST/HTTP invocation chain
Add top-down REST API creation which starts with OpenAPI specs.
Build sequence of actions for inbound http processing
Introduce controllers as entry points for API-related business logic.
Models are the centerpieces of the current LoopBack applications. . They take multiple responsibilities:
Authentication as a component
Implement the core functionality of authentication as a component, which includes:
authenticate action to handle authenticationRebuild our integration and composition capabilities
Declarative metadata and bootstrapping
LoopBack manages a set of artifacts, such as models, relations, datasources, connectors, ACLs, controllers, repositories, actions, sequences, components, utility functions, and OpenAPI specs. In addition to the programmatic approach to describe these artifacts by code (apis and decorators), we would like to add declarative support so that they can be declared in JSON/YAML files.
Tooling (CLI & UI)
Enable cloud native experience
The following diagram illustrates the high-level building blocks of LoopBack 4:
Please note there is a common layer below the different functional areas in the stack. Let's examine the need to build a new core foundation for LoopBack 4.
LoopBack itself is already modular. For example, a typical LoopBack 3.x application's dependency graph will have the following npm modules:
LoopBack manages various artifacts across different modules. The following are a list of built-in types of artifacts that LoopBack 3.x supports out of box:
Metadata for these artifacts form the knowledge base for LoopBack to glue all the pieces together and build capabilities to handle common API use cases.
How to represent the metadata and their relations is the key responsibility of the LoopBack core foundation. It needs to provide a consistent way to contribute and consume such building blocks.
The core foundation for LoopBack 4 is responsible for managing various artifacts independent of the nature of such artifacts.
A consistent registry to provide visibility and addressability for all artifacts.
Visibility: Each artifact has a unique address and can be accessed via a URI or key. Artifacts can also be visible at different scopes.
Extensibility: LoopBack artifacts can be managed by types. New artifact types can be introduced. Instances for a given type can be added, removed, or replaced. Organizing artifacts in a hierarchy of extension points/extensions decouples providers and consumers.
Ability to compose with dependency resolution.
A packaging model for extensions.
LoopBack had always been built on Express so we can leverage the vast community and middleware in the Express ecosystem BUT it presented some challenges for LoopBack. With LoopBack 4 we considered moving away from Express (and even built the framework without Express) but eventually circled back to Express because of its vast ecosystem.
Some of the gaps between what Express offers and LoopBack's needs are:
Express is only extensible via middleware. It neither exposes a registry nor provides APIs to manage artifacts such as middleware or routers.
Express is not composable. For example,
app.use()is the only way to register a middleware. The order of middleware is determined by the order ofapp.use.
In Express, everything is done by JavaScript ... In contrast, LoopBack is designed to facilitate API creation and composition by conventions and patterns as best practices.
The main purpose of LoopBack is to make API creation easy, interacting with databases, services, etc., not middleware for CORS, static file serving, etc. We didn't want to reinvent the wheel by writing new middleware for LoopBack 4.
The team explored leveraging Express or Koa (but only for their middleware support). The final decision was to use Express in a way that bridges the gap by addressing the gaps identified above as follows:
@loopback/context as a registry.You can learn more details in our blog post on improving inbound http processing.
There are several key pillars to make extensibility a reality for LoopBack 4.
Please check out Extending LoopBack 4.
With the extensible foundation in place, we start to rebuild the LoopBack REST API experience by "eating your own dog food" with the following artifacts:
The features are provided by the following modules:
Before we go further, let's try to build a 'hello world' application with LoopBack 4.