docs/USER_EXPERIENCE_DESIGN.md
Vector places high value on its user experience; our goal is to make this a strong differentiator. But this presents a challenge for Vector, which has an inherent large surface area, requiring a large team. As a result, definitions of good user experience differ, contributors have varying levels of expertise in this area, and coordination suffers. To solve this, we must align behind a set of principles, guidelines, and processes that unify us towards a shared vision of good user experience -- the purpose of this document.
<!-- MarkdownTOC autolink="true" style="ordered" indent=" " --> <!-- /MarkdownTOC -->By nature of Vector's design, a data processing swiss army knife of sorts, we often get presented with many different creative use cases -- using Vector for analytics data, obscure IOT pipelines, synchronizing state between systems, or adding new languages for data processing. It's tempting to entertain these use cases, especially if a large, valuable user is inquiring about them, but we should refrain from doing so. Trying to be everything to everyone means we'll never be great at anything. Disparate use cases often have competing concerns that, when combined, result in a lukewarm user experience. Vector is an observability data pipeline, and we strive to be the best in this domain.
Examples:
As an extension of the previous principle, aligning on specific use cases allows us to make assumptions about the user and be opinionated with solutions. This approach reduces the number of decisions a user has to make and follows Hicks Law, a hallmark of a good user experience. Therefore, as much as possible, we should offer opinionated models for use cases core to Vector's purpose and not leave them as creative exercises for the user.
Examples:
pipelines transform as a solution to team collaboration as opposed
to generic config files.Consistency builds momentum by reducing the cognitive load imposed on a user
as they navigate a product. Users carry patterns from previous areas into new
ones; deviation from these patterns introduces unnecessary friction. Consistency
manifests itself in small details, like naming, and larger details, like product
behavior. For example, it would be surprising to find a codec option in one
source and a decoder option in another if they are functionally equivalent.
Even more surprising is a deviation in runtime behavior, like back pressure or
error handling, since this often results in data loss.
Examples:
codec option name in both sources and sinks that support it.Design goals are explicit factors of user experience that are prioritized in all decisions. Other aspects of user experience, like complete functionality or engineering purity, should be sacrificed to optimize these goals.
The combination of extremely high data volumes and complex processing needs makes performance a steep challenge for observability data pipelines. Chasing functionality without considering performance at every turn can quickly result in a web of irreversible performance problems. Therefore, Vector upholds performance as its primary design goal to maintain our competitive advantage in this area.
Examples:
Benchmark performance does not always translate to real-world performance, and to deliver the representative real-world performance, we must make Vector safe to operate. This enables not only performance but also the reliability required by a critical infrastructure tool like Vector. Therefore, safety is Vector's secondary design goal.
Examples:
The following guidelines help to uphold the above principles.
Logs should be flexible, fluid data structures that do not impose a rigid schema for the following reasons:
In general, we believe Vector's flexible nature with log schemas is a boon for Vector's UX. A rigid schema is not required to achieve a best-in-class UX.
When presented with the choice to trade data loss for another benefit, like performance, Vector should retain data by default. Data loss is unacceptable for most Vector users and trading it for benefits like performance should be an opt-in choice by the user.
Examples:
Vector sources and sinks should be plentiful and specific. When possible, they should align with a well-defined protocol. If a well-defined protocol is not available, or the use case does not allow it, then the source or sink should align with its upstream client or target service. Additionally, sources and sinks can overlap. It is preferable to offer specific sources and sinks composed with broad sources and sinks. This promotes discoverability, aligns with user intent, and builds confidence that Vector meets their specific use case. Ideally, Vector would offer hundreds of sources and sinks that specifically integrate with various systems. Maintainability should be achieved through composability.
Examples:
syslog source as opposed to a syslogng source since it aligns with the
Syslog protocol.datadog_agent source as opposed to a datadog_api source since it aligns
on intent and reduces scope.syslog source in addition to a socket source with the codec
option set to syslog since it is more specific and discoverable.As opposed to source and sinks, Vector transforms should be broad and minimal. They should align with a high-level use case, like mapping, aggregation, or routing. The goal here is to reduce the number of choices a user makes and be opinionated with solutions. Our opinions should prioritize performance and reliability over ease of use (within reason), but we should strive to achieve both.
Examples:
remap transform as opposed to multiple parse_json, parse_syslog, etc
transforms.filter transform as opposed to a filter_regex, filter_datadog_search,
etc. transforms.Vector should never require manual intervention to remedy normal processing failures at runtime. Instead, Vector should require any necessary configuration to handle these failures a priori.
For example, rather than requiring users to intervene to unblock processing whenever events fail to be sent to sinks by manually skipping them, instead Vector should let users define dead-letter queues to send events to when the primary sink rejects events.
Users may need to intervene to update invalid configuration, for example in the case of rotating invalid API keys.
When encoding logs for sinks, fields indicated by log_schema keys should be mapped to where the
destination expects to find them in the encoded payloads. For example, if the sink requires
a "hostname", it should fetch it from the log event using LogEvent#host_path to fetch the value
from the appropriate field and place it in the encoded payload where the "hostname" is expected.
To keep things simple, only two roles are involved in the context of Vector's user experience.
Anyone contributing a change to Vector, both public open-source contributors and internal Vector team members.
A select group of Vector team members responsible for Vector's resulting user experience.
As a Vector contributor you are responsible for coupling the following non-code changes with your code changes:
/website/cue folder
(generally configuration changes)/website/content folderYou are not responsible for:
As a user experience design committee member, you are responsible for: