Improved Multiline Support In The File Source

One of the biggest frustrations we've heard from users in this space is the inability to merge lines together. Such a simple task can be incredibly complex and hard. Fear not! We plan to add first-class support for solving this problem.

In addition to the recently added automatic merging of Docker logs, we also added better multi-line support to our file source. These options are very expressive and should solve the vast majority of multi-line merging problems.

For example. Given the following lines:

foobar.rb:6:in `/': divided by 0 (ZeroDivisionError)
  from foobar.rb:6:in `bar'
  from foobar.rb:2:in `foo'
  from foobar.rb:9:in `<main>'

You can merge them with the following config:

sources:
  my_file_source:
    type: "file"
    # ...
    multiline:
      start_pattern: "^[^\\s]"
      mode: "continue_through"
      condition_pattern: "^[\\s]+from"
      timeout_ms: 1000

And if this doesn't do it, you can always fall back to the lua transform.