Before: One input, one output​

Until now, a transformer would take one log record as input and return one log record as output. That works well for parsing, enriching, or reshaping logs.

But real logs are not always that simple.

Sometimes one incoming log line actually contains several events (like in OpenTelemetry when batching multiple logs). Sometimes, we want to send different information from a single log to different services (metrics to a timeserie database, user information to an audit tool, ...).

After: One input, many outputs​

With this release, a transformer can return an array. Each item in that array becomes its own log record.

For example:

. = [
  1,
  2,
  "hello",
  { "foo": { "bar": "baz" } }
]

The above script would produce the following logs:

{"value": "1"}
{"value": "2"}
{"value": "hello"}
{"foo.bar": "baz"}

NB: All data types are normalized and flattened to fit the datamodel of FlowG.

Each generated log is passed to the next nodes in the pipeline.

Simplicity as a consequence​

This feature makes pipelines easier to design. Instead of forcing every downstream node to understand a large nested payload, you can split the payload early and let the rest of the pipeline work on simpler records.

As a result:

• filters can operate on one event at a time
• routing rules become easier to write
• storage output is more predictable
• forwarding to third-party systems becomes more natural

The pipeline no longer has to carry a batch-shaped record when what you really want is a stream of individual events.

A foundational change for the future​

On the technical side of things, this feature introduce some foundational changes to how VRL programs are called by FlowG.

As a reminder, FlrowG is implemented in Go, but the VRL library is implemented in Rust. To make both talk to each other, it requires the C FFI.

Before the v0.55.0 release, the communication went like this:

• convert a Go map[string]string into a C hmap (homemade data structure), lots of allocations done to copy the data
• convert the C hmap to a Rust HashMap<String, String>, lots of allocations done to copy the data (again)
• convert the HashMap<String, String> to a vrl::value::ObjectMap (why the HashMap intermediate step? because I did not think it through) wrapped in a vrl::value::Value (the input for a VRL program)
• convert and flatten the resulting vrl::value::Value of the VRL program to a HashMap<String, String> (more allocations, more copies)
• convert the HashMap<String, String> to a C hmap (here, we actually move the data! to be then freed by Golang's allocator, which in retrospective might not be a good idea)
• convert the C hmap to a Go map[string]string (more allocations, more copies)

Not only there are way more allocations and copies than needed, but there is a dubious memory ownership model, and a not very flexible API.

In the future, we might want to support more than just logs, which means the actual inputs to the VRL program might evolve.

That's when I noticed, the vrl::value::Value is actually (de)serializable using Rust's great library serde.

If we use MessagePack to (de)serialize our events and send through the C FFI boundaries only a byte array, this might simplify everything:

• avoiding unnecessary allocations and copies
• make the VRL program runner completly agnostic of the data shape
• cleaner memory ownership model

The flow becomes:

• serialize the Go map[string]string to a []byte buffer (the buffer is cached to avoid allocating it for every log)
• convert the Go []byte buffer to a C uint8_t* pointer and a size_t length (no copy needed, memory is owned by Go)
• convert the pointer and length to a Rust &[u8] (no copy needed, memory is still owned by Go)
• deserialize the &[u8] directly to a vrl::value::Value (allocations and copies unavoidable here)
• call the VRL program
• serialize the resulting vrl::value::Value to a Vec<u8> buffer (the buffer is cached to avoid allocating it for every log)
• convert the Vec<u8> buffer to a C uint8_t* pointer and a size_t length (again, no copy needed, memory is owned by Rust)
• convert the C pointer and length to a non owning Go []byte
• deserialize the []byte to any, convert and flatten to map[string]string (allocations and copies unavoidable here)

Conclusion​

That's all folks! Conclusions are hard to write, so stay tuned for more content.

\_o< {quack}

Minimum Viable Product​

For the MVP, we wanted to have at least the following features:

• ability to ingest and store logs
• interoperability with many third party services, done via Forwarders
• basis of clustering with the most common cluster formation strategies

After months of work, this is now done. Special thanks to our main contributors, without who we would not be here today:

• @glooopsies
• @Minnerlas
• @atpugtihsrah
• @n4vxn

FlowG has all you need to make sense of the logs of your production environments, and we're quite active on the bug tracker, so don't hesitate to ask any question, or report eventual problems.

There is still a lot of work to be done, and it's all going towards the 1.0, first "stable" version.

Road to 1.0​

What we call "stable" here means that the API will be locked down, no breaking change guaranteed.

We have 2 big projects for this, and probably a few more down the line that are not planned yet, or not even fleshed out yet:

Frontend Redesign​

Historically, the frontend was made with Templ, HTMX, and Tailwind. Due to the difficulties of handling the amount of client-state we had, and keeping the API and UI features in sync, we decided to migrate to a React SPA with React MUI.

Unfortunately, lots of Tailwind code remained, and the result is a mix of 2 conflicting design systems, leading to some visual glitches and inconsistencies that are hard to debug and/or fix.

A proper redesign is waranted:

• removing Tailwind and fully adopting MUI
• reorganizing the code to be more consistent and following best practices
• documenting the architecture to facilitate onboarding new contributors

Over the next few months, this work is gonna be done by @coyote2190, our most recent contributor. Special thanks to him as well!

Replication​

We currently have an experimental implementation of the replication layer, allowing FlowG to be a highly available service.

This implementation is buggy, costly in terms of performance, and wrong on many levels.

A complete redesign is also planned for the next few months. We'll keep using the SWIM Protocol, via Hashicorp Memberlist Go package, for the automatic cluster formation. But instead of relying on the costly "TCP Push/Pull" mechanism to synchronize storages, we'll implement a proper Operation Log, and synchronize changes via a custom workflow, separate from the SWIM Protocol.

As for the replication model, we still want eventual consistency, that hasn't changed. Only the implementation will change, and be correct.

Correctness will be proven with an exhaustive test suite, trying to ensure that most edge cases are covered.

Testing the chaos​

We want to prove that FlowG can perform well in production environments, where many things can go wrong. To do so, we will continuously improve the test suite and set up test environments, following the principles of Chaos Engineering.

Traces and Metrics with OpenTelemetry?​

At the moment, FlowG only supports logs. The OpenTelemetry integration also only supports those.

But OpenTelemetry also has "metrics" and "traces", which are essential parts of any observability solutions. It is not yet clear if we want that in FlowG or not, there is an ongoing discussion about it on Github, and would appreciate any feedback on this.

Nothing has been fleshed out yet for this. It might not be part of the 1.0 release, which is fine because it does not look like that introducing this feature would be a breaking change.

Conclusion​

The list of contributors keeps growing, and you have all our thanks for that!

Any feedback is welcomed, feel free to join our community either on the Github or on Discord.

If you are using FlowG in production, give us a ping, we'll add you to our README.

To infinity and beyond!

Introduction​

FlowG's goals have always been: interoperability and ease of use. Many applications already use the "ELK" stack:

• ElasticSearch for indexing
• Logstash for aggregation
• Kibana for viewing

Usually, logs are sent to Logstash via Syslog, which then forwards them to ElasticSearch for storage and indexing.

FlowG already could be set up as a drop-in replacement for Logstash thanks to its Syslog Server endpoint, and its multitude of forwarders.

But with the latest release, we're taking things up a notch with the partial support for ElasticSearch API.

What does this mean?​

FlowG exposes on the /api/v1/middlewares/elastic endpoint an API compatible with ElasticSearch, allowing you to plug your existing application, which uses the ElasticSearch client libraries, into FlowG, without changing your code.

⚠️ NB: The support is only partial.

At the moment, only 2 operations are supported:

• HEAD /{index}: check if an index exists
• POST /{index}/_doc: index a document

And only HTTP Basic authentication is supported.

Indexes map to FlowG pipelines, the following request would send the document through the default pipeline:

POST /api/v1/middlewares/elastic/default/_doc
Authorization: Basic ...
{"foo": {"bar": "baz}}

NB: In FlowG, the datamodel is flat, so the document is flattened by the API before handing it to the pipeline, this would be equivalent to:

{"foo.bar": "baz"}

What now? The roadmap​

More operations might be added later on to the "compatibility API" to smooth out the integration of FlowG into your existing infrastructure, though the goal is not to have 100% feature parity (only the subset that makes sense to support in FlowG).

But also, more APIs might come in later, depending on user feedback/requests.

Summary​

As of FlowG v0.45.0, logs can be ingested via:

• HTTP as text data (one log per line) on /api/v1/pipelines/{PIPELINE}/logs/text
• HTTP as JSON data on /api/v1/pipelines/{PIPELINE}/logs/struct
• HTTP as OpenTelemetry data on /api/v1/pipelines/{PIPELINE}/logs/otlp
• Syslog protocol
• ElasticSearch REST API on /api/v1/middlewares/elastic/{PIPELINE}/_doc

See the release notes