Introducing SAFE EventStore (event sourcing database)


#1

This is an open sourced database functionality on top of the SAFE Network.
The code base is a library that can be referenced in a .NET project.
The functionality is currently absolutely basic and simplistic, just to get started.

I have been developing event sourced applications in my job, and this is an implementation using SAFE Network as backing storage.
The goal is that .NET developers interested in EventSourcing could be introduced to SAFE Network via this.
Since I myself have been very excited about the network capabilities and what it means in an event sourcing context, I figured many others would be too.

The project includes:

  • .NET wrapper over SAFE FFI (alpha-2). (Almost done, needing some fix to run with alpha-2)
  • EventStore library using the wrapper. (First version done)
  • EventStore db management UI. (cross platform web app) (Simple test version done)

The current mock version can be used to:

  • List available databases
  • List stream categories in each database
  • List stream keys (i.e. stream instances) in each category

I am currently waiting for some support on some ffi-related things needed for the .NET wrapper, to be able to continue testing and refine the actual implementation using alpa-2 network.

UI developer
I am additionally opening up for people who would like to collaborate on designing a SAFE Website (with react+redux for example) to use as the DB management UI, instead of the current one that I made from template (served by the localhost server).
This will give a more cohesive and natural experience as a user can access and view their databases from any SAFE Browser, which then accesses the database service (cross platform) running locally.
Write here or send me a PM if you are interested in participating.

How to use it
A developer that decides to use event sourcing as a database solution - for parts of, or entire applications - can reference the SAFE.EventStore library in their .NET project, and have a complete framework for storing and retrieving data as events in streams.
To have a graphical UI for the DB management, they can access this from the localhost webapp.

Background
Event sourcing is a different database structure than for example relational, document or graph databases.
All databases come with their benefits and downsides, and event sourcing is for example well suited for IOT applications, or financial applications. But the uses are of course uncountable.

A common practice to use together with event sourcing is something called CQRS, which stands for Command and Query Responsibility Segregation (minted by Greg Young, based on the older CQS)
This is because event stores are not well suited for querying data, and for that reason read models (projections) are created from the event streams, hence the CQRS, as commands will lead to events being written into streams, while queries are performed on the projections.

For this reasons I have also created a SAFE.CQRS library (also with DDD concepts included), which can be used by those who do not wish to create their own boiler plate code.
(Greg Young often states though, that a framework for this is not needed. Every application has its own needs. But the opinions are divided in this matter! :slight_smile: )

Next steps
As the .NET wrapper is completed so that it can be used to connect to alpha-2 network, the event sourcing protocol will be tested and refined, as to improve performance and stream line the handling.
As this is done, the code will be published to GitHub for developers to clone.
After that, some additional information will be posted to google groups of EvenStore, CQRS and DDD enthusiasts, as to introduce them to SAFE Network.

Code for building projections, routing cmds, setting up process managers/sagas (CQRS specific state machines) is part of the SAFE.CQRS library and will be added too, for those who do not whish to write their own versions.
The event sourcing protocol will be used by me when I write other event sourced apps for the network, so it will be continuously updated and refined.

I hope this project will bring some more attention to SAFE Network from developers, and add to the Apps being developed for the community :slight_smile: !

EDIT:
2017-10-12 Update

I figured I’d just as well put the mock demo out there :smile: so here’s the repo: https://github.com/oetyng/SAFE.EventStore

Note: The basics of the actual EventStore implementation is there too, but I have not been able to test it on the network, and it is very likely, that something is off with regards to the MD operations etc.
This is my first try of coding something for the network. :slight_smile:


SAFE Crossroads Podcast #38, Event Sourcing on the SAFE Network
Using Maidsafe libraries for my app outside the "ecosystem"
For C# developers: SAFE.DataStructures
Can a c# .net application be deployed to the network?
#2

Ooo - exciting. Is there a link to the project?


#3

So far the wrapper is here:

I did first intend to not publish the SAFE.EventStore project + SAFE.EventStore.UI until I had gotten the wrapper completed (waiting for some support regarding the ffi), so that I can test the real implementation.

But I might go ahead and let the mock version be tested if it cannot be solved quickly :slight_smile:


#4

Really cool. I’ve been thinking of making something somewhat similar myself, more specifically a library for storing entities as Mutable Data where every time you change the entity it creates an Immutable Data containing an event for the change where the events are things like { «UpdateProperty» : «Name», NewValue : «Bob»}, haven’t really had time to get started yet though. It will be interesting to follow your progress with this.


#5

For anyone wanting to know what Event Sourcing is about I found this which is a good intro:

http://cqrs.wikidot.com/doc:event-sourcing


#6

Thanks @happybeing I was also having a quick mooch around the interwebs. This is seems useful, although I’ve only skimmed it so far

https://msdn.microsoft.com/en-us/library/jj591559.aspx


#7

Thanks, @happybeing and @JPL for the references. They helped a lot.

@oetyng, this is a great project! Such possibilities are starting to get very exciting.


#8

Nice one @JPL

From my brief read I can see ES and SAFE working well together, so thanks @oetyng! I too look forward to seeing how this proceeds :slight_smile:


#9

@intrz, that’s a cool idea!

@JPL & @happybeing, thanks for those links! That was missing :slight_smile:

@fergish, thank you, nice to hear! :slight_smile:

And to everyone, I figured I’d just as well put the mock demo out there :smile: so here’s the repo: https://github.com/oetyng/SAFE.EventStore

Note: The basics of the actual EventStore implementation is there too, but I have not been able to test it on the network, and it is very likely, that something is off with regards to the MD operations etc.
This is my first try of coding something for the network. :slight_smile:


#10

Just listened to the safe crossroads podcast and just wanted to say I appreciated it!


#11

Btw, something I noticed closer to the circles I develop in - Apache Kafka seems to be targeting event sourcing in a recent presentation I watched. The idea that the message stream itself becomes the database, rather than it feeding others, seems to be an interesting correlation.

Perhaps as applications are breaking down into smaller microservices, thus becoming more dependent on event streams, is pushing technology like this forward. I think this bodes well for safe net and the overlay networks that provide a simple interface for this.


#12

Listening to a podcast about Urbit. Your EventStore database sounds like just the thing they need for their system. My understanding is that Urbit is a VM that records the history of events such that the current state is computed based on past events. They mentioned backing up the event log on different cloud providers for redundancy. Your database seems like a better solution. https://urbit.org/posts/objections/#log

Edit: actually, I think they already have their own storage, so probably disregard. I’m having a pretty hard time trying to understand their system.


#13

great job that must have taken a lot of dedication!


#14

I have been back to this little project now to add another protocol for the event store.

The first protocol was basically doing this:

Created a database object to store serialized in the app container.
Add category entries to this database object, where category name (stream name) is key and an xor address to an MD is the value.
Each category MD has individual stream keys (stream name + stream id) as key, and xor address of the stream MD as value.
Each stream MD has event batch sequence range as key and the serialized events as value. If the stream exceeds the MD size or entry count, it will link to (or use hash of its own id) to next MD.
(An event batch is a sequence of events raised as a result of a single cmd . Each event in the sequence has a sequence nr in the stream.)

(Instead of adding a database object with categories to the app container, a proper MD for a database, (hash name of database to get MD xor address), could be created.)

It is many times practical and useful to be able to edit streams / events (most often due to bugs, unforeseen situations or other kinds of mistakes - which always happen), and this is possible with the first protocol.

Second protocol will be almost the same, except that instead of having the event batches serialized in the stream MD, it will have a collection of objects with metadata and xor address to immutable data, for each event in the batch. The immutable data will store the actual payload of the event, which is all domain logic related data.
The metadata will contain things as:

  • event clr type (used for deserialization),
  • sequence nr,
  • event name,
  • causation id,
  • correlation id,
  • event id,
  • time stamp.

Stripping the event of these unique property values, we can many times take advantage of the deduplication of immutable data (naturally depending on what other unique data is part of the payload). Some basic searching will be a tad more effective too.

One other significant difference is that the stream MD will have append only permissions. (Event sourcing is in theory supposed to use immutable streams). The ability to correct mistakes and bugs will be there by the concept of RestructureEvents.
When some error in the stream occurred, which is not reflecting something real, but rather corrupts state, a RestructureEvent can be appended, with instructions on how to handle the stream.
When reading the stream, it is first searched for any RestructureEvents, then all changes in these are applied to the stream, then the resulting stream is replayed (applied to an instance of the aggregate) to get to current state.
The restructure event will simply say which key shall point to some other immutable data instead of the corrupt one.
This way the history is kept, but our application can still get instructions how to avoid corrupt state.

The repository on github has not been updated with the above yet (will get there when I have some more time over), but some general cleanup and some CQRS examples (cmd handler, cmd + events) has been added, as to show how to use it.

It’s likely that this library will eventually only be using the second protocol, with additional improvements.

Soon I might be announcing some other project, which will make use of the event sourcing database technique enabled by SAFE EventStore library :slight_smile:


#15

Great work you should be proud of this contribution!:smiley:


#16

The IData protocol (mentioned in previous post as Second protocol) has now been implemented and the PR merged into master. Check out the code here: https://github.com/oetyng/SAFE.EventStore

It is now possible to use SAFE.EventStore with the alpha-2 network.

However, you will still need to do it via debugger in VS, since I have not yet configured paths to match with the release binaries. I’ll update this topic when I’ve fixed that.

Currently you can:

  • Create databases in alpha-2 through the web UI at localhost (only when debugging solution in Windows).
  • Use the TestCQRSApp console UI to write notes into an example NoteBook, stored in a database created through the web UI (or implicitly create a new, with name specified in Startup.cs).
  • Browse the databases, the streams and the events in the streams, through the web UI.

Coming up:

  • Download the binaries and run also on OSx, Linux and even ARM.
  • Include SAFE.EventStore in any C# project, and start building event sourced applications, with SAFENetwork as backing data storage solution.
  • A small secret surprise project based on SAFE.EventStore :wink:

If there is interest I might be able to do a live chat, share screen and go through the various parts and discuss what is going on and generally explain how this can be used. Would be fun, so just let me know and we’ll try to arrange something with as many as possible.


#17

Wow! Very cool! I am going to have to find some time to test this out. I linked the repo url on reddit too.

It is great to see these examples of advanced SAFENetwork use cases!


#18

Could you explain like I’m 5?

I feel dumb and don’t understand.


#19

Hey @Grizmoblust

I’ll give it a try :slight_smile:

(There’s also a podcast which goes through some of it: SAFE Crossroads Podcast #38, Event Sourcing on the SAFE Network)

SAFE.EventStore library contains tooling to be used by software developers.
It is an implementation of a specific database type - a certain way to organize data. It adds a layer of abstraction on top of current SAFENetwork API, which means: it makes it easier for developers to interact with the network.
This library can be used as foundation for data storage as is, both according to Event Sourcing practices but also in other ways, if used under additional layers of abstraction.
Simply put: a protocol to store data in a specific structure, which makes it easier for new developers to get started, without knowing much of the details of SAFENetwork implementation.

There is also a SAFE.CQRS library which is yet another layer of abstraction enabling developers to build applications according to patterns and practices commonly known as CQRS and DDD (Command and Query Responsibility Segregation and Domain Driven Design). Those patterns are ways to handle scalability as well as improve development experience (CQRS) and to be able to handle complex business domains (DDD).
It is supposed to make it easier to build robust, scalable applications that are easier to maintain and that are more apt to handle the problems they are built to solve.

But there is more to be done on those areas. For those who are somewhat familiar with application design patterns it can be interesting to look into DCI (Data Context Interaction). It is yet another level of design practices that I will eventually use for applications that will make use of the libraries mentioned above. (Here is a very good 43 minute talk by Andreas Söderlund on DCI).

All this talk of layers and abstractions is maybe a bit confusing if you’re not a software developer, but each layer of abstraction is supposed to make programming easier in various ways. A good abstraction doesn’t lose much power in exchange for this simplification, but generally you could say that you have more freedom/power with less abstractions, but also a much steeper learning curve and many more open pitfalls.


Existing software applications on Safe
#20

An update!

There was an addition to the family recently and workload has been a lot over normal, so the progress with the previously mentioned milestones are not that fast :slight_smile:

But I can tell you a bit about projections, and the upcoming protocol for it.


Projections

Projections is the way we make the event streams accessible for a client, such as an app or some component in the system. It is the read model, or the Query part in CQRS (Command and Query Responsibility Segregation). We can also call a projection a view model.

So, when you have an event sourced system (i.e. you store the data in some sort of event store) you want to be able to access it in a performant way for the relevant context.

This is the beauty of it. One set of events, from the same or different streams, can build different views. So the same event can contribute with its data, to build up any number of different projections.
For example, we have the famous account example:

  • You can have one viewmodel that represents the balance of your account. For that you need the AmountDeposited and AmountWithdrawn events from your own account stream.
  • There can also be another viewmodel, with the amount of deposits coming in for a specific date. Here, all the AmountDeposited from all users, are applied.

Let’s say we also store the accountid in this viewmodel. And voilá, we have something we can query for, say… give me all users that made a deposit today.

What we realize is that we can build up queryable datasets from the events, which will be much (MUCH) more performant, than iterating through all eventstreams and trying to calculate this data - over and over.

Now, for our SAFE EventStore, the basic version of projections, is executed locally.
A local agent will run in the background, and process the event streams specified by the user, into the projections wanted. Problem here is that this data, with no further connection to outside world from the local machine, is only available locally.

What we need is a protocol to store these projections back into SAFENetwork, so that they can be queried from other locations.

The protocol

When we save a projection, it means we save the current state of something, to the network.
This state is arbitrarily large, and for that reason we use ImmutableData. Also, it is a representation of current state, as we knew it at that time. We get a version of the state.
Now, every update to this view model, say one property Amount is increased with 100. will lead to the entire viewmodel being saved again. Since we use ImmutableData, this is a new data structure stored, and a new data map retrieved.
What we have now is an older and a more recent version.
This is actually something not uncommon to want, for example, account balances at end of year. That means you want to go back and access a certain version.

The picture is coming together here, and what is interesting is, that it is strikingly similar to an event stream.

Our event streams as implemented in SAFE EventStore, use ImmutableData for storing the payload (which can be anything…) and chains the events together in a stream represented of at least one MD.

Do you see it? :slight_smile:

We can actually re-use the EventStore, for our projections.
One stream would be one instance of a viewmodel, with it’s entire change history, saved as individual events, with metadata and datamap to the immutable data in the network.
Current state would be the very last event in the stream.

And there we have the projections for our event sourced system, implemented in SAFENetwork with our own EventStore.

What we need to do next, is to organise this for querying. This I still have to design, but stay tuned, both for the code update in GitHub, to see the actual implementation of projections, and the quest for a queryable store! :slight_smile: