Introducing SAFE EventStore (event sourcing database)

Whatever it is you are doing. Keep at it! In your own time of course. Precious little bundles are number one priority. Work comes second!!

5 Likes

Congratulations on the new arrival! :baby:

Iā€™d love to be able to say yes, but I 'm not sure I do :thinking:
Let me have a shot and you can put me right.

The stream of events is stored on the network as ImmutableData, so each new event (say buying a coffee) is added to the pile of events stored.

At the same time a new projection is created and also stored as ImmutableData. A projection is a snapshot of the current state of the event stream, taking in chosen parameters (amount spent, item bought, etc).

When this happens a pointer to the new projection is stored in a datamap which is a MD. A pointer to the old version is also stored - i.e. the new entry is appended to the MD rather than overwriting the old entry.

So the MD is essentially a record of everything that has happened in the stream (data and metadata), and it allows applications to access and query the stream of events very rapidly as the current state is the last entry and previous states are all there in one place as pointers accessible without having to loop through the data ā€¦ (Iā€™m floundering now).

Am I anywhere near?

2 Likes

Haha, Iā€™ll keep at it :slight_smile: Yes, precious little bundles indeed!

Thanks JPL!

You know, that is pretty near spot on. You do see it :slight_smile:

Iā€™ll just clarify one part and then expand with an example also:

With this organisation, there will be two types of streams (a stream is a chain of MDs, or at least one MD as long as it is not full).

The event stream, which keeps every incremental change.

  1. CoffePouredUp
  2. MilkAdded
  3. SugarAdded
  4. SipTaken

The event CoffeePouredUp might contain info on volume, type of coffee and what not. Letā€™s say we just store the fact that there is a cup of coffee, so nothing more detailed than that:

So then there is the projection stream. It will, at each entry, keep the state as it was at that time.

  1. A cup of coffee
  2. A cup of coffee with milk
  3. A cup of coffee with milk and sugar
  4. A cup of coffee with milk and sugar, less one sip

orā€¦ when looking at the implementation details:

  1. Metadata + datamap
  2. Metadata + datamap
    ā€¦ and so on

And in the metadata we have date time for example, and things like that.

Our current version of the coffee is the number 4, this is what we are holding in our hand. Maybe this info was stored at 15:20.
If we want to see what our coffee cup looked at before 15.20, then we look at number 3.

The model maybe looks like this

CofeeCup
{
    string Content
}

and at 3. we have

CofeeCup
{
    Content = "coffee with milk and sugar"
}

The difference between these streams is that the event stream is immutable by design. What happened happend. Every event represents a fact, we cannot go back in time and change the fact.

If there is anything we found out later, then we add that to the stream:

  1. FoundOutTheCoffeCupWasJustAnImagination

and the current state in projections stream would perhaps then be

  1. null

(meaning, we just deleted it)

But, the difference: the projections stream can be recalculated.
Because we can at any time decide that we want to include more data in our viewmodel.

Say, we also include temperature of coffee, because we happened to have a thermometer that we dipped into the coffee.

Maybe those events come from the CoffeeThermometerStream:

  1. TemperatureRose
  2. TemperatureSank
  3. TemperatureSank
  4. TemperatureSank

They would contain an identifyer so we know where this was (a geolocation specifying the exact location of the coffeecup, or an id for this cup), and the temperature change in degrees.

So, our viewmodel maybe looks like this:

CofeeCup
{
    ContentType
    Volume
    Temperature
}

now we want to see how this looked at 3., and if we rebuild the projection and include the events from the thermometer, we get:

CofeeCup
{
    ContentType = "coffee with milk and sugar"
    Volume = "250 ml"
    Temperature = "35ĀŗC"
}

The first model at 3. with fewer properties, still exist in the network, since it is an ImmutableData, but we wiped the MD clean, and entered new entries with metadata and datamaps.

That is the big difference compared to the event streams, with regards to their lifecycle.

Feel free to ask more! :slight_smile:

3 Likes

Thanks for the great explanation. If I was stuck on what was happening with the MDs but I think I get it now. I also went back to your previous explanation of projections to refresh my memory. So itā€™s the ability to use the same raw material (streams of events going back in time) to build multiple updatable projections in full confidence that the underlying historic data is reliable, with at the same time (if the projection is specified correctly) the ability to query data very efficiently.

1 Like

Aah, yes, I had forgotten about that post. It is very detailed indeed. I recommend it to others who want to know more!

Very much so. It is of big importance that rebuilding is easy, because you want to be able to adapt the projections to the ever changing needs of a consumer (like a view in a website or a new feature etc.).
What I drafted above was how to store the projections. The actual querying will still need some design.

2 Likes

I donā€™t remember the exact phrasing, but David Irvine mentioned some boxing analogy about the patience being the important part, winning the long game.

So, we have another update :slight_smile:

Actually, there will come a day in a not too distant future, when I can actually spend more than a day here and there a month on SAFENetwork related coding. We will get there.

The big news for this update, is that SAFE.EventStore is now upgraded to be used with safe_app_0.6.0 and the latest csharp bindings!
This Saturday I refactored the interaction with the bindings and did some final cleanup and testing yesterday.

(SAFE.EventStore repo is here)

Thereā€™s been a lot of nice simplification and elegancy introduced with the new csharp bindings so itā€™s getting easier to work with. Very nice! I like that there are now less static classes and more instantiation, like I had suggested once earlier (I donā€™t know if my call was heeded or if it was already thought of).

Additionally for the EventStore, I have increased the capacity of the db. I wasnā€™t happy with 1k dbs, 1k stream categories and 1k stream instances per categoy.
So I introduced some sharding by non-cryptographic consistent hashing.
Each category will have almost 1k shards (minus a couple reserved entries per MD for other stuff) which each hold 1000 streamkeys, instead of each category holding the streamkeys directly.
So far, I have just multiplied stream instance capacity by 1000, but I will do the same to the events.
In effect, we will then get a total of 1000 trillion events per database id, and 1 million trillion events per safe account (so, with total of 1k dbs).

I will have to see how the performance is when having any substantial number of events in there thoughā€¦ I mean, the sharding still give fast key value access, but if I would ever need a full scan, then that would probably be completely insane. But weā€™ll get there :slight_smile:

So if you have an app (1 database id) with expected life time of 10 years, you could fill it with more than 3 million events per second, and still not fill the quota during its lifetime. Soā€¦ it will take a lot of devices to be able to push that many events to the network per second :upside_down_face:

(I guess, eventually, Iā€™ll have it expanding to be ā€œunlimitedā€ by some nifty way.)

Thereā€™s still garbage collection of delegates in the csharp bindings, which causes NullReferenceExceptions. This happens inevitably after doing some writing or reading of data. But I think the major bug that was causing me problems with safe_app_0.4.0 is now gone, and that is good, because it was an inexplicable one. Had no idea what was causing it. The garbage collection of delegates is well defined, even though I donā€™t currently know exactly how we should solve it.

So, thatā€™s that for now. As soon as the garbage collection of delegates is solved, I think we have a functional (albeit very simple) eventstore for SAFENetwork :slight_smile:

32 Likes

This is a superb thing to have at this time, huge congrats on this so far. Superb !!

17 Likes

Yay, Iā€™ve been anticipating an announcement from the forests of Sweden for some time. I felt sure you were still beavering away out there, not gnawing down trees but instead refining some valuable backend infrastructure for SAFE. :smile:

12 Likes

Very cool! Glad to see this project progressing well!

3 Likes