App idea: Safe Layer. Let's talk about the data

Greetings everyone,

I’ve been meaning to post this for a while now and I finally took the time to finish it. It’s about a different approach to apps and website that I think fits well with the unique nature of Safe. Hope you like it!




When I think about Safe, the thing that gets me the most excited about is the actual data. While it might not sound very sexy, I think the ability to decide how you interact with the data to fit your personal needs will be an important feature that will help Safe reach mass adoption. But even though the potential of owning your own data and sharing a database with everyone is tremendous, there isn’t a lot of talk going toward the data. So let’s talk about the data, the cool stuff we’ll be able to do with it and more practically: the kind of tool we’ll need to do it.

If you haven’t already, I suggest you check out the thread called The Essence of SAFE apps by @BenMS. It’s short but it explains well the difference between traditional apps and apps powered by Safe. It’s the seed that’s been growing in me and in a way the basis of what I’ll be talking about.

What is Safe Layer?

Safe Layer is the code name of a theoretical tool used to interact with the data on Safe. Its primary goal is to make each piece of data self-contained and reusable, like little Lego blocks, and give the power to the user to rearrange them as they please.

Another goal is to make it easier to create something for Safe. We have a great community of enthusiast who can’t wait to take an active role in helping the project but who don’t necessarily have the knowledge to program or time to learn the API. Safe Layer would lower the barrier to entry and give them the opportunity to contribute to the success of Safe.

I said “theoretical” because it’s not something I’m planning on building. I thought at first I would, but the reality is that I don’t have the time nor the right skillset to properly pull off. But I do believe it could be an important tool for the network and it’s the reason why I took the time to write this up. Worst case scenario at least the idea is now out there and no longer just spinning in my head. Best case scenario MaidSafe decides to build it because they can’t live another day without it. :wink:

With that said, let’s start.

The data layer

Compared with how the current internet works, data on Safe is not confined to a web server, stuck within the boundaries of a web page or an app. Instead, Safe is a giant database and the data can be used directly. Therefore at its core, Safe Layer can interact with data directly. So the first step is to create a simple data editor.

Similar to a browser, you enter the address of a piece of data in an address bar and it loads the raw content of the data. From there you could, depending on the permission of the data, add/modify/delete the information contained within it.

dataexplore1

And since this is a network of interlinked data, when the value contained within the data is another Safe address, you can click on it and be redirected to that other data. Just like you do in your browser when you click on an URL to load another page.

dataexplore2

Interesting information is usually made of multiple piece data so you can also configure the app to automatically fetch a specific type of data when it is found inside a specific context. For example, you go into a data of type user_profile. This data contains a link to a data of type public_broadcast and Safe Layers loads it automatically in a separate window and show them side by side. Up to a certain level of depth to avoid loading the whole network.

Data templates

In order to avoid everyone from creating a bunch of semi-similar data that is hard to mix together, the app would use data templates to create new data. You select the type of data you want to create and the app downloads its template from Safe. It would then show a form where you are asked to fill the key-value pair of the template.

The list of templates is configurable and the user can add a new entry or override an existing one. The list itself would also be downloaded from Safe. At some point, an organization like MaidSafe could take the responsibility of maintaining a list of standard data templates.

Now working with raw data is useful but there’s, of course, something to be said about a presentation layer. So next, we put back the presentation layer around the data but in a way to keep the separation between the two and in a way to allow users to easily change it to their taste.

The presentation layer

The goal of the presentation layer is to provide a customizable way to show a type of data. The type of the data would be written directly in the data itself. For example, a data that contains the profile of a user would have a key named “data_type” with the value set to “user_profile”. Then you can associate this data type with a presentation template that is stored somewhere on Safe. So when you encounter this type of data, you automatically download the template and use it to display the data. Simple.

Since the mapping between the data and the presentation template is done at the user level, you can change it to suit your needs.

Note that the presentation template could be written in any language. For example, if Safe Layer is built with a web engine, the presentation template could be written by using a mix of HTML, CSS, and Javascript. But it doesn’t have to, and at some point, we could even have multiple versions that target different platforms and use different technologies for the templates. The goal here is two folds: to give the user the choice of how a piece of data is presented and to make the data independent of its presentation.

Advanced features

What I described above would be the core features of Safe Layer. The first step. Even though it’s basic, I think it would allow us to start walking on the right path. Then, there’s a lot of other features we can add over this, here are a few examples.

Composition

In short, a composition is the ability to embed a presentation template into another one. The data that would fill the embedded template could come from a direct link from the main data or, as I’ll explain after, a soft link.

For example, a data describing a user profile contains a link to a list of public messages. The list of messages contains a link to each individual message. Then with composition, you can wrap all that into a single view.

Soft links

This idea is inspired by a post by @seneca. He explains how to link to another piece of data without providing an explicit address to it. It goes like this:

Every data has a unique address in XOR space. With this address, you can retrieve that data from the network. If you take this address, add a specific word to it and hash it, what you get is another address that points somewhere else on Safe. If you place your new data there, anyone who knows the specific word can retrieve it by doing the same process. It’s a soft link.

Here’s an example of what you can do with this: community feedbacks. Take the address of a piece of data, add a word like “XYZ_comments” at the end of it and hash it. What you’ll get is another address in XOR space and that’s where you’ll find the comments of your community on that piece of data whether it is a store, a video, a song or anything worth commenting on. What Safe Layer would do is check in the background if these soft links exist and allow you to load it if it’s found.

With soft links, communities aren’t stuck on a website anymore and travel with you as your surf the network.

Unified style

Look at your desktop, now look at the desktop of sci-fi movies. What do you see? Yours is an incoherent mashup of different styles, the other is a unified slick design. Controlling the presentation layer means you can apply the same style to everything. That Star Trek UI design you’ve been craving for all your life, you could finally get it.

This could be your life.
LCARSWT
Reference: https://github.com/matthias-wolff/LCARSWT

Reaching mass adoption

As an aside, here’s how I think a tool like Safe Layer can help us reach mass adoption. I hope you don’t mind me tagging you @nicklambert.

I understand the current focus of marketing is about explaining how Safe works and why it’s different. This is important to get tech people on board. But when we’ll want to get the general public, the thing that people will notice is not the groundbreaking tech but instead what we can do with it. If I may suggest an angle it would go something like this: the one thing that everyone can relate to is how we interact with data on a daily basis. Whether the data comes from our favorite social network website or our favorite calendar app, what we care about is the actual data, the picture, the message, the note, the meeting coordinates. Dealing with the quirks of the container, the website or the app, is a strain that everyone experience on a daily basis.

Therefore I think a strong message would be to enforce the idea that Safe allows us to take control over the container, to mold it to our desire and to make it work exactly how we need it to. As silly as it sounds, having a calendar app that does precisely what you want is the stuff of dream for the people, myself included. I think that’s a positive message that will reach a broad spectrum of people. It’s also a characteristic that is unique to Safe as there’s no other tech out there that can make such a claim.

Though, before we can do that we need to actually make it true. Having a giant global database that everyone can use is not enough. What we need is to free the data from its container and empower the individual to take back the control over its data and I think that’s where a tool like Safe Layer can come in.

So anyway, I could write about this all day but this post is already long and I’ve been delaying it enough. Hope you had a good read and that it gives a different perspective on the unique kind of experience we could get with Safe.

Cheers!

42 Likes

This a really really cool concept! Especially the unified styling, I’d make the whole internet look like windows xp for ultimate nostalgia.

6 Likes

A super piece of work. Really valuable to bring these ideas out like this, and add in some of the special features of SAFE like “soft links”. I’d forgotten all about that :slight_smile:

Similar ideas to these are occurring in different projects, already here as you show (in the post by @BenMS - from 2015 indeed!). The group leading the way in this area is I think SolidMIT. For example, taking your simple data editor the first diagram in the OP, I include a link below to a live demo of exactly that, but working on Solid/www rather than SAFE/www (for now :wink:), see:

https://linkeddata.github.io/mashlib/dist/browse.html?uri=https%3A%2F%2Ftimbl.com%2Ftimbl%2FPublic%2FTest

The above is a generic RDF data browser / editor which is just another Solid Web app on a Solid server.

The link takes you directly to explore some of Tim’s own Solid storage, but you can paste the URI of any Solid server and explore from there (including links which go from one server to another). So the model is like the contacts list you described in your introduction. It also uses different data viewer/editors depending on the type of data it encounters, with a default for data it doesn’t have a special template for. In Solid they call these templates ‘panes’ and there is an extensive library of solid-panes you can grab for use in your app.

This all works because the data is in known formats, RDF (structured, semantic, linked data), images etc.

Any website can provide its own app, template, editor etc, to interact with that same data on a Solid server using the Solid protocol. We can do exactly the same on SAFE, which is what I’m working on.

In Tim’s Solid data browser (above) you can paste any RDF link or Solid container URI into it to browse the contents, or edit and create if you have write permissions, including different types of RDF resource (profiles, contacts, meetings, discussions etc) which have been created by a variety of Solid apps.

It is because RDF / Linked Data already provide some important building blocks that I’m so interested in exploring the ideas and achievements of the Solid team (still developing) and others working with RDF etc.

The building blocks from Solid / Linked Data include the means to describe linked data in ways that different apps can create and reuse, as you envisage, including many ontologies which app developers can just pick up and use.

To help with this, I have been implementing a Safenetwork Web API which includes support for the Linked Data Platform (a subset of Solid). I also have a first Solid app saving RDF to SAFE in late stage of development.

Using that Web API I hope it will then be possible to port other Solid apps, including the Solid Data Browser, and we can really start to see these ideas take shape on SAFE, at least the early stages of what you have outlined here.

So great work! I will come back to your post because you’ve thought these ideas through and presented them in a helpful way.

22 Likes

@happybeing, that’s great, I have been thinking for a while that I should ask you on some appropriate place, about what it is you do with SOLID and what SOLID is about. So this was some great info that I think was new (or I have missed some post). But maybe be you could start a thread were you explain more things, in detail? And maybe elaborate on how you see that it can be used and so on. It would be good to read anyway if that’s something you would want to do.

@DavidMtl, I am too really into the data aspect of the network, how to store it, organise it, access it, work with it. Really like the idea of ui for the raw network data objects. I did something similar with a simple ui for the event streams of my EventStore, and the idea there is to do a data management ui, specifically for event sourcing dbs and projections.
You put down a lot of effort to this post, that’s so good to see.

8 Likes

Thanks for the support everyone, glad you liked it.

I was not aware of the Solid protocol, very interesting stuff. Reading the about section of the Solid MIT website and they really do share the same vision.

I don’t know the detail of their stack but it’s got to be tedious to try to build it on top of the traditional web server architecture. Safe makes thing so much more direct. You don’t need to make an effort to extract the data, it’s just there already. All you need is to make an effort to not screw it up :slight_smile:

I think they are in it for quite a surprise when they find out the things we can do with Safe.

Thanks a lot @oetyng. I’m glad to hear there’s a real interest for a more data-driven approach. Now we just need to build it :slight_smile:

12 Likes

Data templates could be a specific type you add with the data editor. The key would be like in the data editor, for example username, while the value would the type, for example string (or even username could be a type itself, which again contains a core value of string, perhaps with rules for certain restrictions on the value, such as a max or min length), then maybe metadata like if the property is required or optional.

Maybe you’ve though of this already, but on the presentation layer, presentations for different data types could also be made for adding data. When no such presentation exists, you’d just use the normal key/value editor. For the data_type “user_profile”, the form could look pretty much like the read-only UI, but username could be a text field, the avatar image could be clickable to open an image selector etc. The presentation would then use the SAFE API to write the data to the SAFE Network, including uploading files like images and writing the actual data entity.

7 Likes

@DavidMtl I can’t pretend to know what your talking about (It’s way over my head), just wanted to :clap:

It might also be useful to share this on the SAFE dev forum.
:stuck_out_tongue:

5 Likes

I think you might be surprised how close the two approaches are :wink: It surprised me, but when you think that the goals of both projects and the two main visionaries behind them are so similar, it makes sense that the solutions converge in key respects. So both projects can learn from each other.

We don’t have all the answers here and so using the extensive work already put in by Tim et al in this area could be a major win for us as well as for them, as you point out, seeing what SAFE might deliver. So big win win. :slight_smile: David’s response to my earlier Solid + SAFE Synergy post reflected this I think.

To respond to the technical side, both projects follow a similar data model: simple storage API on the one hand, and putting the application complexity in the client (desktop or Web app) which, if you use common data standards, enables an ecosystem of applications creating and refusing each other’s data.

So both projects end up with essentially similar ways of doing this. The LDP in which the Solid standard is based is a RESTful interface for storing and retrieving data much like the SAFE NFS API which in turn was inspired by its carefully thought through predecessors on the Web. It’s like shared DNA, splitting and recombining in each generation, while evolving towards what is needed in the ever changing environment.

My impression (still learning) is that Solid is ahead on data standards, data processing, apps (obvs) and access control, but that SAFE is ahead on security, privacy, anonymity, authorisation, decentralisation, accessibility (not in all respects, so debatable). Solid is way ahead in allowing users to extract the maximum value and utility from data, both their own and whatever is public, or shared with them. While SAFE is way ahead on ease of deployment, scalability and maintenance (which ties into accessibility). Solid has a good story on identity, with some aspects we can learn from, but overall SAFE makes identification and authentication almost trivial, whereas it looks a bit of dogs breakfast on the Web (hard to understand, implement and use currently).

I’m still head down atm, but this will come. Hopefully something useful for the DevCon but it will take me longer to really understand and communicate it. @DavidMtl has helped a lot here by elaborating a framework which we can all use to understand these ideas better.

I’m very encouraged by the interest of this community in these ideas, and yours particularly @oetyng because you bring in so much up to date experience of building complex data systems, and areas I didn’t know existed, at least not as a formalised approach. So I look forward to further discussions, and I will start a topic soon where we can get deeper into Solid + SAFE side (remind and pester me if I’m too slow!). Check out the “Synergy” link above in case you missed that.

10 Likes

This is a really nice idea and tool to have.
I have been tinkering with safe apps and sites, and I found myself several times in the situation where I would have loved to have a simple graphical tool to manipulate the data behind the scene without having to write code.
For instance in the simple safe forum, there are several badly formatted topic names. My fault ! I didn’t take time to implement anything to prevent that from happening. These topics are sitting here and are not browsable. I wish I could use Safe Layer to rummage in the mutables and remove these, without hassle.
I can imagine plenty of similar situations where this tool would be really handy !
Sort of a phpmyadmin for safe, though what you describe goes way deeper than that.

:+1:

6 Likes

Yes, exactly! I almost talked about phpmyadmin in the section on the editor :slight_smile: I think a strong point of Safe Layer is that it can be built in, well, layers. As you mention the editor would already be a useful tool for people working on a Safe app or website.

Damn, now I feel like the guy who yells “Guys, I’ve got a brilliant idea!” and people answer: “Yeah, we know, we’ve been working on it for a DECADE already…” Eheh, great minds think alike, some are just slower than others :sweat_smile:. I really need to dig more into what you have been up to. I feel a bit ashamed I haven’t already.

Here’s a Saturday morning roadmap proposal:

  • Build the editor/explorer first, should be fairly easy.
  • Add the presentation layer with composition so we can start having some fun.
  • Add unified styling so @Nigel can live in a world of xp. (Lol that blue, I agree it brings back memories).
  • Then work on more complex features like linked data and soft links.

Once we do that, going back to a website on the old internet will feel like going back into medieval time. “What? I can’t put the sidebar on the right side? Gee, that’s… rude man!”

10 Likes

Exactly! The data editor would always be available, as you say when there is no presentation layer available you just load the data. And you could create data templates from the app. This would make it easy for anyone with an idea to try it and share it with others. And then other people could add a presentation layer over it. Community-driven development at its finest.

2 Likes

If you or anybody wants to try this using the LDP API I’ve created, it would be a matter of porting Tim’s Data Browser - which I’m sure he would help with. I’d like to have a go myself, but I don’t know when I’ll get to this so it would be a great little project for somebody who wants to explore Solid and how we can enhance SAFE and create a SAFE App ecosystem of tools for creating, sharing and reusing data. The LDP API I’ve built is now complete and solid enough [cough] for this already I think, so it could be started right away. This is a nodejs browser library, so familiar territory for most devs here.

I haven’t looked into the Data Browser to know if it would be easy or not, but if it sits in top of LDP in the way I expect, it could be simple indeed.

A bonus here is that we can grab existing data (calendars, social posts, meetings, notes, even dbpedia! etc) copy them onto SAFE, and make them all immediately available to browse using all the existing editor/viewer templates (solid-panes). Tim has a script to copy data between LDP containers, so stealing data should be easy too. :wink:

That would give us a great data playground for all developers, and which users could also play with and suggest ideas for, and show people the value of using RDF / Linked Data as a portable data standard.

This might be a valuable community project too, so maybe something that could be funded, although it is also small enough for somebody to do just as a learning experience.

@DavidMtl no probs if you (or anyone else) want to go own way on this - but if anyone wants to collaborate, I’d love that too :slight_smile:

9 Likes

Can’t be me, I’ve never been spread this thin of my entire life and it’s gonna get worst soon :sweat_smile:. But I think it would be a good fit for the CEP program and I’d be glad to add my 2 Safecoins, or more, if needed.

1 Like

Another fun thing that could be done with this is composition of logic, not just composition of UI.

Basically make entities with key/values something this

id : 123
data_type : function
input_type : int
output_type : int
name : add_numbers

Then another function

id : 124
data_type : function
input_type : int
output_type : int
name : multiply_numbers

These entities are basically language independent interfaces for a function, perhaps there should also be some more description of what they do.

Next these interfaces needs implementations

data_type : function_impl
function : 123
language : javascript
code : javascript function for adding numbers

data_type : function_impl
function : 124
language : javascript
code : javascript function for multiplying numbers

Then another entity could contain a list of functions to be composed,they just need to have matching input and output types.

More useful than just adding numbers could be functions that do things to types of data found on the network. One function might for example take a user_profile and return it but with a compressed image.

1 Like

Language-agnostic logic, I like that. A quick google search reveals that’s it’s actually a thing and called General Purpose Datatypes. And with this, we would have the MVC pattern where only the view is really dependent on the platform. Of course, the logic as you say would need to be implemented for each platform but the real logic stays the same, it’s like the bytecode of Java.

Here’s some other similar stuff
http://www.jpaulmorrison.com/cgi-bin/wiki.pl?CoordinationLanguage

One usecase of having functions like this is to make validators for data types. You have the data type user_profile and validator could be a function that takes a user_profile and returns true or false if it is valid or not. It could for example be invalid if the username is too long. This would be useful to run before persisting the data.

1 Like

Nice rabbit hole you just sent me in :smile: It seems a lot of concept are converging toward modularity, I feel like Safe will fill a central role. Exciting times ahead that’s for sure. Thx for sharing I need to read more on flow based programming now :slight_smile:

2 Likes

FYI I’ve not looked into this yet, but Tim mentioned that RDF supports rules. At the time I assumed this might be used for validation, but I don’t know. A quick search in case it’s of interest:

https://www.w3.org/2000/10/swap/doc/Rules

I think SAFE could gain a lot by stealing, I mean building on these standards - it saves a lot of time, makes it easier for others to reuse their skills and knowledge on SAFE, and means we can build using libraries tools, and apps that already exist. Very exciting opportunities :slight_smile:

3 Likes

Definitely. I really ought to look into more details of Solid. Solid-panes is quite interesting, I did’t know they had something like that.

It’s a bit like what’s described in these blog posts. Paul Chiusano’s blog has many interesting posts in general I think.
https://pchiusano.github.io/2013-05-22/future-of-software.html
https://pchiusano.github.io/2013-09-10/type-systems-and-ux-example.html

Another related blog post

I also really like Bret Victor’s stuff, like these talks. That kind of stuff combined with tons of linked open data could lead to some relly interesting things.
http://worrydream.com/#!/InventingOnPrinciple
http://worrydream.com/#!/TheFutureOfProgramming
http://worrydream.com/#!/DrawingDynamicVisualizationsTalk
http://worrydream.com/#!/LearnableProgramming

2 Likes

I’m not sure how close this is to your line of thought, but just remembered…

Linked Data Reactor

Linked Data Reactor (LD-Reactor or LD-R) is a framework to develop reactive and reusable User Interface components for Linked Data applications. LD-Reactor utilizes Facebook’s ReactJS components, Flux architecture and Yahoo!'s Fluxible framework for isomorphic Web applications. It also exploits Semantic-UI framework for flexible UI themes. LD-Reactor aims to apply the idea of component-based application development into RDF data model hence enhancing current user interfaces to view, browse and edit Linked Data.
https://github.com/ali1k/ld-r

3 Likes