Putting Web Apps on SAFENetwork by Adapting Existing Frameworks

Development
21

Good to have your input @joshuef. Maybe others will join in as fans of meteor and other frameworks. I think if we can establish a workable core, it will be easy for others to plug their favorite in alongside our as a replacement.

Hoodie

Glad you like hoodie. One worry I’ve seen expressed over it - which appllies equally to SAFE and indeed every noBackend approach - is the lack of data validation which a backend can make much easier to control. I haven’t seen it spelt out, but assume the issue is that anyone can tweak the client and screw up the integrity of data on the server. I do think that’s a big potential problem, so something to bear in mind and what anyone who does understand this stuff has to say on it!

Database API

My ignorance is high everywhere, but most of all in not understanding how to achieve the higher level capabilities of a database API (eg CouchDB) with the least load on client and least leaving copies of data in local storage.

Ideally, everything should be either in memory, or stored on the network - at least that’s the long term aim I think. But at the same time, offline first would ensure the best UX and eliminate the user’s need to be aware of the state of connectivity - except where absolutely needed (e.g for real time interactions with others).

PouchDB

The most interesting answer I’ve seen, because I don’t believe SAFE will provide much in the way of database queries or CouchDB tasks, is something like PouchDB. It’s a cross platform javascript database that provides a complete CouchDB API on the client (offlinefirst capability), and supports sync with other PouchDB/CouchDBs, and with CouchDB sync compatible clouds (of which the are several). All that adds up to easy migration of some powerful web apps I think.

Sync

The aim of CouchDB was to do one thing really well, and that is sync - so I think we should think most about whether sync is very important, and how well we can make it work, say by leveraging PouchDB, in our use case.

I’m wondering if we could use PouchDB to provide apps with a CouchDB API to local or remote instances (i.e peer apps on other client devices) with the data maintained mostly or entirely on SAFE. I think we have to wait for the SAFE collaboration and messaging APIs to answer this. Hopefully they’ll support messaging over REST.

I think this would give developers a lot: lots of capabilities, decentralised, easy to develop, totally cross device platform. All achieved with widely known, proven languages, APIs, dev tools etc., no new skills to learn. Write once, run on anything (Note: PouchDB supports almost every browser on every desktop and mobile device).

2 Likes
22

It’s perfectly possible to do proper data validation and checks in the client, that’s more of a code quality issue (assuming you have some basic auth for your PUTs anyway), but yes, you’d also want this server side for extra security.

As for P/CouchDB; this seems pretty neat. HTTP as the API is also great. Syncing into the browser is a nice feat for smaller apps, but realistically this would only be feasible for smaller datasets. In the end there’s still the main server/shard that you’d need to be syncing from.

It does seem feasible that at least a NoSQL db could be saved to the SAFE network with relative ease. It’s essentially just a JSON file; and this file itself would be pretty easy for a client/webapp to request and parse for initially populating a PouchDB instance or so.

The trickiness would come with adding CRUD facilities to this to keep the file up to date.

I had a read through @Krishna_Kumar’s Firefox doc and sample, and seems like an App handling API interactions will be the way forwards, as it allows us file system access.

From that I think hijacking network requests (chrome.webRequest - Chrome Developers ) to catch anything safe:api related we might be able to run them through something PouchDB Server-like to affect changes to our NoSQL db files via the SAFE browser app itself.

I’m not sure on the limitations of browser extensions yet, a superficial googling didn’t turn up a whole lot of extensions using Node.js as something server like. But it didn’t exclude it either. I’ll come back when I find out more either way.

2 Likes
23

I agree this looks very doable, but not just for small data sets. If it proves possible to sync subsets to/from SAFE we’d have a lot of room :slight_smile:

I sketched out some architectures for shared and distributed databases today. Distributed, if feasible would make very large datasets feasible with web clients.

I now have to research the feasibility, but it’s quite a small task so I’ll let you know.

Regarding validation, it’s not possible to do this securely in the client. This is the point, and it could make it too risky to implement shared databases - because anyone editing the client JavaScript could sabotage the data. That would be a shame, but I don’t see a way around it. We can always recover old versions, but I’m not sure if that could be made workable. Anything decent could effectively be DDoS attacked by a single client. Ugh!

24

Hi @Krishna_Kumar I hope you had a good weekend :smile:

I’ve been thinking about how to support databases in web clients, and am looking again at CouchDB as a model for SAFE REST API, because there’s a JavaScript client DB implementation (PouchDB) which is compatible with CouchDB. If the SAFE REST API is suitable, I can imagine writing some client side JavaScript to allow PouchDB (ie a database in the web browser) to save and restore itelf to SAFE storage… or, even better, to sync to/from SAFE which raises the possibility of a SAFE share allowing PouchDB databases on different client devices to share the same database on SAFE.

There is a problem ensuring data integrity with the last one, since a rogue/broken client could write arbitrary data to the shared database. However, one client could safely share the contents of its database as a read only share with multiple other clients, which also creates a lot of possibilities and might be a better solution anyway.

However, the point of this post and me spoiling your Monday morning is to send you the following link, which describes the CouchDB document REST API. As with the remoteStorage API, they require a PUT/POST to be able to specify the expected revision of the currently stored document, and for the call to error if it doesn’t match (eg if the document has already been changed by someone else). I don’t know if this is useful info for you, but here’s the link:

https://wiki.apache.org/couchdb/HTTP_Document_API

1 Like
25

@happybeing, am working on the document still. Once it is ready, it should be added to the docs section, through this week. But the examples should be ready soon. I can explain from that point to clarify even further.

This would be it :smiley: NFS API is a library as such. Our motive is to get the API stable and then port the library to different platforms. Which would enable devs from other platforms to use the API and build their apps/frameworks on top of the API.

3 Likes
26

Preface
WebDev here (I am biased). This is a topic very close to my heart, since I’ve been looking into server-less/server-optional/decentralized webapps for some years now. Maidsafe could be a dream come true for this. Emphasis on could.
I would like to help this along. Partially cause I would love to build webapps on here myself, and partially because I strongly believe that this - a good webapp infrastructure - will be crucial to the widespread success of Maidsafe. Why? It’s a web technology, and you want web developers flocking to your platform to build apps in their spare time, then later convince their boss why they should adopt it as a business. And I don’t think you’ll get as far with C/Rust developers.
But, as I said, I’m biased here.

To help this along, I want to enumerate some systems that will be required to build webapps of all types here on maidsafe:

Persistence:
While I’m a fan of CouchDB, we know that there is not one tool for everything in database terrority. (Here’s a recent refresher on CAP).
In more detail: a CouchDB approach can solve a multitude of problems, but it is lacking in these areas: validation / authorization, and consistency.

For one, ALL the security logic will be in there. Any shared persistency will need a lot of logic of who can CRUD what and when. Which is often the whole point of the server code in single-page-apps. Trying to reduce this to one function (as CouchDB does) which has to throw immediately or forever hold its peace, is not enough for all types of apps.

And for persistency, nobody will ever build a serious webshop with the eventual consistency of CouchDB. That needs support for transactions.

There are other databases which try to have the best of both worlds, like OrientDB, but I haven’t tested that out yet. Might be a good approach though.

But in essence, a web developer needs to be able to write these things, this “server-side-code”. And it might hurt adoption having to learn a new language just to do that.

Real-Time
I don’t know much about the messaging API yet, but I doubt it will be quick enough for low-latency bidirectional data transfer. And I don’t think it has to. It only needs to offer a signaling-framework for an app, so app-users can connect to each other via WebRTC. (Signaling in WebRTC)
And again, such signaling “rooms” needs authorization code specific to the app.

Pseudonymity
An app moderator needs to be able to constrict rogue users from his app without being able to identify their real account name.
On the other hand, to have such a moderator exist, he needs to hardcode his own, unforgeable, user-id into the app ACL code. Again, without revealing anything about his Maidsafe account.

Conclusion for the moment
That’s all I can think off now. Hopefully somebody has already thought of it all among the devs.

Again: Persistence if of course the most important point, but if web devs need to cut their other cool, modern features because of the Maidsafe-platform, I fear they will be reluctant to invest in it.

3 Likes
27

Hi @Stormking it’s great to have another voice, and that you have up to date experience unlike myself. I’m away, so will keep this short, but would encourage you to share your thoughts on options for making an easy web platform on top of SAFE, using the browser plugin to provide access to the SAFE API.

Personally I’m still wrestling with trying to discover useful frameworks and design patterns that we could support this way. BTW I see CouchDB as a template protocol, not as a database engine here, which we could leverage, for example, by helping devs build apps with PouchDB (browser local DB) but using SAFE as a way to sync across devices, or access the applicationy owned data (pages, product listings, users, etc). How to do that sync is proving how little I know ATM.

That’s just an example. If you have any bright ideas I’d love to hear them cos I’m doubtful what I would like to do can work at the moment, but am sure there will be some very neat solutions to this if some bright experiences minds, and mine, get together and start figuring it out. :smile:

I was talking to the developer if Kanso this week and he seemed very interested, and then someone from the Fossil mailing list showed interest, so I have fingers crossed we can get it heads together and come up with something very tempting indeed.

28

Imo, we have to consider a database architecture which allows different users to control different shards/fragments of it.

As long as the software with is robust enough to function when those fragments are down/corrupted etc, then it could work well.

Obviously, easier said than done.

29

Hi @traktion any suggestions, or rabbit holes you’d be interested in looking into?

I’d like to get practical on this - i.e find things that we can do, even if the capabilities are limited. I’ve been looking for some web app patterns such as: simple website CMS, blog, wiki, photo album, video hosting etc and trying to figure out how to make them work with this, or that etc.

The are quite a lot of demo open source web apps around, so if anyone wants to dive in and figure out how to get one working on the SAFE API I think that would be useful. I’m still not sure which approach to try myself, so will probably be investigating more first, but if someone actually builds a demo I think we would all learn a lot.

Otherwise, ideas for how to implement some typical app, within the SAFE browser plugin API, which someone else might pick up would be a good start - more detail the better.

30

Would it ne possible to let à forum engine work with pouchDB?

31

I don’t know of any technology based around relational databases which provide such features at the service level. Tbh, I don’t know of any middleware that does this either, but that isn’t to say it doesn’t exist.

However, I am aware of nosql databases which can be linked to scale out in a similar way. That is, different nodes handle different ranges of data to be stored, such that horizontal scaling can be achieved.

I suspect it would be possible to design a physical data model which maps user data to databases on user vaults. Essentially, the key on the node linking the data would contain the vault ID as well as the data key.

As the underlying safe net technology is key pair storage, it shouldn’t really matter whether the data is in a single vault or spread over many - the entire storage layer is addressable directly.

Imagine something like Facebook, but with all your data stored in your own vault. While it wouldn’t stop people copying the data, you could dictate - at the storage level - who had permission to see it and whether it was deleted or not. Having a coordinating node (which pulls the data together) would need to be aware that this data could come and go and design the coordinating software with that in mind.

Essentially, all you would need is for users to register their vaults with the coordinator and all other database storage could be stored at the user’s vault. This would allow the coordinator to complete the registration process (not time sensitive from performance perspective), then leave the user vault to do the rest.

Recent data changes could literally be a flag associated with the application on a user’s vault. If changes have been made, further queries could be run in the user vault to classify them.

At first thought, there shouldn’t be a big potential for DoS, as vaults are distributed too. Bad actors could be unregistered if they were attempting such, which the coordinating node could dictate.

Just some thoughts, which may be of use anyway!

32

Btw, to add - this isn’t a million miles away from web service oriented architectures. Instead of web service end points, they are nosql end points, which can have access control and be filtered.

33

@gmason Yes. But this would not solve the question of how to share the data across clients - each user would see only their own posts.

34

Ok, this has spawned a proposal to build a Web App to build websites on SAFE Network. Help wanted! See: