SAFE Wiki... Wikipedia on the SAFE Network

Apps
1

Please see my original forum post for more context. :smiley:

Without a doubt in my mind, the most important website in the world is Wikipedia. The importance of having free access to educational material cannot be overstated. For many people around the world, it is a luxury that they can only dream of. In the SAFE Network I saw an opportunity to help bring this luxury to those unfortunate few. Since October of last year, I have been working away on developing my application SAFE Wiki. SAFE Wiki is an Electron powered adaptation of the WebExtension Kiwix-JS that can use the SAFE Network to both store and read ZIM files.

ZIM files are magical in that they provide an ā€œoffline storage [solution] for content coming from the Web.ā€ Specifically, ZIM files can be used to store Wikimedia and StackExchange content.

Kiwix provide a number of tools/programs to easily browse these ZIM files. Kiwix-JS is just one of many projects that Kiwix maintain. If you wish to know more I highly suggest you visit their website.

SAFE-Wiki brings the functionality of the SAFE Network to Kiwix-JS. Once you use SAFE-Wiki to upload a ZIM file, it is publicly available to any other user of SAFE-Wiki. This means that we can easily publish decentralised ā€˜archivesā€™ of popular websites like Wikipedia.

When you upload a ZIM file to the network, it getā€™s stored in what I am calling a ā€˜ZIM folderā€™ that lives in your _public container. When you create a ZIM folder, you specify a name. The name is important as it is the ā€˜nameā€™ of the MD on the network and is how other users will find the ZIM files you upload. This ā€˜ZIM Folderā€™ is stored in your _public container against the key ā€˜zimā€™. To store and read ZIM files, a ā€˜ZIM folderā€™ is accessed through NFS emulation. For exampleā€¦ If you are Wikimedia, you might have a ā€˜ZIM folderā€™ called Wikimedia. Within that MD you store a ZIM file called Wikipedia. For another user to access Wikipedia, all they need to do is type in Wikimedia as the target ā€˜ZIM Folderā€™, and enter Wikipedia as the filename. It is as simple as that!

The main features of the application work. You can upload ZIM files and anyone can browse them! The functionality to delete a file that you have uploaded has also been implemented. I will be introducing further ā€˜house keepingā€™ functionality in due course. Please note there are issues with ZIM files that contain imagesā€¦ A fix for this is my top priority at the moment so I should have it sorted soon.

As some as you will remember, last year at University can be rather busy. The application doesnā€™t have nearly as much polish as I would like. Since the application is now useable and I have overcome many of the big technical challenges, I thought it best to share it with you all. Ugly bugs and all. My project is due near the end of March, so you can expect to see lots of activity over the coming weeks/months as I fix bugs and improve functionality. There is still much to be done! If circumstances allow it, I will continue to work on this project after I have finished my degree. I truly feel that SAFE-Wiki could be a useful tool to many people around the world.

I STRONGLY recommend that you use mock routing to test the application if you so desire. I have done very limited testing with the Alpha-2 test network and I donā€™t feel the application is fully ready to be used on the public network. Completely up to yourself though if you wish to live dangerously :wink:. You can download ZIM files from Kiwix (http://wiki.kiwix.org/wiki/Content), They have a vast array of different ZIM files for you to choose from. Please note that without any configuration of the SAFE Browser you have a file size ā€œlimitā€ of ~1GB, so please use the smaller files if you wish to avoid the trouble. If you are going to download ZIM files, please try the BitTorrent links firstā€¦ Reduces load and cost on their end. I have been working with the SAFE Browser so far, not sure how things function with Peruse so please let me know how you get on if you go down that route. Development has been entirely on macOS, so if you have OS specific bugs then please let me know and I will have it fixed before I hand in my project to the Uni! As this is a final year project at University, I have to be careful about seeking help from outside sources. So if anyone submits any pull requests for features or things like that, please donā€™t be offended if I donā€™t do anything with them until after I have submitted my project/dissertation.

Please find the source here.

Full instructions on how to run the program are in the README, if you have any trouble then please donā€™t hesitate to ask. I did put the instructions through the ā€˜rubber ducky testā€™ but I still might have left some details out. Very excited to hear what you all think. :grinning:

56
561495Ɨ868 182 KB

A ZIM file being uploaded.

15
151391Ɨ926 369 KB

Browsing Wikivoyage. Note there are issues with ZIM files that contain images, hence why this doesnā€™t look very pretty. Fix coming soon!

35
351378Ɨ901 121 KB

Wikispecies. This ZIM file has no images so it displays perfectly.

45 Likes
2

The apps are coming thick and fast now! :sunglasses:

I havenā€™t tried this yet, but it sounds like the basis for getting Wikipedia on SAFENetwork. That would be super cool! :slight_smile:

16 Likes
3

Iā€™ve managed to get it working on Windows without a hitch which is something. Nothing ever compiles on Windows so I count that as a victory in itself. The app works fine. I can read zim files stored to on my hard drive but canā€™t work out how to use it with the mock network - not helped by the fact the web hosting manager doesnā€™t seem to be talking to the mock browser for some reason. Can you give us an idiotā€™s guide? :clown_face:

Nice work by the way - keep us updated on your progress!

8 Likes
4

Seems to be hanging at the authentication stage although there are no obvious errors.

kiwix
kiwix1281Ɨ707 141 KB

1 Like
5

Absolutely! This is brilliant @DaBrown95. This app is one of the most anticipated, so good on you for getting this done. Itā€™s looking great already imo, Iā€™m going to try and give it a go this weekend. Canā€™t wait to see what it will look like polished :smiley: Bravo man, bravo :clap:t3: Iā€™ll be sharing on Twitter

9 Likes
6

Very nice! That momentum rise we all spoke of a few months back is here. With data chains coming into place the flocks are going to storm this forum. Canā€™t wait! :grinning:

Q: any idea of how large the current wikipedia repository is on the clearnet?

8 Likes
7

Hello !!

That is an amazing project, one of which give the Safe network a true meaning.
At some point it could be an idea to tell Sci-Hub people about it ?

Build and launch went flawlessly on linux x64 fedora 26.

I can open zim files and browse them without issues.
When trying to connect to the mock network, the app fails with :

Requesting auth

network.js? [sm]:60 Uncaught (in promise) 
E {message: "Failed to load native libraries: Error: Dynamic Syā€¦libsafe_app.so: undefined symbol: test_create_app", stack: "1ā†µ    at module.exports (/home/d2r2/archives/maidsā€¦-wiki/node_modules/jquery/dist/jquery.js:4991:28)"}

If I try to connect to the real alpha2 network, it just hangs there with :

Requesting auth
8 Likes
8

Awesome! I have kiwix on my phone! For starters bringing snapshots of Wikipedia to safe (and with them free knowledge to all the world) that would be an awesome start before Wikipedia itself decides to utilize the safe network

Ps: @Stark German Wikipedia with pictures is roughly 25gb ā€¦ English will be a little bit larger I guess xD

5 Likes
9

Thank you so much for trying it out! :grinning: If it is Windows you are using it looks like it is an OS specific bug. What happens is SAFE-Wiki sits and waits for a response from SAFE Browser to say, ā€œYou are authenticated now!ā€ That response isnā€™t being heard so it will be an issue with the way I listen to the ā€˜auth-responseā€™ on Windows. Will fix that soon. Thanks for discovering it.:grinning:

10

Thank you very much for the kind words. :smiley: Let me know how you get on!

1 Like
11

The ZIM file for the full English Wikipedia is >75GB last time I checked! The ā€˜realā€™ Wikipedia is obviously much larger when you start to include revision data etc.

3 Likes
12

I had never even head of SciHub beforeā€¦ Interesting suggestion! :smiley:

It is great to hear that builds are running smoothly!

Hmm okay, that is a weird error! Iā€™ll try it on Linux soon and see if I get the same error. Might be an OS related bug. I have seen errors like that before being caused by the version of Node, any idea what version you are using?

Thanks for trying it out!

2 Likes
13

Kiwix on mobile is awesome! Yeah that was a thought that I had. Hopefully can bring SAFE-Wiki to mobile once the mobile authenticator is up to scratch :wink:

3 Likes
14

Yes - not needing to have 75gb of free space for using Wikipedia on your phone would be awesome xD

1 Like
15

sure, this is v8.5.0

16

+1 for Sci-hub, would love to see that sort of thing blossoming early on on Safe

3 Likes
17

Just thought I would post a small updateā€¦ I have fixed the issue with ZIM files containing images.

46
462880Ɨ1800 1010 KB

Here is Wikivoyage displaying correctly.
06
062880Ɨ1800 909 KB

The Ayr page displaying correctlyā€¦ You can check above to see how this originally looked.

53
532880Ɨ1800 907 KB

A page from the Coffee Stack Overflow.

I now successfully have my application running against a local SAFE Network instead of using Mock Routing! It was amazingly simple to get working. I am using four vaults on my local machine at the moment, but I intend to scale this up to use multiple machines on my local network. It is very exciting to see it working for ā€˜realā€™ instead of using Mock Routing!

I will be addressing the problems you all have been having very soon. I havenā€™t forgotten. :grinning:

22 Likes
18

@nice @JPL
After the fixes in bothā€¦

The application should work fine on both Windows and Linux! I have just tested the application running on: macOS, Windows and Ubuntu. Both packaged and running through the command line, all works fine on my end.

Please donā€™t hesitate to let me know if there are any more issues.:grinning:

7 Likes
19

Your latest commit fixed connecting to the live alpha2 network, thank you !

On the other hand, when using mock, I still get :

Requesting auth

network.js? [sm]:60 Uncaught (in promise) 
E {message: "Failed to load native libraries: Error: Dynamic Syā€¦libsafe_app.so: undefined symbol: test_create_app", stack: "1ā†µ    at module.exports (/home/d2r2/archives/maidsā€¦-wiki/node_modules/jquery/dist/jquery.js:4991:28)"}

Did someone upload something on alpha2 ? I tried my luck at a few combinations for collection name and filename, but just got some ā€˜requested data not foundā€™ so far :slight_smile:

2 Likes
20

That is what I like to hear. :slightly_smiling_face:

Hmm that is strangeā€¦ Try a combination of deleting your node_modules and running npm rebuild. I am using Node v8.9.0 and having no issues so maybe try updating your Node version.
An alternative if you are so inclined is to run your own local SAFE Network, if you can connect to Alpha2 you will be able to connect to that. :smiley:

2 Likes