Safe Network Community: How long could you lend me 1/4th a cpu core and 256 or 512 MB of RAM?

Related Projects
1

Hey guys,

I’ve got to do a test of Klouds at larger scale than I have access to machines or people to do. I wracked my brain over how to accomplish this, and I came up with VirtualBox, since it will be easy to distribute. All these VirtualBox Images do is boot and connect to a network. Once they are connected to the network, They need to arrange themselves into 20ms concentric circles of “distance.” Right now I know that I have the networking to a point where I need to test it at a large scale, and the self-arrangement/geo-tagging would take maybe another week.

What am I asking of you, the awesome volunteer?

If you chose to help us out with this, then here’s what you’d do:

  1. Download a virtualbox image
  2. Run it

As this is a security conscious community, I shall tell you exactly what will be in the VirtualBox image:

Debian Linux
Docker
Weave
ZeroTierOne

for debug and the like I will need ssh access to the containers, and I’ll add my public key to the authorized_keys file. Provided you ensure that you do NOT somehow change the network mode to “bridged” on the Virtualbox software, then it will give me no access whatsoever to your home network or host PC. I’ll have access to the VM via the weave/zerotier network ONLY.

What am I up to?

I’m trying to build a new kind of infrastructure for applications that serve the “regular, plain-jane, vanilla http web.” So, things like…

  • Wordpress Installations
  • Forums
  • CMS Systems
  • Anything fitting “web app”
  • Mobile app APIs
  • Tor relays & exits
  • Game Servers

To say thanks for helping out in all this, we’re making beta participants an offer: Help us with this by running one of our VMs and then using your PC as you normally would, including reboots and we will:

  1. Give you first shot at hosting apps in our system.
  2. Host one application of your choice for the rest of your life.

This is a great community, and I’m honored to kick off this work with you. Here is a sign-up sheet for the beta:
(google forms signinup link coming momentarily.)

PS:
KLOUDS WILL ONE DAY SHARE SOME CHARACTERISTICS WITH MAIDSAFE, BUT I WOULD LIKE TO MAKE IT EXCEPTIONALLY CLEAR THAT IT IS NOT OUR MISSION OR INTENT TO PROVIDE ANY KIND OF ANONYMITY / SAFETY OTHER THAN ALLOWING USERS TO SUBSCRIBE FOR SERVICES USING CRYPTOCURRENCY AND NOT ASKING THEIR NAMES DURING THE SUBSCRIPTION PROCESS. WE ARE AIMING FOR RESILIENCE AND COMPATIBILITY, NOT PROVABLE ANONYMITY. PLEASE DO NOT CONFUSE KLOUDS WITH MAIDSAFE, AS KLOUDS WOULD NOT OFFER THE SAME CHARACTERISTICS.

PPS:
The purpose of klouds is to reduce the global aggregate cost of computation. While Klouds can host privacy protection tools in docker containers, and it’ll do it without knowing your name, our software does not contain anything beyond what the normal https:// web provides in terms of protection of your identity.

PPPS:
Project Chat server at chat.klouds.org

1 Like
2

Interesting. Possibly interested. Could you link some data about the project and team?

1 Like
3

Most certainly.

Dev Team is presently just two and Tuan in an advisory role:

Business Team:

I should mention that I’m not much of a programmer. Paul slings the gocode, and I get his code into docker images, or connect it with other code that we find in 2016’s wonderful hodgepodge of an open source ecosystem.

So I should mention that if you see this, and you’re handy with Golang, go ahead and drop a line.

This is our very basic daemon, which Paul is currently re-doing, as this one (we knew this all along) wouldn’t be able to scale-- it would not be running on the VMs, at least not at this stage. Presently, it will be centralized, and once we get the 20ms regions worked out, we’ll have two super peers per region ensuring that services meet SLAs.

Klouds is (will be-- waiting on paperwork… :/) a Vietnamese Cong Ty entity, owned in equal portions by:

Myself (Jacob Gadikian)
Paul Mauviel
Tuan Pham Anh
Nguyen Tien Dung

It may also become a United States corporation, but due to the myriad variables, really only time will tell. Klouds has funding in place for about 5 months at twice our current budget, and that’s provided by Dung. Dung is a “legacy businessman” turned technology investor/enthusiast and we’ve been working out the software components of our system for almost a year. Dung has stated his intention to fund the project further if we are not earning more than expenses at the 5 month mark, provided that the whole thing doesn’t go to hell during testing, or _____ or ____ or _____ where _____ means that the project isn’t viable.

If you have any further questions about anything at all, please don’t hesitate to ask.

4

@fergish

Basically we hope that Klouds will be a large, decentralized docker container host that is able to offer the lowest possible levels of latency. I see it as an alternative to GCE/AWS/Azure, and we will be focusing on expanding our footprint in the global south first. We might even use GCE/AWS/Azure for some things in the global north at first, though it’s looking like we will not need to because there’s a large pool of locked away compute that we think we can access: “Legacy Hosting Providers.” My early calculations show that we should be able to offer customers higher performance while givng the legacy hosting providers a combination of:

  • Longer HW lifespan
  • remote management
  • somewhat better monthly earnings (I don’t plan for it to be much better, but it should be better than what they make now on a per-compute-unit basis)

I’ve begun to describe it as “the docker hosting borganism someone had to go and make, because it was the clear next step…” I say higher performance because our production setup does not use Virtual machines and in containerized setups, virtual machines really harm overall performance.

5

Thanks. I think I follow most of this.

Basically you’re talking about a decentralized compute system. The VMs you’re planning to have us install are a way to deploy a test system in a way that is secure and easy for those who are contributing their hardware for testing, but it will be done differently later?

Not sure what you mean by “Legacy Hosting Providers” though.

Details will probably go over my head if you go too far. If my summation above is more or less correct then clarify the Legacy . . . bit and we’ll probably be tracking about as well as I’m capable at this time. :blush:

6

Sure thing. Okay, so-- recently with the introduction of things like:

  • Virtual Machines (not so recent)
  • OpenStack
  • Bare Metal Management Tools
  • Containerization
  • Clustering (in common usage)

…hosting has become much more complex in terms of operational requirements. Because of this, there are companies out there that (today) are financially doing fine, whose owners have not yet relized that they need to adapt or be doomed. They’re starting to realize this, as (especially) containerization brings much, much more efficency. It also brings with it a tendency to use the public cloud, however, because all of the things that we’re baking into that VM image are already done, and in a quite high-grade manner, too. So customers are moving away from either buying VMs (due to compute overhead vs containers, oversubscription, etc) or renting whole servers at hosting providers who, for example, cannot provide any of the network management type stuff needed to have a “cloud.” So, a legacy hosting provider is a hosting provider that falls into any of these categories:

  • Lacks ability to implement a containerized PaaS
  • Lacks user-controlled network management tools
  • Only sells Virtual Instances
  • Cannot manage their metal programatically
  • Hasn’t implemented a private cloud based on OpenStack or (this is of course highly unlikely as openstack enjoys a near-monopoly) one of its competitors

Basically: Hosting providers that lack serious talent are headed down dinosaur alley. All this said, they have large pools of compute, and we have a PXE boot system for their metal. They can continue to operate without paying attention to the direction of the industry, and we can leverage their compute at an extremely low price.

I’ve got to guess that in the US and Europe alone there’s >2000 businesses meeting those criteria, and I’m guessing conservatively.

Then we get into the global south: Today, there is no significant cloud presence anywhere in the global south except for Singapore (well-developed) and Brazil (nascent.) This leads to situations like what is faced frequently in today’s Nigeria: a booming web, with everything running on foreign servers.

7

I am expert on virtualization (KVM / Xen / LXD / Hyper-V) and OpenStack… I can help!

I think we can use Ansible automation and Vagrant…

Even Packer would be great to build ready-to-use images.

VirtualBox sucks, it is unreliable, crashes every day, it is near to useless.

1 Like
8

Can packer images run on top of Windows?

Also, awesome! Here’s the Chat I’m setting up for Klouds, let’s talk :slightly_smiling:

http://chat.klouds.org

9

Windows? What heck is this? :stuck_out_tongue:

Yes, Packer can build RAW images, then, we can use qemu-img to convert it to VDI (VirtualBox), to VHD (Hyper-V Windows Hypervisor), OVA for VMWare, QCoW2 for KVM and OpenStack…

There is no need to lock an image to a specific hypervisor. An image can run anywhere, including directly on bare-metal.

A better environment for developing things like this are an Ubuntu PC (16.04 is already good enough) or a Mac… Windows sucks… And you can’t run Ansible from a Windows machine, and Ansible is the most powerful automation out there…

Currently, I’m working with Packer, Vagrant and Ansible, to do things like this:

  • Developers uses:

Vagrant + Ansible to fire up ready-to-go VMs;

  • Production uses:

Packer + Ansible to build up ready-to-go HDD images.

This way, both dev and prod shares the very same deployment procedure (Ansible playbooks), to create a very consistent environment.

Also, it helps Operators as well, because Ansible playbooks can be a Cronjob, for example, that it will make sure that your environment is always consistent.

10

Windows is needed in this case because of the numbers. Hop in the chat I’ll tell ya all about it. Trust me-- I’d rather go as you’re describing, but few people outside of our bubble run linux :(.

11

I understand that… But I never used Windows in my entire life. To be honest, I don’t know what to do with it… It is a useless brick for me… =P

I know that Packer can build images that runs on Hyper-V (never tested myself but it will work) or on any VirtualBox (Mac, Linux and Windows hosts).

Ubuntu is a very good and stable (LTS) O.S., to be used to power NASA clusters, the SAFE Network, or my grandma’s laptop! :wink:

I hope that SAFE developers do NOT prioritize Windows just because of the numbers, it must work on Linux (specially Ubuntu, I can help with DPKG packaging and Snappy - Rolling Upgrades) and on Mac as well, without losing any feature.

Which chat are you in? IRC?

Cheers!

2 Likes
12

Actually I’ve yet to build the IRC bridge. I’ve got a rocket.chat server up: http://chat.klouds.org/ clients available for Android, ios, linux, mac and windows.

As regarding SAFE devs, I think they are in the same boat I am in: windows ()@&()&$)&(&(#&(#!!! many nasty things (&())(*^(()&(&&_&!!!..

but at the end of the day it’s where the userbase is, and ignoring that would be tantamount to suicide. I imagine that both Klouds and Maidsafe will have to sacrifice some things for windows compatibility-- in fact, that’s why I am running Klouds in a VM on both Windows and Mac. It’s the best way to get access to the awesomeness that is the kernel. I like your qemu idea, though. Is qemu capable of running on mac & win?

13

I’m confused. Is this something that will be made to work with Safenet? Or is it a platform on it’s own with a somewhat similar idea?

14

VirtualBox (and fxxxing Firefox) are two apps I hate the most, but have to use (at work).

Who enjoys a near monopoly? Did you mean Rackspace in Openstack or VMware in virtualization?

15

You may want to look at LXD too, which is a container technology designed to replace Linux VMs, rather than facilitate micro services. It is something I have been experimenting with for our platform at work.

1 Like
16

It’s both and neither.

  1. I’d like to use all of the network’s excess capacity to support the SAFE network.
  2. I think it’s quite possible people may want to run SAFE infrastructure on Klouds
  3. The idea isn’t that similar: Maidsafe is earthshaking stuff. Klouds is merely a play at making the hosting business suck much, much less than it does today, and a part of a strategy to reduce the aggregate cost of computing on a global basis via HW modularization, process improvement, and SW containerization.

Anyway, yes-- I definitely want to bake safenet into Klouds. Why aggregate all of that computing power to let it go to waste?

I don’t think that the whole world will just stop using http-based internet services, and we have a plan to deliver those services in a way that is faster (far less latency, plus no hypervisor overhead) and cheaper than the public cloud model.

@traktion I’ve looked at it but AFAIK it is still too much of a PITA to run docker properly inside of LXC/LXD.

@janitor Openstack enjoys a near monopoly in private/hybrid cloud infrastructure.

1 Like
19

I understand that Ubuntu are integrating LXD with Openstack as a hypervisor option. I haven’t tried that yet, but I managed to get a CentOS LXC container running and our application deployed to, on a local Ubuntu Server virtualbox VM.

I need to do more experimentation with it, but we have loads of qa environments which get little use, yet require full Rackspace VMs to run on. Likewise, it would be nice to breakdown our production environments logically, without the VM overhead , which LXD could help a lot with.

I think it is one to keep an eye on in 2016 though. Docker has all the attention at the moment, but LXD has some great use cases for improving density.

20

@Traktion
Bingo!

Yes, I strongly agree. LXD’s maturation into something that allows me to run multiple docker containers inside of a single LXC container is very exciting indeed. The density improvements are enormous.

@ThiagoCMC

You were dead right about packer. It is a far better solution than the one I had initally planned on, thank you!

PS:

klouds @ http://chat.klouds.org

Hope to see y’all there.

21

Let me know if you need help with Packer! I’m working with it for about a year now.

Specially Packer + Ansible… Something like this:

But what I have today is simpler and easier to use… Let me know if you guys need help with it!

:smiley:

1 Like