Permissions for apps using safe db

I was looking around and it seems like app development is possible using javascript with maidsafe serving add the back-end storage. I’ve also read that there is some similarity between this and firebase or crouchdb?
Both firebase and crouch have a structure allowing for permissions and data structure to be defined at the database level which eliminates the need for middleware. This is essential for the js/db model to work since any client based application is fundamentally insecure… the html and js can be manipulated before sending output to the db making this essential.
I’m wondering how this would be handled in maidsafe? I like the firebase system as it seems you can create object based permissions and data structures. In crouch it ends up being permissions per db. I’m also not sure (as I’m new to the concept) how I could set up something like a simple voting app using a crouchdb sort of backend as a client could theoretically send +1000 votes if they have write permissions and manipulate the js on their end (and they would need to have write permissions). Now I could see me using a boolean value for that case but if the input needs to be anything but boolean there are issues… like a 5 star feedback app.
I’m also wondering how time is kept… so say i want a decay algorithm for values (an example would be a 5 star feedback vote where older votes decay over time to be less relevant)… How would that work?

I’m just trying to grasp these concepts so I can do as others are; developing with a substitute back end. I get that data is secure but my concern is malicious users manipulating shared data from an app.

1 Like

These are issues we (the SAFEpress team) are grappling with at the moment so I don’t expect you’ll get any easy answers just now - though I’d love to be wrong!

We’re not just trying to build a CMS, but also to create a client side framework on which anyone can build, so it would be great to have you join in our discussions and experiments. Yesterday I submitted an RFC on how the browser plugin could enable dynamic HTML pages in the client. The more heads feeding back on things like this, and on your questions, the better the result well get.

I’m guessing you’ve already read the main thread about SAFEpress (or at least some of it) but if you would like to join the discussion please come aboard.

You don’t have you be building with us to help both you and us at this stage, though if you want to join in some of the building that would be fantastic. I’m about to set up a google group, but if you want to get up to speed join us on http://projectsafe.slack.com and take a look at the activity on http://trello.com/safepress

If you need invites for either PM me with your email.

4 Likes

I wonder if it might be possible to incorporate some of what they have done at http://deployd.com/ which is open source.

After searching a bit it seems like they may be using mongo db anyways as a basis for thier file system… and if js is supported via a wrapper then this could be possible.

1 Like

If so then the free courses from MongoDB University should be your next stop

Could be, nice find. We need more eyes!

Sounds similar to CouchApp and Kanso which used CouchDB. We can learn from all these. Hoodie and the noBackend.org projects too. By all means jump in!

I looked into the coding for the Maidsafe that I had on my PC… this seems to be C++ so pre-rust conversion (and so a bit more understandable to the rust novice).

It appears that data is stored as a “Blob” in a “container”… and these containers have keys and containers have child containers. The only permissions I had seen where that there are storage ID’s which identify a storage instance allowing it to be decrypted. It then seems the Blobs would either be encrypted or not… public or private… and that is about it. In that regard, I’m not sure to what extent permissions would be feasible.

it also seems in regards to time, the system has no way of keeping time so it relies on the clients which can be misconfigured. “> The network currently has no time server of its own, so the timestamps are from the clients. If a client has a misconfigured clock, the timestamps stored will also be incorrect.”

The POSIX API

Maidsafe App Posix API [DRAFT]

NOTE: This API is subject to change.

The Maidsafe Posix API strives to be consistent, and high performant.

Storage Abstractions

Blobs

Data on the SAFE network is stored in Blobs. A Blob can contain text or binary data, and the SAFE network has no upward limit on size. However, local system contraints may apply to maximum size. Each Blob is immutable, once it is stored it cannot be modified.

Container

A Container stores Blobs or a pointer to another Container at keys that have no restrictions (any sequence of bytes are valid). Each key is versioned, so past Blobs or Container pointers can be retrieved (which gives the appearance of mutability since a new Blobs can be stored at an existing key).

Most users should not use nested Containers, see nested containers.

Nested Containers and Blob Forks

The chunk information for each Blob is stored directly in the Container, but only a reference ID (a pointer) is stored for child Containers. Since a child Container is a pointer to another Container on the network, a key can have multiple reference IDs stored in its history for a child Container. If the client treats children Containers as directories on a local filesystem, the result can be a fork in the history. The problem is if a child Container is deleted and re-created while another process is writing to the same Container:

                        Container(users)-->V1["user1":foo]-->V2["user1:foo2] 
                        / 
Container(root)-->V1["users"]-->V2[]-->V3["users"] 
                                              \ 
                                             Container(users)-->V1["user1":bar] 

If treated as a filepath, “/users/user1” would have two different histories depending on what version of the root was retrieved. Clients are encouraged to only create a container at the top-level, and rarely delete them. Advanced clients will have to handle these data fork issues; no mechanism for detecting forks and reacting to them currently exists.

Container Keys != Filesystem Paths

Containers are nested, but they cannot be used like paths. You cannot open “/container1/container2/document”; a “/” character has no special meaning in a Container key. This is intentional, nested containers are complicated, and should generally be avoided.

Storage

Storage has 0 more Containers. The Storage can be public, private, or privately-shared.

StorageID

A StorageID identifies a particular Storage instance on the SAFE network, and contains the necessary information to decrypt the contents.

Examples

Network operations in the Posix API are done asynchronously so that client code doesn’t block while network operations are being performed. The Posix API uses the AsyncResult framework from boost::asio. This allows clients to use callbacks, stackful co-routines, or futures as the completion signalling mechanism.

Hello World (Callbacks)

bool HelloWorld(maidsafe::nfs::Storage& storage) { 
  using ExpectedContainer = 
      boost::expected<maidsafe::nfs::Container, std::error_code>; 
  using ExpectedBlob = 
      boost::expected<maidsafe::nfs::LocalBlob, std::error_code>; 
  using ExpectedVersion = 
      boost::expected<maidsafe::nfs::BlobVersion, std::error_code>; 
  using FutureExpectedString = 
      maidsafe::nfs::Future<maidsafe::nfs::ExpectedBlobOperation<std::string>>; 

boost::promise<boost::expected<std::string, std::error_code>> result;

storage.OpenContainer(
“example_container”,
maidsafe::nfs::ModifyContainerVersion::Create(),
[&result](ExpectedContainer container) {

    if (!container) { 
      result.set_value(boost::make_unexpected(container.error())); 
      return; 
    } 
    container->OpenBlob( 
        "example_blob", 
        maidsafe::nfs::ModifyBlobVersion::Create(), 
        [&result, container](ExpectedBlob blob) { 
          if (!blob) { 
            result.set_value(boost::make_unexpected(blob.error())); 
            return; 
          } 
          blob->Write(boost::asio::buffer("hello world"), []{}); 
          blob->Commit( 
              [&result, container](ExpectedVersion version) { 
                if (!version) { 
                  result.set_value(boost::make_unexpected(version.error())); 
                  return; 
                } 
                container->Read("example_blob", *std::move(version)).then( 
                    [&result](FutureExpectedString future_read) { 
                      const auto read = future_read.get(); 
                      if (!read) { 
                        result.set_value(boost::make_unexpected(read.error().code()); 
                        return; 
                      } 
                      result.set_value(std::move(read)->result()); 
                    }); 
              }); 
        }); 
  }); 

const auto value = result.get_promise().get();
if (!value) {
std::cerr << value.error().message() << std::endl;
return false;
}

std::cout << *value << std::endl;
return true;
}

### Hello World (Stackful Co-routines and Monadic) ### 
```c++ 
bool HelloWorld(maidsafe::nfs::Storage& storage) { 
  maidsafe::nfs::Future<boost::expected<std::string, std::error_code>> result; 

boost::asio::spawn([]{}, [&result](boost::asio::yield_context yield) {
result.set_value(
storage.OpenContainer(
“example_container”, maidsafe::nfs::ModifyContainerVersion::Create(), yield).bind(

            [&yield](maidsafe::nfs::Container container) { 
              return container.OpenBlob( 
                  "example_blob", maidsafe::nfs::ModifyBlobVersion::Create(), yield); 
            } 
    ).bind([&yield](maidsafe::nfs::LocalBlob blob) { 
      blob.Write(boost::asio::buffer("hello world"), []{}); 
      return blob.Commit(yield).bind( 
          [&yield, &blob](maidsafe::nfs::BlobVersion) { 
            std::string buffer; 
            buffer.resize(blob.size()); 
            return blob.Read(boost::asio::buffer(&buffer[0], buffer.size()), yield).bind( 
                [&yield, &buffer](const std::size_t read_size) { 
                  buffer.resize(read_size); 
                  return buffer; 
                }); 
          }); 
    })); 
}); 

const auto value = result.get();
if (!value) {
std::cerr << "Error: " << value.error().message() << std::endl;
return false;
}

std::cout << *value << std::endl;
return true;
}

Posix Style API

All public functions in this API provide the strong exception guarantee. All public const methods are thread-safe.

StorageID

maidsafe/nfs/storage_id.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Represents the StorageID abstraction listed above. Obtaining relevant StorageID objects are out of the scope of this document.

class StorageID { /* No Public Elements */ }; 

BlobVersion

maidsafe/nfs/blob_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Blobs stored at the same key are differentiated/identified by a BlobVersion object. The BlobVersion allows Posix API users to retrieve older revisions of Blobs, or place constraints on operations that change the blob associated with a key.

class BlobVersion { 
  static BlobVersion Defunct(); 
}; 
  • Defunct()
    • Returns a BlobVersion that is used to indicate a deleted Blob. This is only used when retrieving the history of the Blobs.

ContainerVersion

maidsafe/nfs/container_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Each time a Blob is stored, or a container pointer is modified, a new version of the parent Container is created. The ContainerVersion object allows users of the Posix API to reference specific versions of the Container.

class ContainerVersion { 
  static ContainerVersion Defunct(); 
}; 
  • Defunct()
    • Returns a ContainerVersion that is used to indicate a deleted Container. This is only used when retrieving the history of Containers.

ModifyBlobVersion

maidsafe/nfs/modfy_blob_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Operations in Container that change the Blob stored at a key require a ModifyBlobVersion object.

class ModifyBlobVersion { 
  static ModifyBlobVersion Create(); 
  static ModifyBlobVersion Latest(); 
  ModifyBlobVersion(BlobVersion); 
}; 
  • Create()
    • Returns an object that indicates the Posix API should only succeed if the specified key is unused.
  • Latest()
    • Returns an object that indicates the Posix API should overwrite any existing Blob at the specified key.
  • ModifyBlobVersion(BlobVersion)
    • Creates an object that indicates the Posix API should only overwrite the Blob at the specified key if it matches the BlobVersion.

RetrieveBlobVersion

maidsafe/nfs/retrieve_blob_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Operations in Container that retrieve a Blob stored at a key require a RetrieveBlobVersion object.

class RetrieveBlobVersion { 
  static RetrieveBlobVersion Latest(); 
  RetrieveBlobVersion(BlobVersion); 
}; 
  • Latest()
    • Returns an object that indicates the Posix API should retrieve the latest Blob stored at the specified key.
  • RetrieveBlobVersion(BlobVersion)
    • Creates an object that indicates the Posix API needs to retrieve a specific Blob version stored at the specified key.

ModifyContainerVersion

maidsafe/nfs/modfy_container_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Operations in Container or Storage that change the Container stored at a key require a ModifyContainerVersion object.

class ModifyContainerVersion { 
  static ModifyContainerVersion Create(); 
  static ModifyContainerVersion Latest(); 
  ModifyContainerVersion(ContainerVersion); 
}; 
  • Create()
    • Returns an object that indicates the Posix API should only succeed if the specified key is unused.
  • Latest()
    • Returns an object that indicates the Posix API should overwrite any existing Container at the specified key.
  • ModifyBlobVersion(BlobVersion)
    • Creates an object that indicates the Posix API should only overwrite the Container at the specified key if it matches the ContainerVersion.

RetrieveContainerVersion

maidsafe/nfs/retrieve_container_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Operations in Container or Storage that retrieve a Container stored at a key require a RetrieveContainerVersion object.

class RetrieveContainerVersion { 
  static RetrieveContainerVersion Latest(); 
  RetrieveContainerVersion(ContainerVersion); 
}; 
  • Latest()
    • Returns an object that indicates the Posix API should retrieve the latest Container stored at the specified key.
  • RetrieveBlobVersion(BlobVersion)
    • Creates an object that indicates the Posix API needs to retrieve a specific Container version stored at the specified key.

maidsafe::nfs::Future

maidsafe/nfs/future.h

  • [x] Thread-safe Public Functions
  • [ ] Copyable
  • [x] Movable

Currently maidsafe::nfs::Future is a boost::future object, but this may be changed to a non-allocating design. It is recommended that you use the typedef (maidsafe::nfs::Future) in case the implementation changes.

In the Posix API, the Future will only throw exceptions on non-network related errors (std::bad_alloc, std::bad_promise, etc.). Values and network related errors are returned in a boost::expected object.

template<typename T> 
using Future = boost::future<T>; 

maidsafe::nfs::Storage

maidsafe/nfs/storage.h

  • [x] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

This object has a single shared_ptr, and is shallow-copied. This makes it extremely quick to copy.

Represents the Storage abstraction listed above. Constructing a Storage object requires a StorageID object.

Parameters labeled as AsyncResult<T> affect the return type of the function, and valid values are:

  • A callback in the form void(boost::expected<T, std::error_code>); return type is void
  • A boost::asio::yield_context object; return type is boost::expected<T, std::error_code>.
  • A maidsafe::nfs::use_future; return type is maidsafe::nfs::Future<boost::expected<T, std::error_code>>.
class Storage { 
  unspecified GetVersions(AsyncResult<std::vector<ContainerVersion>>); 

unspecified GetContainers(
RetrieveContainerVersion, AsyncResult<std::vectorstd::string>);

unspecified OpenContainer(std::string, ModifyContainerVersion, AsyncResult);
unspecified DeleteContainer(std::string, RetrieveContainerVersion, AsyncResult<>);
};

- **GetVersions(AsyncResult<std::vector<ContainerVersion>>)** 
  - Request the version history of Storage. 
  - AsyncResult is given the version history of Storage. A new version is created each time a Container is created or deleted. Oldest `ContainerVersion` is always `ContainerVersion::Defunct()`, and is used subsequently when the key had no associated Container for some period of time. `std::vector::begin()` will be the newest `ContainerVersion`, and `std::vector::end() - 1` will have the oldest `ContainerVersion` (which is always `ContainerVersion::Defunct()`). 
- **GetContainers(RetrieveContainerVersion, AsyncResult<std::vector<std::string>>)** 
  - Request the list of nested Containers. 
  - AsyncResult is given the list of nested containers. 
- **OpenContainer(std::string, ModifyContainerVersion, AsyncResult<Container>)** 
  - Make a request to open a container at the specified key. 
  - AsyncResult is given the nested `Container` with the specified name. 
- **DeleteContainer(std::string, RetrieveContainerVersion, AsyncResult<>)** 
  - Make a request to delete a container at the specified key. 

maidsafe::nfs::Container

maidsafe/nfs/container.h

  • [x] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

This object has a single shared_ptr, and is shallow-copied. This makes it extremely quick to copy.

Represents the Container abstraction listed above. Constructing a Container object cannot be done directly; Container objects can only be retrieved from Storage::OpenContainer.

Parameters labeled as AsyncResult<T> affect the return type of the function, and valid values are:

  • A callback in the form void(boost::expected<T, std::error_code>); return type is void
  • A boost::asio::yield_context object; return type is boost::expected<T, std::error_code>.
  • A maidsafe::nfs::use_future; return type is maidsafe::nfs::Future<boost::expected<T, std::error_code>>.
class Container { 
  unspecified GetVersions(AsyncResult<std::vector<ContainerVersion>>); 

unspecified GetContainers(
RetrieveContainerVersion, AsyncResult<std::vectorstd::string>);

unspecified GetBlobs(
RetreieveContainerVersion,
AsyncResult<std::vector<std::pair<std::string, BlobVersion>>>);

unspecified OpenContainer(std::string, ModifyContainerVersion, AsyncResult);
unspecified OpenBlob(std::string, ModifyBlobVersion, AsyncResult);

unspecified DeleteContainer(std::string, RetrieveContainerVersion, AsyncResult<>);
unspecified DeleteBlob(std:string, ModifyBlobVersion, AsyncResult<>);

unspecified Copy(
const LocalBlob& from, std::string to, ModifyVersion, AsyncResult);
};

A key can only store a Blob or a nested Container at a given point in time.

  • GetVersions(AsyncResult<std::vector>)
    • Request the version history of Container.
    • AsyncResult is given the version history of Container. A new version is created each time a Container is created or deleted. Oldest ContainerVersion is always ContainerVersion::Defunct(), and is used subsequently when the key had no associated Container for some period of time. std::vector::begin() will be the newest ContainerVersion, and std::vector::end() - 1 will have the oldest ContainerVersion (which is always ContainerVersion::Defunct()).
  • GetContainers(RetrieveContainerVersion, AsyncResult<std::vectorstd::string>)
    • Request the list of nested Containers.
    • AsyncResult is given the list of nested containers.
  • GetBlobs(RetreieveContainerVersion, AsyncResult<std::vector<std::pair<std::string, BlobVersion>>>)
    • Request the list of Blobs.
    • AsyncResult Retrieves the names of Blobs in the Container. The BlobVersion is provided for each Blob.
  • OpenContainer(std::string, ModifyContainerVersion, AsyncResult)
    • Make a request to open a container at the specified key.
    • AsyncResult is given the nested Container with the specified name.
  • OpenBlob(std::string, ModifyBlobVersion, AsyncResult)
    • Make a request to open a Blob at the specified key.
    • AsyncResult is given a LocalBlob that represents the Blob at the specified key.
  • DeleteContainer(std::string, RetrieveContainerVersion, AsyncResult<>)
    • Make a request to delete the Container at the specified key.
  • DeleteBlob(std:string, ModifyBlobVersion, AsyncResult<>)
    • Make a request to delete the Blob at the specified key.
  • Copy(const LocalBlob& from, std::string to, ModifyVersion, AsyncResult)
    • Make a request to copy the contents of the LocalBlob to the specified key.
    • AsyncResult is given the new LocalBlob associated with the destination of the copy.

maidsafe::nfs::LocalBlob

maidsafe/nfs/local_blob.h

  • [ ] Thread-safe Public Functions
  • [ ] Copyable
  • [x] Movable

Upon initial creation, LocalBlob represents a Blob stored at a key/version in the associated Container object. Write calls are reflected immediately in that object, but the LocalBlob becomes unversioned because it does not represent a Blob on the network. The current LocalBlob can be saved to the network with a call to a LocalBlob::Commit, and success of the async operation indicates that the LocalBlob now represents the new version returned.

Function State After Throw State After Return State after Successful Async Operation
Read Valid and Unchanged. Unchanged. Unchanged (buffer has requested contents from LocalBlob).
Write Valid and Unchanged. Unversioned. Buffer can be read from LocalBlob. Buffer has been copied, but not visible to remote Blobs.
Truncate Valid and Unchanged. Unversioned. Truncation can be read from LocalBlob. N/A
Commit Valid and Unchanged. Unchanged. Local changes are visible to remote Blobs. Version provided to AsyncResult is ModifyBlobVersion::Latest().

Since write operations are reflected immediately in the LocalBlob object, users do not have to wait for the previous operation to complete to make additional read or write calls. The AsyncResult object provided to LocalBlob::Write calls is notified when the data has been safely copied. Writes stored on the network are hidden from other clients until the async operation for LocalBlob::Commit succeeds.

If a LocalBlob is unversioned, the async operation for LocalBlob::Commit will wait for all uncompleted LocalBlob::Write or LocalBlob::Truncate calls to complete, and then try to store the new Blob version. If LocalBlob::Commit signals failure to the AsyncResult<>, all subsequent calls to LocalBlob::Commit will continue to fail, however subsequent LocalBlob::Write or LocalBlob::Truncate operations can succeed. Changes to the LocalBlob object can always be be stored with Container::Copy, which will wait for any remaining write calls to complete, and then commit a new version.

If multiple LocalBlob objects are opened within the same process, they are treated no differently than LocalBlob objects opened across different processes or even systems. Simultaneous reads can occur, and simultaneous writes will result in only one of the LocalBlob objects successfully writing to the network (the first to successfully call LocalBlob::Commit). All other LocalBlob objects become permanently unversioned.

Parameters labeled as AsyncResult<T> affect the return type of the function, and valid values are:

  • A callback in the form void(boost::expected<T, std::error_code>); return type is void
  • A boost::asio::yield_context object; return type is boost::expected<T, std::error_code>.
  • A maidsafe::nfs::use_future; return type is maidsafe::nfs::Future<boost::expected<T, std::error_code>>
class LocalBlob { 
 public: 
  typedef detail::MetaData::TimePoint TimePoint; 

const std::string& key() const; // key associated with Blob
std::uint64_t size() const;
TimePoint creation_time() const;
TimePoint head_write_time() const; // write time of this revision

const std::string& user_metadata() const;
void set_user_metadata(std::string);

// Version at open
const BlobVersion& head_version() const;

unspecified GetVersions(AsyncResult<std::vector>);

std::uint64_t offset() const;
void set_offset(std::uint64_t);

unspecified Read(boost::asio::buffer, AsyncResultstd::uint64_t);
unspecified Write(boost::asio::buffer, AsyncResult<>);
void Truncate(std::uint64_t);

unspecified commit(AsyncResult);
};

The network currently has no time server of its own, so the timestamps are from the clients. If a client has a misconfigured clock, the timestamps stored will also be incorrect.

  • key()
    • Returns the key associated with the Blob
  • size()
    • Returns the size of the LocalBlob in bytes. This is not necessarily the size of any Blob stored on the network.
  • creation_time()
    • Returns the timestamp of when key() last went from storing nothing to storing a Blob.
  • head_write_time()
    • Returns the timestamp of when the head_version() was stored.
  • user_metadata()
    • Returns the user metadata being stored.
  • set_user_metadata(std::string)
    • Sets the user metadata. Binary data is allowed.
  • head_version()
    • Returns the version from when the LocalBlob was opened. This is not updated after a Commit succeeds.
  • GetVersions(AsyncResult<std::vector>)
    • Request the version history of the Blob.
    • AsyncResult is given the version history of BlobVersions at the key. Oldest BlobVersion is always BlobVersion::Defunct(), and is used subsequently when the key had no associated Blob for some period of time. std::vector::begin() will be the newest BlobVersion, and std::vector::end() - 1 will have the oldest BlobVersion (which is always BlobVersion::Defunct()).
  • offset()
    • Returns the offset that will be used by the next Read, Write, or Truncate call.
  • set_offset(std::uint64_t)
    • Change the value returned by offset().
  • Read(boost::asio::buffer, AsyncResultstd::uint64_t)
    • Read from the LocalBlob starting at offset() into the provided buffer. The buffer must remain valid until AsyncResult returns.
    • offset() is immediately updated to min(file_size() - offset(), offset() + buffer::size())
    • AsyncResult is given the number of bytes actually read.
    • Can be invoked before other calls to Read, Write, Truncate, or Commit complete.
  • Write(boost::asio::buffer, AsyncResult<>)
    • Write to the LocalBlob starting at offset() from the provided buffer. The buffer must remain valid until AsyncResult returns.
    • offset() is immediately updated to offset() + buffer::size()
    • Can be invoked before other calls to Read, Write, Truncate, or Commit complete.
  • Truncate(std::uint64_t, AsyncResult<>)
    • Change the size of the LocalBlob to offset() + size bytes.
    • offset() is immediately updated to offset() + size
    • Can be invoked before other calls to Read, Write, Truncate, or Commit complete.
  • Commit(AsyncResult)
    • Make a request to store the contents of the LocalBlob at key()
    • Storing a LocalBlob at key() will fail if another LocalBlob modified key() since head_version().
    • AsyncResult is given the BlobVersion of the new Blob stored on the network.
    • Can be invoked before other calls to Read, Write, Truncate, or Commit complete.

And the REST API

Maidsafe App REST API [DRAFT]

NOTE: This API is subject to change.

The Maidsafe REST API strives to be easy to use, consistent, and flexible for advanced users requiring performance.

Storage Abstractions

Blobs

Data on the SAFE network is stored in Blobs. A Blob can contain text or binary data, and the SAFE network has no upward limit on size. However, local system contraints may apply to maximum size. Each Blob is immutable, once it is stored it cannot be modified.

Container

A Container stores Blobs at keys that have no restrictions (any sequence of bytes are valid). Each key is versioned, so past Blobs can be retrieved (which gives the appearance of mutability since a new Blobs can be stored at an existing key).

Storage

Storage has 0 more Containers. The Storage can be public, private, or privately-shared.

StorageID

A StorageID identifies a particular Storage instance on the SAFE network, and contains the necessary information to decrypt the contents.

Behavior Overview

Every REST API function call that requires a network operation returns a Future<T> object. This prevents the interface from blocking, and provides an interface for signalling completion. Every Future<T> object in the REST API returns an expected object, which either holds the result of the operation or a network related error. An expected object allows for exception-style programming, return-code style programming, or monadic style programming. When an expected object contains a successful operation, it will have the result of the operation and version information. When an expected object contains a failed operation, it will contain an error code and a retry mechanism that returns a Future<T> with the same type as the original Future<T>.

Examples

Hello World (Exception Style)

bool HelloWorld(maidsafe::nfs::Storage& storage) { 
  try { 
    maidsafe::nfs::Container container( 
        storage.OpenContainer("example_container").get().value().result()); 
const auto put_operation = container.Put( 
    "example_blob", "hello world", maidsafe::nfs::ModifyBlobVersion::Create()).get(); 
const auto get_operation = container.Get( 
    "example_blob", put_operation.value().version()).get(); 
std::cout << get_operation.value().result() << std::endl; 

}
catch (const std::runtime_error& error) {
std::cerr << "Error : " << error.what() << std::endl;
return false;
}
catch (…) {
std::cerr << “Uknown Error” << std::endl;
return false;
}

return true;
}

The `.get()` calls after `GetContainer`, `Put` and `Get` indicate that the process should wait until the SAFE network successfully completes the requested operation (the `.get()` is called on the `Future<T>` object). The `Future<T>` object allows a process to make additional requests before prior requests have completed (see [Hello World Concatenation](#hello-world-concatenation)). If the above example issued the `Get` call without waiting for the `Put` `Future<T>` to signal completion, the `Get` could've failed. So the `Future<T>` will signal when the result of that operation can be seen by calls locally or remotely. 

The Future<T> returns a boost::expected object. In this example, exception style error-handling was used, so .value() was invoked on the boost::expected. The .value() function checks the error status, and throws std::system_error if the boost::expected object has an error instead of a valid operation.

The Put call uses ModifyBlobVersion::Create() to indicate that it is creating and storing a new file. If an existing file exists at example_blob, then an exception will be thrown in the Get call because put_operation contains an error (so after running this program once, all subsequent runs should fail). The Get call uses the BlobVersion returned by the Put, guaranteeing that the contents from the original Put (“hello world”), are retrieved. Alternatively, RetrieveBlobVersion::Latest() could’ve been used instead, but if another process or thread updated the file, the new contents would be returned, which may not be “hello world” as desired.

Hello World Retry (Return-Code Style)

namespace { 
  template<typename Result> 
  boost::optional<maidsafe::nfs::BlobOperation<Result>> GetOperationResult( 
      maidsafe::nfs::ExpectedBlobOperation<Result> operation) { 
    while (!operation) { 
      if (operation.error().code() != std::errc::network_down) { 
        std::cerr << 
            "Error: " << operation.error().code().message() << std::endl; 
        return boost::none; 
      } 
      operation = operation.error().Retry().get(); 
    } 
    return *operation; 
  } 
} 

bool HelloWorld(maidsafe::nfs::Container& storage) {
const boost::optional<maidsafe::nfs::BlobOperation<>> put_operation(
GetOperationResult(
storage.Put(
“example_blob”, “hello world”, maidsafe::nfs::ModifyBlobVersion::Create()).get()));
if (put_operation) {
const boost::optional<maidsafe::nfs::BlobOperationstd::string> get_operation(
GetOperationResult(
storage.Get(
“example_blob”, put_operation->version()).get()));
if (get_operation) {
std::cout << get_operation->result() << std::endl;
return true;
}
}
return false;
}

This example starts from the `Container` object for brevity. It is identical to the [hello world](#hello-world) example, except `Put` and `Get` operations that failed due to the network being down (no connection) are retried. In production code you may want to limit the attempts, or have a signal that indicates the return of network connectivity. 

If the retry mechanism returns std::errc::not_supported then no retry is possible. It is important that clients check the error code after a retry, or clients could continually attempt an operation that will never succeed.

Hello World (Monad Style)

bool HelloWorld(const maidsafe::nfs::Storage& storage) { 
  namespace nfs = maidsafe::nfs; 

return nfs::monadic(storage.OpenContainer(“example_container”).get()).bind(

  [](nfs::ContainerOperation<nfs::Container> open_operation) { 
    return nfs::monadic( 
        open_operation.result().Put( 
            "example_blob", "hello world", nfs::ModifyBlobVersion::Create()).get()).bind( 
            [&open_operation](nfs::BlobOperation<> put_operation) { 
              return nfs::monadic( 
                  open_operation.result().Get("example_blob", put_operation.version()).get()); 
            }).bind([](nfs::BlobOperation<std::string> get_operation) { 
              std::cout << get_operation.result() << std::endl; 
            }); 
  }).catch_error([](std::error_code error) { 
    std::cerr << "Error: " << error.message() << std::endl; 
    return boost::make_unexpected(error); 
  }).valid(); 

}

This is an example of monadic programming, which is better described in the [Expected](#expected) documentation. The callbacks provided to the `bind` function calls are only invoked if the operation was successful, and the `catch_error` callback is only invoked if *any* of the previous operations failed. This eliminates the need for client code to check for errors after each operation. Also, in this example all values are *moved*, not copied, so it is efficient as well. 

Using monadic programming with boost expected will require the usage of maidsafe::nfs::monadic.

Hello World Concatenation

bool HelloWorld(maidsafe::nfs::Container& container) { 
  auto put_part1 = container.Put( 
      "split_example/part1", "hello ", maidsafe::nfs::ModifyBlobVersion::Create()); 
  auto put_part2 = container.Put( 
      "split_example/part2", "world", maidsafe::nfs::ModifyBlobVersion::Create()); 

const auto put_part1_result = put_part1.get();
const auto put_part2_result = put_part2.get();

if (put_part1_result && put_part2_result) {
auto get_part1 = container.Get(“split_example/part1”, put_part1_result->version());
auto get_part2 = container.Get(“split_example/part2”, put_part2_result->version());

const auto get_part1_result = get_part1.get(); 
const auto get_part2_result = get_part2.get(); 
if (get_part1_result && get_part2_result) { 
  std::cout << 
    get_part1_result->result() << 
    get_part2_result->result() << 
    std::endl; 
  return true; 
} 

}

return false;
}

In this example, both `Put` calls are done in parallel, and both `Get` calls are done in parallel. Each `Get` call cannot be requested until the corresponding `Put` operation completes. Also, these files are **not** stored in a child `Container` called "split_example", but are stored in the `container` object directly. 

This examples uses the -> operator on the boost::expected object instead of .value() like in the exception example. The -> operator does not check if the boost::expected has an error (similar to -> being unchecked for boost::optional); the conversion to bool in the if statement is the check for validity.

REST Style API

All public functions listed in this API provide the strong exception guarantee. All public const methods are thread-safe.

StorageID

maidsafe/nfs/storage_id.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Represents the StorageID abstraction listed above. Obtaining relevant StorageID objects are out of the scope of this document.

class StorageID { /* No Public Elements */ }; 

BlobVersion

maidsafe/nfs/blob_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Blobs stored at the same key are differentiated/identified by a BlobVersion object. The BlobVersion allows REST API users to retrieve older revisions of Blobs, or place constraints on operations that change the blob associated with a key.

class BlobVersion { 
  static BlobVersion Defunct(); 
}; 
  • Defunct()
    • Returns a BlobVersion that is used to indicate a deleted Blob. This is never returned by a BlobOperation, and is only used when retrieving the history of the Blobs stored at a key.

ContainerVersion

maidsafe/nfs/container_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Containers are also versioned, but none of the REST API functions accept a ContainerVersion. This class is mentioned/returned by Container operations for users that wish to use the Posix API in some situations.

class ContainerVersion { /* No Public Elements */ }; 

ModifyBlobVersion

maidsafe/nfs/modfy_blob_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Operations in Container that change the Blob stored at a key require a ModifyBlobVersion object.

class ModifyBlobVersion { 
  static ModifyBlobVersion Create(); 
  static ModifyBlobVersion Latest(); 
  ModifyBlobVersion(BlobVersion); 
}; 
  • Create()
    • Returns an object that indicates the REST API should only succeed if the specified key is unused.
  • Latest()
    • Returns an object that indicates the REST API should overwrite any existing Blob at the specified key.
  • ModifyBlobVersion(BlobVersion)
    • Creates an object that indicates the REST API should only overwrite the Blob at the specified key if it matches the BlobVersion.

RetrieveBlobVersion

maidsafe/nfs/retrieve_blob_version.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Operations in Container that retrieve a Blob stored at a key require a RetrieveBlobVersion object.

class RetrieveBlobVersion { 
  static RetrieveBlobVersion Latest(); 
  RetrieveBlobVersion(BlobVersion); 
}; 
  • Latest()
    • Returns an object that indicates the REST API should retrieve the latest Blob stored at the specified key.
  • RetrieveBlobVersion(BlobVersion)
    • Creates an object that indicates the REST API needs to retrieve a specific Blob version stored at the specified key.

ContainerOperation

maidsafe/nfs/container_operation.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Most REST API users should only need to invoke the result() function to retrieve a Container returned in a Storage::OpenContainer call. A version is returned for consistency with BlobOperations, and for users that wish to use the Posix API in some situations.

template<typename T = void> 
class ContainerOperation { 
  const ContainerVersion& version() const; 
  const T& result() const; // iff T != void 
}; 

BlobOperation

maidsafe/nfs/container_operation.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

Every operation on Blobs return a BlobOperation object.

template<typename T = void> 
class BlobOperation { 
  const BlobVersion& version() const; 
  const T& result() const; // iff T != void 
}; 
  • version
    • Returns the BlobVersion involved in the operation.
  • result()
    • If the operation is NOT void, this will return the result of the operation. Compile error on void.

OperationError

maidsafe/nfs/operation_error.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

This object is returned if a network operation fails.
In the event of a failure, retrieving the cause of the error and a Retry attempt can be done with the OperationError interface. The error is a std::error_code object, and the retry attempt will return a new Future object with the exact type of the previous failed attempt.

template<typename ExpectedOperation> 
class OperationError { 
  using RetryResult = Future<boost::expected<ExpectedOperation, OperationError<ExpectedOperation>>; 
  RetryResult Retry() const; 
  std::error_code code() const; 
}; 
  • Retry
    • Return a Future to another attempt at the failed operation. Be careful of infinite loops - some operations could fail indefinitely (ModifyBlobVersion::Create() for example). If the retry returns an error code std::errc::not_supported, then a retry is not possible.
  • code()
    • Return error code for the failed operation.

Future

maidafe/nfs/future.h

  • [x] Thread-safe Public Functions
  • [ ] Copyable
  • [x] Movable

Currently maidsafe::nfs::Future is a boost::future object, but this may be changed to a non-allocating design. It is recommended that you use the typedef (maidsafe::nfs::Future) in case the implementation changes.

In the REST API, the Future will only throw exceptions on non-network related errors (std::bad_alloc, std::bad_promise, etc.). Values and network related errors are returned in a boost::expected object.

template<typename T> 
using Future = boost::future<T>; 

Expected

When a network operation has completed, the future will return a boost::expected object. On network errors, the boost::expected object will contain a OperationError object, and on success the object will contain a BlobOperation or a ContainerOperation object depending on the operation requested. For convenience, the templated types ExpectedContainerOperation<T> and ExpectedBlobOperation<T> are provided, where T is the result of the operation (i.e. a std::string on a Get request). Both types assume OperationError as the error object for the operation.

ExpectedContainerOperation

maidsafe/nfs/expected_container_operation.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable
template<typename T = void> 
using ExpectedContainerOperation = 
    boost::expected<ContainerOperation<T>, OperationError<ContainerOperation<T>>>; 

ExpectedBlobOperation

maidsafe/nfs/expected_blob_operation.h

  • [ ] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable
template<typename T = void> 
using ExpectedBlobOperation = 
    boost::expected<BlobOperation<T>, OperationError<BlobOperation<T>>>; 

Monadic

The REST API returns ExpectedContainerOperation<T> or ExpectedBlobOperation<T> objects which use an error type that depends on T. This makes monadic programming difficult because the unwrap functions in boost::expected will not work as desired. The REST API includes some standalone functions that return a boost::expected object with a consistent error type, std::error_code. After removing the OperationError<T>, retrying the failed operation is not possible. An example of the standalone functions in use.

maidsafe/nfs/expected_container_operation.h

template<typename T> 
boost::expected<BlobOperation<T>, std::error_code> monadic( 
    const ExpectedBlobOperation<T>& expected); 

template
boost::expected<BlobOperation, std::error_code> monadic(
ExpectedBlobOperation&& expected);

maidsafe/nfs/expected_blob_operation.h

template<typename T> 
boost::expected<ContainerOperation<T>, std::error_code> monadic( 
    const ExpectedContainerOperation<T>& expected); 

template
boost::expected<ContainerOperation, std::error_code> monadic(
ExpectedContainerOperation&& expected);

Storage

maidsafe/nfs/storage.h

  • [x] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

This object has a single shared_ptr, and is shallow-copied. This makes it extremely quick to copy.

Represents the Storage abstraction listed above. Constructing a Storage object requires a StorageID object.

class Storage { 
  explicit Storage(StorageID); 

Future<ExpectedContainerOperation<std::vectorstd::string>> GetContainers();

Future<ExpectedContainerOperation> OpenContainer(std::string);
Future<ExpectedContainerOperation<>> DeleteContainer(std::string);
};

- **Storage(const StorageID)** 
  - Creates a Storage object. The `StorageID` provides the keys necessary for decrypting the data. 
- **GetContainers()** 
  - Retrieves the names of containers currently in Storage. 
- **OpenContainer(std::string)** 
  - Retrieves a `Container` with the specified name. The Container is created as necessary. 
- **DeleteContainer(std::string)** 
  - Deletes a container with the specified name. 

Container

maidsafe/nfs/container.h

  • [x] Thread-safe Public Functions
  • [x] Copyable
  • [x] Movable

This object has a single shared_ptr, and is shallow-copied. This makes it extremely quick to copy.

Represents the Container abstraction listed above. Constructing a Container object cannot be done directly; Container objects can only be retrieved from Storage::OpenContainer.

class Container { 
  Future<ExpectedContainerOperation<std::vector<std::pair<std::string, BlobVersion>>>> GetBlobs(); 

Future<ExpectedContainerOperation<std::vector>> GetBlobVersions(std::string key);

Future<ExpectedBlobOperation<>> PutMetadata(
std::string key, std::string, ModifyBlobVersion);
Future<ExpectedBlobOperationstd::string> GetMetadata(std::string key, RetrieveBlobVersion);

Future<ExpectedBlobOperation<>> Put(std::string key, std::string, ModifyBlobVersion);
Future<ExpectedBlobOperationstd::string> Get(std::string key, RetrieveBlobVersion);
Future<ExpectedBlobOperation<>> Delete(std::string key, RetrieveBlobVersion);

Future<ExpectedBlobOperation<>> ModifyRange(
std::string key, std::string, std::uint64_t offset, ModifyBlobVersion);
Future<ExpectedBlobOperationstd::string> GetRange(
std::string key, std::size_t length, std::uint64_t offset, RetrieveBlobVersion);

Future<ExpectedBlobOperation<>> Copy(
std::string from, RetrieveBlobVersion, std::string to, ModifyBlobVersion);
};

- **GetBlobs()** 
  - Retrieves the names of Blobs in the Container. The `BlobVersion` is provided for each Blob. 
- **GetBlobVersions()** 
  - Retrieves the history of `BlobVersion`s at the key. Oldest BlobVersion is always `BlobVersion::Defunct()`, and is used subsequently when the key had no associated Blob for some period of time. `std::vector::begin()` will be the newest `BlobVersion`, and `std::vector::end() - 1` will have the oldest `BlobVersion` (which is always `BlobVersion::Defunct()`). 
- **PutMetadata(std::string key, std::string, ModifyBlobVersion)** 
  - Stores the contents at the key as user metadata. Same effect as storing a blob (new BlobVersion). Maximum size is 64KB. 
- **GetMetadata(std::string key, RetrieveBlobVersion)** 
  - Retrieves the user controlled metadata for a Blob. 
- **Put(std::string key, std::string, ModifyBlobVersion)** 
  - Stores the Blob contents at the key. 
- **Get(std::string key, RetrieveBlobVersion)** 
  - Retrieves the Blob contents at the key. 
- **Delete(std::string key, RetrieveBlobVersion)** 
  - Removes the key from the Container. 
- **ModifyRange(std::string key, std::string, std::uint64_t offset, ModifyBlobVersion)** 
  - Stores a new Blob by re-using a portion of an existing Blob. The size of the Blob is automatically extended if offset exceeds the size of the original Blob. 
- **GetRange(std::string key, std::size_t length, std::uint64_t offset, RetrieveBlobVersion)** 
  - Retrieves a portion of the Blob contents at the key. Returned std::string is automatically truncated if length + offset exceeds Blob size. 
- **Copy(std::string from, RetrieveBlobVersion, std::string to, ModifyBlobVersion)** 
  - Copies user metadata + data from a Blob at one key to a Blob at another key.

I question the value of posting large extracteds of out of date documentation, rather than links to the latest information you can find (or asking where it is).

3 Likes

Is there updates on these topics? Where can they be found?

For this at the moment best to check the dns nfs and client crates (or github repo’s) there is documentation produced as code is changed to maintain the current API documentation.

1 Like

Is something like this helpful?
http://billpatrianakos.me/blog/2013/10/22/authorize-users-based-on-roles-and-permissions-without-a-gem/