Fri 23 May 2014 3:52AM

Basic idea to move users between pods on D*

R Ruxton Public Seen by 183

Move's are permanent.

See: https://gist.github.com/Ruxton/7d762520dd74fab87f07

Moving human@diaspora.pod.a to human@diaspora.pod.b

  1. User logs into human@diaspora.pod.a
  2. Click settings
  3. Scroll down
  4. Click "Move to another pod"
  5. User is presented with a download of their private key (and maybe serialized profile)

  6. User visits to diaspora.pod.b

  7. User clicks "Moving from another pod"

  8. User submits a form with their old d* ID, a new username for this pod, a password and the download from pod.a

  9. pod.b attempts validate the key is correct for the old d* person (fetches the person if they're not on pod)

  10. pod.b creates a user for the person with the supplied details

  11. pod.b sends a SignedUserMove to pod.a

  12. pod.a receives the SignedUserMove and federates it to all the users contacts

  13. pod.a updates the person with pod.b D* ID & deletes the user

  14. User should now be able to login to pod.b and pods will receive an update

* GUID of the person will remain the same
* there's an issue of what happens when a pod doesn't receive the federated SignedUserMove
* Admins shouldn't be allowed to move, they need to dethrone first


Jonne Haß Fri 23 May 2014 6:05AM

I'd actually like to see an export of all data we wish to retain during the move, the reason is that it would make the move feature a backup feature with no basically additional effort, allowing a move even if the original pod went down. With that idea a few more semantics can change, see my old writeup: https://github.com/MrZYX/old_diaspora_wiki/blob/master/Seed-migration-proposal.md


Ruxton Fri 23 May 2014 7:16AM

Totally agree, I'm only focusing on the move right now, I wan't to shut down one of my pods as cleanly as possible and move the 1 user on it to my newer pod.

I'm trying to keep it simple first, move the user, let them rebuild their stuff. If someone feels like focusing on export/import (regardless of pod) that would take care of both too.

I've looked a the existing export, no ones maintaining it. People are adding relations to the user/person models and none of them are getting put in the export.

To me the export is a solved problem, it's just a dump and reload of data, someone just needs to take ownership of it. People have manually moved before, I (yes i now its not for everyone) can sit there and poke my data back into the DB, not a big deal. The big issue was how to move the actual user/person relationship from one to another and notify everyone it happened, I believe what I proposed solves this.


Jonne Haß Fri 23 May 2014 7:46AM

Sure and I wouldn't mind a first version not migrating the data, but I think we should decide whether we want to support migration when the original pod is down and you just have such a download laying around. We can totally strip the data stuff from my old proposal and have that, if we don't want to support that use case yours is fine as it is.


Jonne Haß Fri 23 May 2014 7:49AM

Another issue both our proposals don't accommodate:

It's quite frequent that a pod receives a comment or a post (through a reshare) from a third pod and fetches the authors profile there. The third pod doesn't keep track of that and there's no contact from or to a user on that third pod. So the home pod of the fetched user doesn't know that he should send a move notice to the third pod. If then after a move someone from that pod want to contact that user, that'll fail.


Ruxton Fri 23 May 2014 8:11AM

Perfect, that last thing is the kind of issue I was looking to locate. I knew it could happen, but I couldn't work out exactly how. Will work on something to try solve that.


Jason Robinson Thu 29 May 2014 12:17PM

Totally love this idea - I think having this solved would bring a lot of confidence in to the project, since many people seem to feel it was one of the original promises in the kickstarter (umm it was I guess :)).

+1 for implementing this if you are up for working on it @ruxton :) No solution needs to be perfect at the beginning - that is impossible, the best way is to make something and iterate over it.

As for the export, it would be nice that these two would complement each other. After all, both ideas can use the same contents for the profile ("(and maybe serialized profile)" -part). Please make it JSON, not XML? :)

One idea I was going to write about was some kind of automatic backup of user profile/contacts/posts (without picture uploads) to*another pod*. This pod could be chosen randomly from participating pods (like "allow_backups" setting) and each user would automatically have their data backed up on a random pod. The data would be encrypted so that it could only be opened with their private key, which would given to download for the user on registration (with instructions to store it safely - loss of key=loss of backups).

In the event of catastrophe at the users pod, the user has the option of either downloading data or opening an account using the backup profile, on the other pod - which is when they would need to private key.

In this way user data would be backed up somewhere off-site, making it less likely for user to lose everything if the pod goes down. Also the pod which has the backups would not be able to read the backed up data, until user opens it with their key. The only issue I would see is that the backups would have to of course be incremental - which could be an issue as the data would need to be sent over encrypted.. Also the whole thing shouldn't cause too much strain on the pods of course.. DB space is hardly an issue for just post/profile data with no relations - and I'm 100% sure lots of pods would gladly participate.

But really, the only way imho to solve the problem of dying pods is to make it somehow not require user action. Requiring one to manually download data "as a precaution" is not really a working model. Maybe when we have the API there could be clients to automate it of course, but then that would be a big strain on servers ;)

But sorry for rambling in (mostly) unrelated things - would totally love to have the possibility for users to migrate pods!


goob Thu 29 May 2014 6:38PM

Whoever solved this would be my absolute hero for ever!

Jason's right: we agreed to use JSON for the export: proposal here.


Florian Staudacher Sat 31 May 2014 9:45AM

@jasonrobinson In principle, your idea is quite nice, even with all the technical details to be figured out. The only real conceptual problem I see is that the "automatic backup to a random pod" defeats the whole "choose the pod you trust" idea. ;)


Jason Robinson Sat 31 May 2014 9:56AM

The only real conceptual problem I see is that the “automatic backup to a random pod” defeats the whole “choose the pod you trust” idea. ;)

Well of course there could be a "let me choose" instead of being random :)


Jason Robinson Sat 31 May 2014 9:57AM

My point is - default experience IMHO should be as little need to decide as possible. Because most users don't like clicking through a gazillion things to just get to the application.


Maciek Łoziński Fri 13 Jun 2014 7:36PM

Maybe uploading to FTP/OwnCloud/DrobBox/something would be an option for automatic backup? Or isn't it better to put effort into implementing an API instead of automatic backup? Server strain could be reduced by allowing a backup of selected timespan through API.


Brad Koehn Tue 15 Jul 2014 5:11PM

Stolen from a post I made online:

From what I can see, account migration would have to have a few objectives:

Let's assume we want to migrate a user's data from account x on pod A to account y on pod B

  • The user informs pod A that he's migrating to y@B. The user initiates the migration from his account y@B.
  • Pod B requests Pod A send copies of the x@A data to y@B. This includes posts, images, comments, messages, etc. and account connections (e.g., start sharing with the same accounts on y@B as x@A was). Pod A would reject any migration other than to y@B, and uses https certificate validation to ensure it is sending data to the correct pod. Pod B could send a nonce/token to Pod A in its initial message, which pod A sends back with the x@A data for additional security.
  • Pod A could send out a message to other pods indicating the y@B is the new home for x@A. Pods hosting accounts sharing with x@A could verify the move with pod A and automatically start sharing corresponding accounts with y@B while unsharing x@A. Another alternative would be to send a message to any user sharing with x@A about the move.
  • After the move is complete, manage interactions with x@A. This could be as simple as a disallowing comments on existing posts/messages with an appropriate message, or as robust as redirecting posts/image requests/etc. to the new pod via 302. The user x@A should not be allowed to create new posts/messages/comments anymore. In fact, deleting the pod A account should be the only modification x@A is allowed to do, or perhaps initiating another migration.

I've outlined a fairly complex state machine and will probably require more transitions than what I've described (e.g., aborting the migration). And probably a fair amount of code.


Sean Tilley Tue 15 Jul 2014 8:20PM

I like what everyone has put here so far! My thoughts on how the process could work:

  1. User logs into their home pod
  2. In user settings, user can authenticate to an account on a remote pod using OpenID Connect and a Diaspora Migration API. You could almost think of it as the remote Diaspora pod being treated as an authenticated app.
  3. Posts, images, and contacts get sent securely to remote pod using API.
  4. Existing posts on original pod somehow are turned into references/redirects of some sort that point to the duplicates on the new pod. We may need to implement a proper way to revise posts via the API in order to successfully do this.
  5. After a determined amount of time, original posts are deleted once the new references get federated out to everybody.

Elm Sun 3 Aug 2014 7:21AM

I recall it was ONE of THE FEW BASIC features promoted by Diaspora !


goob Sun 3 Aug 2014 11:28AM

It may be basic, meaning fundamental to Diaspora's concept, but it is in no way basic, meaning simple to create or implement. On the contrary, it's extremely complex to do properly. That's why it's taken so long!


Jason Robinson Thu 13 Nov 2014 6:45AM

@ruxton confirmed he will not be working on this (on Twitter) - so anyone who wants to, feel free to participate in tuning the idea and/or creating an actual spec for the migration.

@jonnehass also has an old draft - https://github.com/jhass/old_diaspora_wiki/blob/master/Seed-migration-proposal.md

We really need to lock down a spec before any work can be done..


Jason Robinson Thu 13 Nov 2014 6:53AM

One (IMHO) good place to write up a spec is here: https://wiki.diasporafoundation.org/Category:Proposals


Robin Stent - Outreach Sun 16 Nov 2014 7:23PM

It would be great to get this implemented as I think it would reinforce the idea that D is decentralised in potential users minds, and give them more confidence in it. Also it would be a really cool feature.

I'm not sure that allowing the user to back up to another pod is a good idea, mainly from a perception point of view. I think any backups should be dealt with from the podmin end, however I still think allowing a user to download their data is a good thing.


goob Mon 17 Nov 2014 6:11PM

I’m not sure that allowing the user to back up to another pod is a good idea

The proposal is about migrating data from one pod to another, not backing up data from pod 1 to pod 2 while remaining active on pod 1.


Robin Stent - Outreach Mon 17 Nov 2014 8:57PM

@goob I was referring to an idea Jason mentioned in his post.


Jason Robinson Tue 18 Nov 2014 8:35AM

I think any backups should be dealt with from the podmin end, however I still think allowing a user to download their data is a good thing.

What does the user do if the podmin just stops running the pod? ;)

IMHO, there should be a way to optionally ensure podmin cannot keep the data. But I guess that could be a third-party client to automatically download data every day, which someone would surely make. That would just cause a huge load on the servers ;) But a planned feature to share encrypted data would not.


y.semin Wed 19 Nov 2014 4:52PM

What does the user do if the podmin just stops running the pod? ;)

The problem does already exist with mail providers. When a provider stops its service, the mails are often lost (IMAP is wonderful, but most of people I know do prefer get their mails on the webmail, not with a MUA, so they just loose everything...)


Jason Robinson Wed 19 Nov 2014 8:24PM

The problem does already exist with mail providers.

I'm not sure it's good practise to copy failed examples... ;)


Sean Tilley Wed 19 Nov 2014 11:28PM

One concept that might be worth investing in would be Channel cloning, which RedMatrix handles pretty well. From the documentation:

Account Cloning

Accounts in the Red Matrix are called to as nomadic identities. Nomadic, because a user's identity (see What is Zot? for the full explanation) is stuck to the hub where the identity was originally created. For example, when you created your Facebook, or Gmail account, it is tied to those services. They cannot function without Facebook.com or Gmail.com.

By contrast, say you've created a Red identity called tina@redhub.com. You can clone it to another Red hub by choosing the same, or a different name: liveForever@SomeRedMatrixHub.info

Both channels are now synchronized, which means all your contacts and preferences will be duplicated on your clone. It doesn't matter whether you send a post from your original hub, or the new hub. Posts will be mirrored on both accounts.

This is a rather revolutionary feature, if we consider some scenarios:

  • What happens if the hub where an identity is based, suddenly goes offline? Without cloning, a user will not be able to communicate until that hub comes back online. With cloning, you just log into your cloned account, and life goes on happily ever after.

  • The administrator of your hub can no longer afford to pay for his free and public Red Matrix hub. He announces that the hub will be shutting down in two weeks. This gives you ample time to clone your identity(ies) and preserve your Red relationships, friends and content.

  • What if your identity is subject to government censorship? Your hub provider is compelled to delete your account, along with any identities and associated data. With cloning, the Red Matrix offers censorship resistance. You can have hundreds of clones, if you wanted to, all named different, and existing on many different hubs, strewn around the internet.


Jason Robinson Fri 21 Nov 2014 7:21AM

Really neat feature indeed. I'm not sure about the "can use both id's" thingy - a bit confusing no? But otherwise it's something that maybe in a more limited manner would be perfect. Actually the backup thingy I suggested has the same principle except the profile would be encrypted and only activated if required.

Definitely concepts worth looking into. We shouldn't limit the security of users (concerning pod sudden deaths that happen all the time) by not planning any feature that allows users to somehow be backed up via an automated process.

And after the pod migration has been implemented, I could see the possibility to trigger a migration from the already copied profile - in the case of the original pod going down for example. But I guess in this case the ownership claim to data process would have to be somewhat different.