Loomio

Personal Data Import/Export

TS Tom Scott Public Seen by 193

[Edited by Fla]
This discussion is about account migration between pods.
Current github issue

Summary: We all agree this is an important feature, but it's hard to deal with considering privacy and security issues.

TS

Poll Created Thu 10 Jan 2013 9:22PM

Abandon the feature? Closed Fri 11 Jan 2013 4:01AM

Outcome
by Tom Scott Tue 25 Apr 2017 5:58AM

We're not abandoning this feature. :)

We could always abandon this feature. If we do, it means clearing that advert text on diasporaproject, letting people know in the GH Issue that it's not gonna happen in the near future, and closing it out.

I don't necessarily want to do this, but I figured this is the best time to bring up abandoning an issue, as no work has been done yet.

Results

Results Option % of points Voters
Agree 0.0% 0  
Abstain 14.3% 1 M
Disagree 57.1% 4 ST TS JH DM
Block 28.6% 2 JR G
Undecided 0% 134 JL BK FS MS AA S CB HF BO DM GC JH F M EG G AX PC PP BB

7 of 141 people have voted (4%)

TS

Tom Scott
Disagree
Thu 10 Jan 2013 9:23PM

Our users have the rights to the content they post on DIASPORA. Therefore, I think we should press on with this feature as long as we can get it done quickly and without any security issues. Those are my main two concerns.

ST

Sean Tilley
Disagree
Thu 10 Jan 2013 9:34PM

We shouldn't abandon it. It's a great concept for decentralized social, and being able to easily move off of these larger pods for personal pods is important.

JR

Jason Robinson
Block
Thu 10 Jan 2013 9:39PM

This is one of the most important features missing. Without the ability to migrate to another pod (be it a full wizard or manual export/import with limited dataset) - Diaspora* is just a collection of separate pods, not a proper network.

G

goob
Block
Thu 10 Jan 2013 10:23PM

This feature is absolutely one of THE key elements of the Diaspora concept of privacy and user ownership of their own data. Without it, Diaspora will not be the network it was set up to be.

However long it takes, it must be implemented eventually.

ST

Sean Tilley Thu 10 Jan 2013 9:33PM

I don't feel that the concept should be abandoned. Rather; we need to figure out how to make it work securely based on what's out there now.

Interestingly enough, both Friendica Red's Zot2 implementation and Tent are focusing on this idea of a user being able to literally "pack up and move" from one node on a network to another. Maybe they have some ideas on how it could be done; better yet, maybe we could talk to these different projects about possibly allowing users to move not just between Diaspora hosts, but onto Friendica and Tent hosts as well.

We need an existing implementation of this concept to look to for inspiration on how best to do this. I don't think just a basic "upload your data that you downloaded from the other pod" is exactly going to cut it in the long run.

JH

Jonne Haß Thu 10 Jan 2013 9:42PM

Just digging out what I thought about it like a year ago https://github.com/diaspora/diaspora/wiki/Seed-migration-proposal

TS

Tom Scott Thu 10 Jan 2013 9:44PM

@seantilleycommunitymanager why wouldn't a quick-n-dirty solution like that work? i thought we could just export some sort of big XML feed with all the text of every post along with a folder of images that were posted by the user, collect it in a .ZIP file and be ready to go. would that not be a good solution?

TS

Tom Scott Thu 10 Jan 2013 9:49PM

@jonneha i know it would blow the package up a bit but i think photos are crucial to the migration. it would just be a pain in the ass for users to have to separately download each image.

the way Facebook does it, you can actually pick whether you want photos in the archive. i think we should echo this feature. they also queue the job and email you when its done, since the archive files can get pretty large. I'd be all about doing it this way.

Can this kind of shit even be done on Heroku? We can't alter the filesystem, so where will we put the .ZIP archive?

JH

Jonne Haß Thu 10 Jan 2013 9:58PM

Hm maybe we can get access to a tmpfs on Heroku but in general it's readonly.

ST

Sean Tilley Thu 10 Jan 2013 10:33PM

@tomscott The issue I have with the quick 'n dirty approach is that it could open up a potential security problem. It's not necessarily a technical problem, so much as that it's a problem of social engineering.

If all the process requires is a download/upload of data, what's stopping someone from finding a way to download someone else's data, upload it to a new pod, and pretend to be that person?

TS

Tom Scott Fri 11 Jan 2013 4:04AM

@seantilleycommunitymanager in terms of pod-to-pod migrating, could we have the pods actually do the transfer and migration? the user would never actually touch the .ZIP archive (though they could optionally download it), rather the pods would facilitate all of the archive creation, unpacking and loading of data.

ST

Sean Tilley Fri 11 Jan 2013 5:50AM

@tomscott That's exactly what I was thinking of. There could be an option in Settings to move to a different pod, where you could type in the URL of your Diaspora pod and authenticate with it like it's an app.

This would push your posts, contacts, Aspects, and bio to the new pod, and your old account would simply become a redirect to your current one.

Of course, that's all easier said than done, but from an end user's standpoint, it'd be very easy to migrate by just typing in a URL and clicking a button.

LV

Louigi Verona Fri 11 Jan 2013 6:20AM

I think this is a key feature for Diaspora, as a decentralized network.
Speaking about a quick approach, the possibility of abuse that Sean Tilley voiced - how possible is it?

RS-

Robin Stent - Outreach Fri 11 Jan 2013 8:26AM

really glad to hear this is not being abandoned, I think its a really important feature. As far as downloading a zip of your data goes, I think that's fine for people to have a copy of their stuff, but not as a way of transferring data between pods.

TS

Tom Scott Fri 11 Jan 2013 4:42PM

@seantilleycommunitymanager it may be difficult, but i believe this is the first step to providing 3rd-party applications access to DIASPORA as well as providing us with a much-requested feature. we can use the pod-to-pod authentication and transmission knowledge we gain here to build future things that deal with 3rd-party apps.

but that's all out of the scope of this proposal. i'll write up a decision and let's get to work.

TS

Poll Created Fri 11 Jan 2013 4:46PM

Export pod data to an archive Closed Sun 13 Jan 2013 6:53PM

The first step to this whole project is exporting a person's entire persisted history to an archive. Without this key ingredient, there's no point in coding pod-to-pod communications. In order to maintain the philosophy that you should not only be in control of, but own the data you post to DIASPORA, it will be an option to download the archive from your pod. Pods will only retain archives for 4 hours, after which they will be purged.

Results

Results Option % of points Voters
Agree 40.0% 2 TS M
Abstain 60.0% 3 JH G M
Disagree 0.0% 0  
Block 0.0% 0  
Undecided 0% 137 JL BK ST FS MS AA S CB HF BO DM GC JH JR F M EG G AX PC

5 of 142 people have voted (3%)

TS

Tom Scott
Agree
Fri 11 Jan 2013 4:47PM

One more thing...is it even possible to write archive files on Heroku? If not, this may need to be an optional feature..

JH

Jonne Haß
Abstain
Fri 11 Jan 2013 6:20PM

I don't quite see the benefits without upload, with upload this would be too insecure as it's written in this proposal. If we really do this I want a wiki page explaining why there's no upload, that I can direct the shitstorm to.

M

matl
Abstain
Sat 12 Jan 2013 3:10PM

Data transfer to another pod is more essential than data export to an archive

G

goob
Abstain
Sat 12 Jan 2013 3:30PM

I'm abstaining as I agree there should be the possibility to export an entire account, but not without a corresponding ability to import that entire account into a new pod. As Jonne says, without upload/import, there's no benefit to download/export.

TS

Tom Scott Fri 11 Jan 2013 4:50PM

@louigiverona depends on how we make it. we can't just let someone POST to /users and have us set up a secure keypair and password for them. Since we're not running a centralized network, it would be possible for me to take Sean's data and upload it to a different pod, and therefore pretend to be him on DIASPORA. Although sean@mypod.com and sean@joindiaspora.com would be slightly different, it would be difficult for the common person to distinguish and may lead to some clever social engineering/phishing scams. We don't want to facilitate that in any way, so we need to make sure this whole thing is done securely.

G

goob Sat 12 Jan 2013 3:31PM

Export of data is only of use if you can import it to your new pod. Otherwise you may as well just close one account and open a new one.

TS

Tom Scott Sun 13 Jan 2013 6:53PM

@jonneha, @goob and @mataloutreach -- lol i think i got the idea of "decisions" here incorrect. i should have put the whole proposal up there.

But before I do, does anyone know if we can write files to the HD on Heroku? If not, we need to make this an optional feature so Heroku pods don't crash when this happens.

JR

Jason Robinson Sun 13 Jan 2013 7:39PM

We already can export some data? Why would that not be needed - even Facebook allows you to do that :)

All we need is an import. I don't see the security problem since you're not supposed to give your data archive to anyone else. Own fault if you upload it to the internet and someone uploads your data on another pod.

JH

Jonne Haß Sun 13 Jan 2013 8:33PM

The security concern is that we need a way for another pod to authenticate as the existing user, otherwise one could simply pretend to be someone else to the network. We just need the private key of the user on the new pod and generate and establish a new keypair on/from the new pod, so that the old pod can make no further posts in the name of the user.

JR

Jason Robinson Sun 13 Jan 2013 8:45PM

I say let's just do a simple upload for now and reuse the actual data processing logic there when the automatic wizard thingy is done. This way there is no risk - any more than there is now. I could create jonne@iliketoast.net and pretend to be him at any time :)

For data processing for example contacts in aspects would be nice to be auto-created - imho this could be first step.

ST

Sean Tilley Sun 13 Jan 2013 9:16PM

The other small concern of mine is facilitating full data export: that is to say, if a user is on a big pod like JoinDiaspora.com, and has two years worth of posts, photos, and a couple hundred contacts, exporting all of that could be considered database-intensive, especially if lots of people try to do a full export on the same day.

Would setting a variable that defines how much time a user must wait to perform the download after requesting all of their data be a good idea, so that the database doesn't get too strained if everyone tried to download all of their data all at the same time?

JR

Jason Robinson Mon 14 Jan 2013 10:11AM

Facebook does it nicely by putting it in a queue and then emailing the user later when the archive is ready.

F

Flaburgan Mon 14 Jan 2013 10:36AM

@jonneha about authentication on the new pod, one more time, Persona can be the solution there, because the user will be able to log in on every website, so we can simply ask the user to log in. We really need to use Persona for authentication. Will take a look at that in February.

JH

Jonne Haß Mon 14 Jan 2013 12:28PM

@flaburgan I was talking about server to server authentication, not client to server.

F

Flaburgan Mon 14 Jan 2013 1:13PM

Hm, if the new pod is able to know that the user who wants to import data is the one who is connect on the old pod, the problem is solved, isn't it ?

JH

Jonne Haß Mon 14 Jan 2013 1:26PM

How do you prove to the other pods?

G

goob Mon 14 Jan 2013 3:43PM

Jonne, I'm not sure we need such rigorous authentication if we're talking about a two-step process, as follows:

  1. User exports their entire account data to a local file on their machine.
  2. User creates account on new pod and imports account data file to new account.

Only the authenticated user of the original account will be able to export the data from that account, and if they're downloading it to their local machine before uploading it to the new pod, unless they do something stupid, the file won't get into third-party hands.

If we're talking about direct transfer of account data from one pod to another pod, then yes, secure keys will be needed.

TS

Tom Scott Mon 14 Jan 2013 4:44PM

@flaburgan i looked into it, and while Persona would be our choice for SSO i don't think it's what we need here. i'm not going to rip out our entire authentication system so we can add one little feature.

@jonneha an alternative idea to this is tokens. say a user downloads their data from podA and wants to migrate to podB. we could embed a token inside the archive in a JSON somewhere, then before an upload happens on podB, the server checks the token (like a checksum) against podA's /user/:id/token endpoint and makes sure they match. if they don't, that archive didn't come from podA and it's an impostor. this allows podB to "make sure" the user in question is actually coming from podA. it proves identity, and IMHO building a system where identity doesn't matter is just opening the floodgates for spam and other social engineering.

i don't think most people are observant enough to notice jonne@givemeyourcreditcardmotherfucker.com is not the same as jonne@social.mrxyx.com. Especially with the exact same profile, exact same pic, exact same status updates. It would be less obvious if it was something like jonne@social.mrzyx.co vs. jonne@social.mrzyx.com...

TS

Tom Scott Mon 14 Jan 2013 4:46PM

@goob again, there's no way to prove that said User is in fact the User from their originating pod. I personally think identity matters, and identity isn't just your login@pod_host.com...

JH

Jonne Haß Mon 14 Jan 2013 4:51PM

My aim would be a seamless migration, lets say we have bob@podA who friendes alice@podB and wants to move to bob@podC. I'd like to have the contact for bob and alice on podB change without any action from alice. This could be done by sending a "I'm here now!"-kind of message from podC to podB. The only way for podC to prove that he now owns bob@podA is signing that message with bobs existing private key, podB already has bob@podA's public key and can thus verify it and update all records.

G

goob Mon 14 Jan 2013 5:19PM

@tomscott I think you're missing the point of what I said.

There is currently nothing to stop me from opening an account on a pod under the name Tom Scott, and pretending to be you. However, what I won't be able to do under the scheme I outlined is to import your account data from your existing account to the new fake account I have set up, because I won't have access to the account data. Only you (and, perhaps, your podmin) have access to those data, so only you will be able to create a new account and import the data from your existing account. I suppose a simple key could be added to the data export, which the new pod could then check with the old pod to confirm that this is a genuine export from the old pod.

@jonneha if you think a seamless transfer is feasible, that would be far preferable. It may prove to be the easier way of transferring all the connections with other accounts. The only thing I'd add to what you said is that from a privacy point of view it might be wise to notify Alice (and others) that

'Your contact Bob has moved his account to podC. Click here to confirm that you are happy to continue sharing with him at this new pod, or click here if you wish to stop sharing with Bob'.

Reason being that Alice might feel she can't trust the podmin of podC and doesn't want anything to do with it, nor for any of her postings to be potentially available to that podmin through her sharing with Bob. There may be no logical reason for her to feel this way, but Diaspora seems to be all about being up-front with exactly what's happening with your data, so it would, I think, be good to notify a person's contacts when that person moves pods, just to make people aware of what is happening and allow them to make the choice of whether to continue sharing.

I suppose, when someone starts the process of migrating to another pod, a pop-up could be shown them to notify all their contacts 'Hi, just to let you know I'm moving to podC. This won't affect our contact and you don't need to do anything' so that Bob's contacts know in advance.

TS

Tom Scott Mon 14 Jan 2013 5:35PM

@goob i see what you're saying. i think we're on the same page. :)

what do you guys think is easier though, a seamless transfer from pod->pod where we have to test and code every step of the process, or allowing humans to do the downloading and uploading of data, as long as the server(s) maintain the authorization process. i'm thinking the latter would be much easier to implement and just as secure as if we had pods take care of the "whole shebang"

G

goob Mon 14 Jan 2013 5:45PM

I think the seamless transfer Jonne is talking about is best as an ultimate aim, but suspect there is a lot of coding and checking to do on the way to that. I think a two-step manual export/import could work well as a stop-gap measure, allowing those who really want to do so to migrate pods sooner than later. (I'd like to, as I want to set up my own pod, and this is one of the things which is stopping me.)

If the manual two-step process, perhaps with an authentification key as part of the export package, is easy to implement and can be done soon, I'd say let's implement that as an interim measure, and work on the seamless migration in the longer term.

JH

Jonne Haß Mon 14 Jan 2013 6:15PM

My use of the word seamless was entirely scoped to setting up/maintaining existing relationships. IMO a migration is a transfer, a move, not a duplication of a account and leaving the old one abandoned. As I basically outlined in my over a year old but still standing proposal. The way the archive is transferred is more of a implementation detail, it could be download/upload, POST to special route of old pod, generating a URL and leaving it on the old pod, entering the URL on the new pod, all of the above, whatever else. One benefit of my approach (given the archive is downloadable) is that it could serve as a backup, since the deletion of the old account is tried but not required for the process to complete and thus there's no point where the old pod needs to be still reachable.

JR

Jason Robinson Mon 14 Jan 2013 7:16PM

The data exporting is needed regardless of whether we allow it to be uploaded :) But of course we already have that. Just saying.

G

goob Mon 14 Jan 2013 8:04PM

Jonne, with the two-step download/upload method, there could be a step built in to the activation process of the new account at which, having had chance to check that everything has uploaded properly and is working as it should, the user is prompted to click a button to fully activate the new account (ie it was in a testing mode before that). At this point, the key is sent back to the old pod to instigate closing the old account, if the user hasn't already done that. This should solve the problem of duplicate account lingering all over the place. I hope.

G

goob Mon 14 Jan 2013 8:04PM

Jason, at the moment you can only export a very limited range of your account data. At least, that was still true the last time I tried it.

TS

Tom Scott Mon 14 Jan 2013 9:42PM

@jasonrobinson is right about the account export options. we got photos and the profile, not sure if we have posts yet but it doesn't matter.

i say since that's already shipped we work off that, and if we choose to we can expand the export option(s) in the future. but i think that's enough data down there to experiment with building a pod->pod migration option.

@jonneha one issue with your data export model is, wouldn't we need to eventually write the archive to the local disk, and then POST it to an endpoint somewhere else? If so, I have a number of questions about how we're gonna do it:

  1. A lot of web servers aren't configured to handle uploads of such a large size (given a reasonable amount of photos these archives could get rather large quickly). Would POSTing somewhere around 50mb of data to an endpoint on a pod require any sort of custom server configuration?
  2. How would Heroku users handle this upgrade? Heroku can't write files to the disk, so how would we export to Heroku pods?

IMO it seems that making users manually manage the upload/download process would be a lot easier and still retain the level of security we need. API calls that check a token for validity would be all we need to make sure the "right" user is trying to establish the "right" account on a different pod.

FS

Florian Staudacher Wed 16 Jan 2013 5:25PM

I could imagine a solution using STUN to puch a connection through any possible firewall or NAT and the servers involved connecting directly and exchanging as much data as they want over random ports they negotiate over the normal HTTPS protocol.
(A little like FTP, with a control connection and a separate data link)

Of course ... that's a LOT of work

TS

Poll Created Tue 22 Jan 2013 1:10PM

Import pod data from the Export archive Closed Fri 25 Jan 2013 8:09AM

Only registered users can do this off of their Account page. At this time, we will require migrating users to register on the new pod, export their data off the old pod, and then import it onto the new one. The data import feature will be simple, and basically just read in the JSON and photos to the new user's account.

You literally just upload JSON (or however we're storing it) and photos, and we take care of the rest.

Results

Results Option % of points Voters
Agree 100.0% 6 ST TS JR M M DM
Abstain 0.0% 0  
Disagree 0.0% 0  
Block 0.0% 0  
Undecided 0% 136 JL BK FS MS AA S CB HF BO JH DM GC JH F M EG G AX PC PP

6 of 142 people have voted (4%)

TS

Tom Scott
Agree
Tue 22 Jan 2013 1:11PM

This is a simple and elegant solution to the data import problem. Pod-to-pod communication will take too long, and we can't really decide on a good way to do it, so I propose we get this code out there in the short term and stop worrying about it.

ST

Sean Tilley
Agree
Thu 24 Jan 2013 8:33PM

I'd like to see this done securely, with little chance of user identity theft, but I suppose we all need a stepping stone to get from Point A to Point B.

G

goob Tue 22 Jan 2013 1:26PM

Didn't we already agree to do this, Tom?

But, as discussed before, an important step in the import process should be, once the user is happy that the import has taken place correctly, the new pod communicating with the old pod and providing it with a key generated by the old pod during the export process, which then prompts the old pod to delete the old account. This will mean it is a proper migration (leaving the old pod and moving to the new pod) rather than simply opening a duplicate account.

The new pod will also need to communicate with all pods with which the account is connected to tell those pods where the account is now located, so that contacts will not lost touch with the account.

TS

Tom Scott Wed 23 Jan 2013 4:06PM

@goob it will be the user's responsibility for now to clean up their old account. we'll provide a message stating that but for now, it would just be much easier

and for the record, no, this decision was not made before. As you can see below, "Export pod data to an archive" was already decided upon, but I closed it early after Jay pointed me to the code that was already in master dealing with export archives.

JH

Jonne Haß Fri 25 Jan 2013 11:51AM

Uh, that was a rather short running time...

TS

Tom Scott Fri 25 Jan 2013 7:26PM

@jonneha all of the decisions have been 3 days i think. should i make them for longer in the future?

JH

Jonne Haß Fri 25 Jan 2013 9:31PM

Maybe in this thread, but I usually try to include 2 weekends.

SG

Sam Gleske Tue 5 Feb 2013 4:42AM

I think this topic is being over complicated. Realistically, due to the distributed nature of diaspora there could be duplicate contact public names on different pods. The issue should be how to safely transfer from one Pod to another without losing touch with friends on the current and other Pods.

I have drafted simple proposal for transfer authorization. Please tell me your thoughts.

https://docs.google.com/file/d/0B9CMUJ5UVuEkSlRUUU1mNjhGU1k/edit?usp=sharing

JH

Jonne Haß Tue 5 Feb 2013 11:46AM

Thanks for your work.

This is pretty much what I outlined, except for the additional email step. My proposal has the advantage that PodA could be (permanently) down when the transfer happens. And I'm missing a form of authentication against third parties in your proposal. How would you ensure that a rogue pod doesn't simply announce that a user now lives at him (given he somehow managed to know a few contacts of the user)?

SG

Sam Gleske Tue 5 Feb 2013 6:15PM

It is the job of PodA to announce to the contacts that the user has moved. PodB has no part of that process. The optional contact list I mentioned as part of the archive is just that, a list. It bears no ability to set up communication with the other pods unless manually executed. Whereas, the PodA contact update is automatic which friends' pods will autoupdate to the new pod without interaction.

TS

Tom Scott Sat 9 Feb 2013 7:03PM

@samgleske That's an awesome proposal, and I agree with the way it plans to carry out the ultimate goal of having fully-transferred, unobtrusive migrations from pod to pod in our own network.

JR

Jason Robinson Tue 19 Feb 2013 7:57AM

Awesome @samgleske - good step forward having something written down in a document.

FS

Florian Staudacher Tue 5 Mar 2013 1:36PM

That is definitely a good step in the right direction.
We just really need to focus on the authenticity of the data we let someone import. By which I mean I want to avoid people creating accounts and then spamming a server with fake imported user archives, thus opening a door for DoS. Also, I am not convinced our asynchronous protocol is set up to handle the handshake and communication necessary to establish what is outlined in step 3.

Also, I would at least inform the contacts of a user that has changed Pods about it, maybe even let them confirm it first. e.g. "Your contact 'name' has moved from 'podA' to 'podB'. Please confirm."
Another possibility would be to display a move in the stream as a "special status update" for all of the users contacts.

F

Flaburgan Fri 19 Apr 2013 8:32AM

Waiting for someone having the time to develop the whole solution, a temporary solution could be allowing to export and import a list of contacts. This would be only an handles list so no problem about identity or anything.

JR

Jason Robinson Fri 19 Apr 2013 9:07AM

I've made a half-ready script that uses cliaspora and the existing export functionality. Due to way things work unfortunately cliaspora is not able to trigger a search for users not known for a pod, so it only works for users known to the pod where moving. Need to clean up the code also and somehow make it work, either by patching cliaspora or triggering a search outside it.

G

goob Fri 19 Apr 2013 11:04AM

Fantastic, Jason. More power to your coding elbow.

G

goob Mon 20 May 2013 1:51PM

Should we create a working group to investigate and decide on the best approach to solving the account migration issue? It's such a key issue to Diaspora's success that I wouldn't want it to fall on to the back-burner.

F

Flaburgan Thu 23 May 2013 8:00AM

I think we have to put energy here, as joindiaspora / diasp.org become really too big. Allow users to move from a pod to another can solve a part of this problem.

ST

Sean Tilley Thu 23 May 2013 7:28PM

Yeah, this is a huge priority. Bigger pods have to deal with more workers, and therefore more potential hiccups. If we can facilitate a way to easily and securely migrate to another pod, we could potentially address some of these network issues with federation, while also encouraging smaller pods to pop up.

F

Flaburgan Fri 24 May 2013 11:59AM

As pointed there, I think we should at least provide a way to export and then import contacts (and with them, Aspects). I don't see any downside with a simple json containing that, so we really should do it soon.

G

goob Sat 25 May 2013 8:20PM

How labour-intensive would it be to code a simple manual migration process for now:

  1. User exports their entire account data to a local file on their machine.
  2. User creates account on new pod and imports account data file to new account.
  3. Contacts are then informed that this person has moved from pod X to pod Y, and are asked to authorise connection to new account on new pod.
  4. User is prompted to close their old account manually.

Just to get the possibility of moving one's account, even if it's not the tidiest method of doing it, as quickly as possible, while the exact method of creating a seamless transfer of the type Jonne proposes which is also secure is debated.

G

goob Sat 25 May 2013 8:21PM

If an interim method isn't going to be prohibitive in terms of coding effort, I could start a proposal on this, but I don't want to do so before hearing from devs on feasibility.

F

Flaburgan Mon 27 May 2013 9:29AM

@goob I was thinking about that but without a special information for the contacts. They would just receive the classic "bob started sharing with you" when contacts will be imported in the new account, I think this is enough.

G

Poll Created Sun 9 Jun 2013 1:14PM

Create manual export/import as short-term measure Closed Sun 30 Jun 2013 11:00PM

Outcome
by goob Tue 25 Apr 2017 5:58AM

There's agreement in principle for a short-term manual solution. Now to work out how this can be implemented, and if there's a way to do it which will require less effort than the full solution.

Just to get things moving, I propose that we implement a manual two-step export/import process while deciding on the best method for implementing automated pod migration in the future.

The user is responsible for the security of their data (and those of their contacts) during the export/import process. They download their account data in toto, including contact information, to a local source (hard drive etc), then separately upload this account information to a new pod.

The exact method of doing this can be agreed if the proposal is passed in principle.

This is just a measure to allow account migration in the short term.

Results

Results Option % of points Voters
Agree 88.9% 16 TS T JR F G PP BB E T M M S L SM R N
Abstain 11.1% 2 DS H
Disagree 0.0% 0  
Block 0.0% 0  
Undecided 0% 131 JL BK ST FS MS AA S CB HF BO JH DM GC JH M EG G AX PC LP

18 of 149 people have voted (12%)

JR

Jason Robinson
Agree
Sun 9 Jun 2013 5:46PM

There is an export feature - but it is broken. I doubt anyone will object if it is rewritten.

The upload should be for contacts (automated add-to-aspects, tags (to follow) and some stuff like profile details. I don't see how posts could b imported

G

goob
Agree
Mon 10 Jun 2013 3:57PM

Just as a quick fix so we can say 'At Diaspora you can move from one pod to another' as was promised from the outset.

Best to keep it simple at this stage just to get it done.

H

hewiak
Abstain
Mon 10 Jun 2013 4:44PM

chances are high that a temp fix will be an almost equal aggravation.

PP

Pirate Praveen
Agree
Thu 13 Jun 2013 7:56PM

at the least contacts, aspects and tags should be importable. If pods have keys, we can think of signing posts with pod key and verifying it before importing.

TS

Tom Scott
Agree
Sat 15 Jun 2013 5:55PM

Jason hit the nail on the head, this feature already exists but is somewhat broken. So I would be down for someone fixing it.

F

Flaburgan
Agree
Mon 17 Jun 2013 2:40PM

A simple json containing contacts, aspects and tag following looks great for me. Nothing personal for the moment.

S

Shmerl
Agree
Sun 23 Jun 2013 3:59PM

If done properly, this can give some workaround before automated moving between the pods would work.

L

L3MNcakes
Agree
Tue 25 Jun 2013 4:03PM

Would love to see this feature! :)

DS

Dennis Schubert
Abstain
Sat 29 Jun 2013 10:10PM

Quick and dirty solutions are not good. I'd prefer working on the federation and the API first...

G

goob Sun 9 Jun 2013 1:16PM

I thought a good idea to create an interim proposal in the hope of moving things forward.

Would it be worth doing a call-out from the Diaspora HQ account for people to form a working group to work on this issue? It is a key one and likely to be quite labour-intensive. If so, what should the notice say? I can post it once we've decided.

JR

Jason Robinson Sun 9 Jun 2013 5:47PM

Importing posts is a bit tricky as they would all have to be new instances of the post. Don't really see what the benefit of this would be since all interactions with the posts would be left behind.

FS

Florian Staudacher Wed 12 Jun 2013 11:02PM

I'm not convinced a temporary export/import will solve the "problem", but it would at least be a start.
We need a working export function anyway...

JR

Jason Robinson Thu 13 Jun 2013 7:30AM

Diaspora* claims that you own your data - but we don't have a working export like many other services, Facebook for example ;) I'd consider this a high prio.

G

goob Thu 13 Jun 2013 10:43AM

I don't think it's a long-term answer, @florianstaudacher , but it would provide at least a rough-and-ready means for people who are really keen to move pods to do so. And as it's been one of the cornerstones of the Diaspora philosophy and marketing from the beginning that you can move pods, I think it would be good to have at least some means of doing so as soon as possible.

G

goob Mon 17 Jun 2013 3:02PM

I'd suggest also including profile information as well as contacts/aspects/followed tags.

F

Flaburgan Wed 19 Jun 2013 1:14PM

@goob profile information could be personal data, I think we should not include that while we don't have a secure process.

G

goob Wed 19 Jun 2013 6:54PM

Fla, I don't see it that way: we're entrusting the user who is migrating their account with the security of that account data between export (which can only be done by the person with the account password, so is secure) and import to a new pod.

This includes contact information, which are other people's private data, so if we can entrust the migrating user with those data, we must be able to entrust them with their own profile information.

if they are silly and lose/give away the export before uploading it to a new pod, that's their own fault, isn't it?

JR

Jason Robinson Thu 20 Jun 2013 9:02AM

@flaburgan - why should a user not be able to import profile data from an archive? It's just text after all - they can set it by hand to whatever they want anyway.

The security issue comes from migrating posts which is the content that has interactions with other users and those interactions cannot be remapped easily. But profile data is just personal data that the user should be able to take with them and import.

JR

Jason Robinson Thu 20 Jun 2013 9:04AM

Also we really should allow users to export their own post (without interactions, like comments and likes by other people). Just for archiving, not uploading. We advertise that users own their own data. Even facebook and google services allow export of data :)

R

Ruxton Thu 27 Jun 2013 1:15AM

I'd like to point out how integral this feature is once again. Yesterday on a post of Sean's the discussion got crazy essentially over the fact it's way to hard to pick up and move to a pod. https://joindiaspora.com/posts/2773187 we definitely need to commit time to working on this.

G

goob Thu 27 Jun 2013 11:00AM

Do you think it would be good to advertise on D itself for developers to form a working group to help create this? I can post from the DHQ account if someone who has technical knowledge will help me draft a post.

I think for this and the other key but major things, it's a good idea to set up dedicated working groups who will work on the task. There's a proposal at the moment to do this for federation, for example.

G

goob Mon 1 Jul 2013 10:02AM

Dennis, I agree that 'quick and dirty' solutions aren't always ideal, however they can sometimes be better than no solution, which is what we have at the moment regarding account migration.

And this manual solution need not be 'quick and dirty' in coding terms, but might well be an easier thing to code than a fully automated process such as Jonne proposed earlier in the discussion, and therefore could be developed and implemented more quickly.

ST

Sean Tilley Mon 15 Jul 2013 9:17PM

Just to provide reference for anyone that missed it, information and code for the migration script @jasonrobinson has been working on can be learned about in this post: https://joindiaspora.com/posts/2814800

JR

Jason Robinson Tue 16 Jul 2013 6:31AM

Aiming to get a new version out soon that uses the latest diaspy and also does tags. But this is just a little hack helper thingy, we need proper export and after that a little-by-little import functionality (contacts at least first, like this script does them but internally).

ST

Sean Tilley Wed 14 Aug 2013 5:44PM

@jasonrobinson Hey, what's the progress on this?

JR

Jason Robinson Wed 14 Aug 2013 7:13PM

@seantilleycommunit just haven't had time :( Hopefully I'll get it done asap, so user doesn't need to know anything except run some commands in the terminal.

G

goob Fri 8 Nov 2013 11:50AM

@jasonrobinson has there been any progress on your script, and do you think it's something on the base of which a migration feature might be able to be worked into the Diaspora codebase in the future?

JR

Jason Robinson Fri 8 Nov 2013 2:55PM

@goob sorry haven't touched it since I've involved myself in way too many things and since I myself don't need a pod migration script any more, I've not needed to touch it yet. I plan to at least package it properly and then that will be it.

This script is just a hack and not usable for permanent code. The proper way should be to allow the user to 1) export contacts as CSV and 2) import them on the other pod - that would do exactly the same thing.

G

goob Fri 8 Nov 2013 4:20PM

No problem, and no need to apologise. I just came to this discussion to bump it, and saw the end of it, I thought I'd ask where you'd got to.

Anyone got any suggestions about how we can get this moving? What would people need in order to create such a facility? And how can we enthuse people to work on this? We have agreement that creating a manual export/import as an interim measure is desirable, but what steps do we need to take now to try to get things moving.

Wish I could help with code!

JR

Jason Robinson Fri 8 Nov 2013 6:10PM

A manual export is not an interim solution - we need that anyway, so would be a good place to start. I guess we need some blueprint on the data export format, contents, etc.

G

goob Sat 25 Jan 2014 5:57PM

@jasonrobinson how would you suggest we proceed to get this blueprint and so on?

JR

Jason Robinson Sat 25 Jan 2014 6:45PM

@goob someone needs to write something up and propose I guess :P On my todo but I don't think I'll be having time to tackle that any time soon :(

Maybe a good idea would be not to try and cover everything at once - instead just some basics like contacts which imho is the most important thing to start with.

G

goob Sun 26 Jan 2014 12:23PM

You suggest exporting in CSV. The old system was XML.

Does anyone else have suggestions for the best format to use for export? As suggested by Jason, we'll take a vote just on contacts export first, but it might be necessary to take into account other things that will be exported in the future (posts, photos, etc) in deciding which format will be best.

G

goob Sun 26 Jan 2014 12:27PM

In fact, the 'old' exporting in XML still works - I just tried it. I thought it had been broken at some point. Is there any reason we can't retain this export as it is, and look to build an import for at least some of the contents?

ST

Sean Tilley Sun 26 Jan 2014 10:14PM

What about exporting everything as JSON?

X

Xophael Mon 27 Jan 2014 11:37AM

I agree with @jasonrobinson; exporting and re-importing contacts is the only feature I need to migrate to another pod and definitely abandon joindiaspora. Jason, I tried your hack script but it fails at login on source pod, returns 406.
No preference on format since I don't develop on this project t--I'd just say, on the projects I work on, whenever I can I dump XML for other lightweight solutions - csv mostly.

JR

Jason Robinson Sat 1 Feb 2014 6:33PM

@goob I'm pretty sure I meant to write JSON :)

I'm not sure we really need to vote whether we want to export contacts? Surely we do not want to be worse than Facebook - which allows you to export just about all the content you post ;)

But of course if someone wants a vote we can vote. Still, someone needs to write some code for it, I think that is the real reason the export still sucks - not that we don't need or want it.

G

goob Sat 1 Feb 2014 7:02PM

I agree we don't need a vote on whether or not to create an export facility - it's essential, and in any case the vote has already been taken - I meant that we should vote on which format to use for export. My previous comment wasn't very clear about that.

Worth taking a vote on using JSON as the export format?

JR

Jason Robinson Sat 1 Feb 2014 7:10PM

@goob yeah sure, we can do that, then make a blueprint for the actual contents in the wiki

G

Poll Created Sat 1 Feb 2014 8:19PM

Use JSON as format for account export Closed Tue 11 Feb 2014 8:09PM

Outcome
by goob Tue 25 Apr 2017 5:58AM

We will use JSON as the format for exporting account data.

We need to have a means of exporting account data (contacts, posts, etc) for migration to be possible. It has been suggested that we use JSON as the export format. Do you agree?

Results

Results Option % of points Voters
Agree 66.7% 20 ST JH T JR DU BC SAM SM SVB S W N CM FTL KAK MM QD A DU DU
Abstain 30.0% 9 PH PG IGM JL N M A DB SP
Disagree 3.3% 1 R
Block 0.0% 0  
Undecided 0% 129 JL BK FS MS TS AA S CB HF BO DM GC JH F M EG G AX PC PP

30 of 159 people have voted (18%)

DU

[deactivated account]
Agree
Sat 1 Feb 2014 8:28PM

Having just written a JSON API for a .NET project, and not having used JSON before that point (been doing XML all my life!), I was amazed at how light weight JSON is. So yes, the JSON format definitely gets my vote.

JR

Jason Robinson
Agree
Sat 1 Feb 2014 8:46PM

XML should be banned and burned with fire

BC

Balasankar C
Agree
Sun 2 Feb 2014 12:05PM

JSON is an excellent choice over XML.

IGM

Ivan Gabriel Morén
Abstain
Mon 3 Feb 2014 6:13PM

My experiences of json are all positive.

M

Maciek Łoziński
Abstain
Mon 3 Feb 2014 8:29PM

I think a format is not so important, although I find JSON easier to use in many applications.

DU

[deactivated account]
Agree
Wed 5 Feb 2014 3:19PM

JSON is super flexible and the best choice

A

Airon90
Abstain
Wed 5 Feb 2014 5:38PM

Just improve the feature ;)

MM

Marek Marecki
Agree
Mon 10 Feb 2014 9:14PM

Definitely "yes". I think JSON is superior to XML in any way important to Diaspora.

MM

Marek Marecki
Agree
Mon 10 Feb 2014 9:15PM

Definitely "yes". I think JSON is superior to XML in every way important to Diaspora.

R

Ryuno-Ki
Disagree
Mon 10 Feb 2014 10:34PM

Well, JSON is about data, whereby XML can contain information about structure as well. (Same holds true for CSV).

At least for posts I'd demand a XML dump.

Downloading contacts/aspects via JSON/CSV is fine.

JR

Jason Robinson Sat 1 Feb 2014 8:50PM

@goob about the old export btw - it doesn't really work. It exports a huge amount of stuff that a lot of is totally irrelevant to a user. Did you look at it? :) Also the XML doesn't necessarily even come out valid - bug https://github.com/diaspora/diaspora/issues/4032

G

goob Sat 1 Feb 2014 10:55PM

@jasonrobinson I've not looked at it in detail, just saw that it contains contact information, post information and one or two other things (I think, can't remember now). When I said 'it works', I simply meant that the export button does still provide you with a dump - not necessarily that it was in any form which would be useful for importation elsewhere.

G

goob Sat 1 Feb 2014 10:55PM

@rich1, now you're in the swing with JSON, how about creating an export feature...? ;)

DU

[deactivated account] Sat 1 Feb 2014 10:58PM

I could write one in .NET - not sure it'd be of much use to anyone though?? Unless it's externally hosted perhaps... Hmm... I'll have a think about that :)

JR

Jason Robinson Sun 2 Feb 2014 8:11AM

@rich1 well it needs to be part of the d* core code - but if you have good ideas on the structure, feel free to create some specs on the content ;)

JR

Jason Robinson Thu 6 Feb 2014 9:30PM

//offtopic
Hey, new version of diaspora-tools out, Python script to migrate contacts across pods - a little more reliable now, uses fresh diaspy and also works out of the box with SNI SSL pods with Python 2.x (like my pod :P).

https://github.com/jaywink/diaspora-tools
offtopic//

R

Ryuno-Ki Mon 10 Feb 2014 10:35PM

As written down in my explaination: I'd go for JSON as it comes to contacts/aspects, since those are simply data.

On the other hand, posts are embedded in context, which XML can better map, I think.

When it is zipped anyway, why not split those information?

CG

Christian Giménez Mon 10 Mar 2014 8:47PM

Hey! :-) What's going on with the "import" feature?

Is it developing?
Cheers!

SVB

Steffen van Bergerem Tue 30 Sep 2014 11:44PM

This is a really old topic and we discussed a lot to find out how to implement this feature but codewise nothing happened. I'd suggest that we divide the feature in some smaller steps so we can see some progress with less effort. The first step could be:

Implement manual import/export from the settings page including the following user data:
name, 5 profile tags, bio, location, gender, birthday, nsfw profile, searchable profile, language, show community spotlight in stream, email notifications

We will need two buttons: one to export the data (as json?) and another one to upload a file and overwrite the existing settings with the ones given in that file. Later we could let the user decide which profile information he/she would like to export/import.

The implementation should be easy and even could be doable for a newcomer (as long as he knows some RoR).

What do you think about this? Is it a good idea to divide the feature into smaller ones like I proposed or am I missing something?

JR

Jason Robinson Wed 1 Oct 2014 5:36AM

Absolutely - small steps as all other things. Maybe we could make a master task with subtasks - that approach seems to have worked well with the UI rewrites?

@ruxton I think you we're working on some parts of this - any progress?

M

MrFrety Fri 3 Oct 2014 11:55AM

I agree with @steffenvanbergerem .

The next step is enabling re-import of the existing .xml-profile-exports.

SVB

Steffen van Bergerem Fri 3 Oct 2014 2:14PM

@mrfrety I am not sure if we really want that. Even the export we have right now looks broken to me and adding support for the existing xml files adds extra complexity. As a first step I'd prefer export and import of basic profile information. The next steps would be adding more information to the import/export feature and drop the old xml export as soon as we are able to export (and import?) all personal data.

I think there is just a small number of users who already closed there accounts and really need to import their old xml files. For those we could create a tool (separated from the diaspora* codebase) which converts the old xml file.

JR

Jason Robinson Tue 14 Oct 2014 4:09PM

I made a new "meta issue" to track this whole thing with different smaller tasks - https://github.com/diaspora/diaspora/issues/5343

Plz feel free to work on any of these :)

PS

Pavithran S Fri 8 Jan 2016 1:17PM

Nothing much happened in the Jasons post https://github.com/diaspora/diaspora/issues/5343 . I and many other community members want this feature desperately and we all remember that backing up and shifting house to another pod was promised by diaspora* long back.

U

unity100 Fri 26 Aug 2016 6:38PM

Im commenting here by assuming that with personal data import/export, we also mean pod migration.

This, is a critical issue for a decentralized network:

Biggest issue with there being no pod migration is that if the pod i
am in goes bust, my account, connections, everything go bust.

that practically kills all the purpose of decentralization.

as long as i am locked into one pod, i am on a centralized network.
content spreading in a decentralized fashion does not make up for it.

as appalling as it is, i can trust that my account on facebook will be
there for a loooong time. even if they use my data as a means to
profit off of me by selling it to government and advertisers, that
also ensures continuity of my account.

but with diaspora, i am obliged to trust the effort of a small group
of people or even one person setting up a server somewhere and then
being able to maintain it through the years. which can go bust at any
time due to a million reasons.

if, somehow, an established organization with endurance sets the
server up (ie a corporation, a ngo etc) i have to be wary of their
motives. yeah that's a great diaspora server with tens of thousands of
users, but it can even be a front by an intelligence organization to
spy on their users - they do it with tor.

or i can set up my own server, and have to deal with everything that
comes with setting up and maintaining a server. which, many of
diaspora users cannot do.

or i can use the apparently not so mature p2p client and battle with
many issues.

G

goob Fri 26 Aug 2016 6:57PM

@unity100 that's why this is one of the highest priorities for the project, and why community members contributed €4750 towards the development of this important feature. You can follow the current progress of development via the #senyadiasporaaccountmigration tag.

U

unity100 Fri 26 Aug 2016 7:55PM

thats great to hear, goob. i already have a wide social circle on diaspora, but the fact that i wouldnt be able to move my account if something happened was considerably discouraging me from using it as my main social service.

i look forward to this feature.

G

goob Fri 26 Aug 2016 10:56PM

It is taking a long time to complete it properly, because it is a very complex mechanism to produce a migration feature which is both secure and easy to use, but it is absolutely one of the highest priorities for development.