Loomio
Fri 8 Jan 2016 1:18PM

Backup and restore

PS Pavithran S Public Seen by 262

The last I heard about backup and restore hasnt been given much interest. Any reason for that?
https://github.com/diaspora/diaspora/issues/5343 has some progress.

CS

Comrade Senya Fri 8 Jan 2016 2:27PM

I hope I can participate in it in a while, after some present jobs are done.

JR

Jason Robinson Fri 8 Jan 2016 9:20PM

The latest automatic back up / restore spec proposal includes the import parts in this issue. Would be fancy to get more comments on it, split it into smaller chunks and dump a lot of bounty money on the issues - maybe get something to happen :)

Anyone wanting to give their insight, please check the proposal and comment in issue #908 . If no comments are received after some time I could just create a proposal to accept the proposal as a guideline for development.

Please don't start working on any import thingy before at least considering that proposal ;)

PS

Pavithran S Sat 9 Jan 2016 1:50PM

Its a very detailed proposal. Isn't there a simple way like mysql dump to csv/json and import csv/json ? Pod to pod communication sounds very complex.

JR

Jason Robinson Sat 9 Jan 2016 6:45PM

Dumping database would not work for ordinary users very well - and certainly would not work in the situation where the pod just disappears. And, importing raw SQL dumps into other pods would probably not be a very nice thing to do :)

Regarding JSON, yes we already allow exporting data. The proposal details how this data could be to not only restore the data to a new pod (profile, etc), but also migrate the identity, and thus take control of previously shared posts etc.

PS

Pavithran S Sat 9 Jan 2016 8:59PM

Regarding JSON, yes we already allow exporting data.

Yes now we need an import function but your proposal looks too big and lot of stuff needs to be implemented. Agreed its the optimal way to go forward.

but also migrate the identity, and thus take control of previously shared posts etc.

Can't we have something just for exporting all the posts and its comments which are imported back under new account. " Taking Control" sounds complex.

JR

Jason Robinson Sat 9 Jan 2016 9:02PM

Can’t we have something just for exporting all the posts and its comments which are imported back under new account. “ Taking Control” sounds complex.

That would create duplicates and there would be no interaction possibility. No comments to the old ones would arrive to your new identity, for example.

Sure, you can have (almost) anything (sane) - just find a coder to do it :) The issue has been there for a while waiting for someone to grab it. This proposal aims to solve the disappearing pod problem and clean migration. That doesn't stop anyone doing partial or full imports before a "proper" migrate solution is done.

SVB

Steffen van Bergerem Sat 9 Jan 2016 9:04PM

Can’t we have something just for exporting all the posts and its comments which are imported back under new account. “ Taking Control” sounds complex.

So you would like to share all your previous posts again from your new profile?

CS

Comrade Senya Fri 22 Jan 2016 2:33PM

So we have User and Person models. On migration we must create a new user but do we preserve the same Person model which was known for that person? I suppose we should.

CS

Comrade Senya Fri 22 Jan 2016 2:39PM

Profile data export package contains also posts and comments which can be ignored during the restore process.

I don't think posts and comments can be ignored. It is possible that the new pod doesn't contain some posts (especially private). I believe it must not be dropped and we should fetch them on restore also.

CS

Comrade Senya Fri 22 Jan 2016 2:42PM

https://wiki.diasporafoundation.org/Account_Backup_And_Restore#Backup_delivery_message

"backup" is signed and encrypted using the user private key

Where do we store signature? Don't we have to have a separate field in the schema for that?

CS

Comrade Senya Fri 22 Jan 2016 3:08PM

Pods should schedule backups of user data once a week to the backup pods.

I think it is not frequent enough. Some people post frequently, and one week would be a huge loss for them. How about making that adjustable according to person's activity level? Or simply backup every day if there were changes. If there weren't, don't backup at all.

CS

Comrade Senya Fri 22 Jan 2016 3:19PM

I also beleive we must support some kind of limitation for pods backup capacity. For example a pod wants to store backups, but no more than for 100 people. Then it shows that it doesn't receive backups anymore, but previous still work.

JR

Jason Robinson Fri 22 Jan 2016 7:30PM

So we have User and Person models. On migration we must create a new user but do we preserve the same Person model which was known for that person? I suppose we should.

Yes, we should keep the same Person as it is the same identity, just moved local instead of remote.

I don’t think posts and comments can be ignored. It is possible that the new pod doesn’t contain some posts (especially private). I believe it must not be dropped and we should fetch them on restore also.

Well, it's not feasible to import posts or comments imho. What purpose would that even solve? Your contacts would not see any of the uploaded private posts unless you also push them out to them - which would be bad because they already have them from before.

The main point is to save the identity, not every crumb of data related to it.

Where do we store signature? Don’t we have to have a separate field in the schema for that?

This would be the same signing method we use for delivering content - ie the receiver only accepts it after verifying the signature in the payload against the public key of the person. Whether this should live in the diaspora federation gem where the code is readily available is up to question. I'd say no, but of course it would be easier to implement that way. I don't consider this part part of the protocol as such, but then that is not in any way under specification anyway.

I think it is not frequent enough. Some people post frequently, and one week would be a huge loss for them. How about making that adjustable according to person’s activity level? Or simply backup every day if there were changes. If there weren’t, don’t backup at all.

I believe this can be left as detail when the thing actually works. If we leave out posts and comments, which I think is the only real way to do it, then there is absolutely no sense in sending backups daily.

I also beleive we must support some kind of limitation for pods backup capacity. For example a pod wants to store backups, but no more than for 100 people. Then it shows that it doesn’t receive backups anymore, but previous still work.

Again, if posts and comments are not part of the archive, it will always be very small. Additionally, backup pods would be randomized, guaranteeing some balancing of load between pods. Of course we can introduce a setting like this, but personally it doesn't sound super useful, considering how much data pods store normally compared to how much this would add.

One interesting case is what happens if the pod has turned off sign-ups in the event that someone wants to restore? I think we should bypass the sign-up not being on and let the user restore. This would speak towards the setting you mention, so that pods don't gather too many possible identities that could activate suddenly.

Great comments, thanks!

CS

Comrade Senya Sun 24 Jan 2016 7:30PM

Well, private posts may contain some valueable information for the user herself. I don't have extra copies of texts I posted, and I don't like to lose these texts. Moreover it would be definitely nice to preserve a possibility to continue conversation on some private post that was going on before the move. I don't see why we should push them again. A guid of the post is preserved, so all new comments will be federated well to the right place.

Not to say about public posts which might be wanted to be shared with some new contacts.

So I think posts, the content is extremely important part of the network. That's why people are in it. And the restore feature without posts restore would look unfinished.

CS

Comrade Senya Sun 24 Jan 2016 7:38PM

Maybe if we push posts to the restore pod as if there were some subscriber on it could make restore easier, since we won't need frequent backup then and any special posts restore feature. At least we could do that for public posts, since for private posts that would imply trust for restore server even before a password was entered. That is not really acceptable.

CS

Comrade Senya Sun 24 Jan 2016 7:46PM

https://wiki.diasporafoundation.org/Account_Backup_And_Restore#Backup_delivery_message

“backup” is signed and encrypted using the user private key. Once opened, it should contain the following schema...

It's not very clear the structure of the backup field before we "open" it. It must be some data array containing signature and encrypted data, right?

CS

Comrade Senya Mon 25 Jan 2016 11:59AM

BTW, does any other software in the federation (friendica, redmatrix) do some sort of backup restore?

CS

Comrade Senya Tue 26 Jan 2016 6:33PM

So here are my changes to the spec according to what we've discussed

https://wiki.diasporafoundation.org/index.php?title=Account_Backup_And_Restore&diff=4454&oldid=4418

JR

Jason Robinson Tue 26 Jan 2016 8:26PM

@comradesenya unfortunately I'll unlikely have much time for going through these before friday - I'll reply then. Until then, it would be nice if the wiki could be left alone before accepting changes here in Loomio. I don't agree (still) to some things like including posts and comments in the backup archive and I'm not prepared to change my opinion unless some others join in and support that idea and also give valid technical ways to do it sanely which at the moment is missing.

Anyway, lets continue discussion on that, for a few days I'll have to pass on comments. I'll clean up the wiki page to reflect those items that have been mutually agreed upon then.

Once we have a clear understanding we can do a proposal.

CS

Comrade Senya Tue 26 Jan 2016 9:10PM

TBH, I don't see any technical problems on posts restore. Everything seems to me pretty straightforward. We just add the posts to the database of the backup pod if they aren't there yet. Maybe I miss something?

On contrary, not restoring posts would lead to weird situations, like when after you've moved some of your contacts does comment on an old post of you. Comment gets federated to your new pod, but parent post is not there! Because we haven't merged it.

JR

Jason Robinson Sun 7 Feb 2016 8:57PM

So, I've now cleaned and published the spec for final discussion, cleanup and voting.

@comradesenya I left in your good idea to retry moved messages (which a small change to play with status codes instead of sending more messages around). But I removed the pointer regarding posts and comments and filed it as an issue instead.

There are also quite a few TODO's in the spec (will log issues out of these) - and I'm pretty sure there are in general things that need fixing.

All in all, no one has given strong opposition to the idea itself, so hoping we can lock down a spec 1.0, approve it for implementing in diaspora* and then it can be implemented (hopefully by @comradesenya ;)).

So basically;

  • The spec is written as markdown and should be worked on via preferably issues and pull requests
  • I chose Gitlab because not everybody likes github and because it is nice and because TheFederation was not available in Github ;) You can create an account using your Github login easily. For those who want to participate but don't like Gitlab, please feel free to use other ways to communicate or even send patch files if you want.
  • The spec is "owned" by The Federation, or more like namespaced. I would like to keep editorship until 1.0 at least though.
  • The spec was generalized into diaspora* like federated social networks, not just diaspora*. I like the idea of being able to move across platforms too. Platform specific stuff doesn't belong to the spec but should be left to implementation.

The spec is live as a working draft version 0.1.0 here: https://the-federation.info/specs/backup-restore/

The git repository is here: https://gitlab.com/TheFederation/backup-restore

Sorry this took a little while. Ping also @jhass as you've commented quite a bit on this before in github.

CS

Comrade Senya Mon 8 Feb 2016 4:54PM

There is a thing we haven't discussed before.

If the pod from which a person has moved to some new place is still alive, shouldn't we block registration for that name on an old pod at least for a while?

  1. If a user has moved, and some new user registers on the old pod with the same name as the previous user had, then it could be possible, that somebody will try to discover the old person by the handle or even send a message

  2. It is theoretically possible, that somewhere exists a pod, which has discovered a person, but the new pod didn't know about that pod, so "I've moved" message wasn't sent to it. In this case discovery with webfinger could return 302 code and "I've moved" message as a body so that the pod could process this message instantly.

  3. At first after the feature is introduced in the diaspora* source code, it will be usual that there are many pods that are not up-to-date and they won't get "I've moved" message in time. So, probably, it would be nice to block registrations with the same handles as some moved users had, to lower inconsistencies produced on the network.

  4. At first I thought about making kinda blocking period for a handle before it could be available again. However, I think, that we could consider user handles as an inexhaustible resource, so probably we don't need to unblock them authomatically after a period someone has moved. Maybe we could introduce an action at the podmin page to free the account that was previously occupied by someone who had moved with statistics of the discovery requests for this old account shown. Then, the podmin could free this account on request by user like "Hi! Could you please make the handle of a previously moved person available for registration again?".

CS

Comrade Senya Mon 8 Feb 2016 5:00PM

TODO: To avoid an extra endpoint, should we use a version of NodeInfo instead?

@jhass had strong objection against it, so probably his opinion is to be considered.

CS

Comrade Senya Tue 9 Feb 2016 4:57AM

TODO: Define full schema of content or place content keys in the first level.

I guess it's fine if I base the content schema on this?

JR

Jason Robinson Tue 9 Feb 2016 8:39PM

If a user has moved, and some new user registers on the old pod with the same name as the previous user had, then it could be possible, that somebody will try to discover the old person by the handle or even send a message

We could add a recommendation to the spec that old local usernames are reserved permanently - that would make sense since that is what happens when an account is deleted in diaspora afaik (though would have to check to make sure). I've made an issue.

But really, the keys will have been regenerated so there isn't really a possibility of hijacking an identity like this.

It is theoretically possible, that somewhere exists a pod, which has discovered a person, but the new pod didn’t know about that pod, so “I’ve moved” message wasn’t sent to it. In this case discovery with webfinger could return 302 code and “I’ve moved” message as a body so that the pod could process this message instantly.

You mean store the whole "I've moved" message payload on the old pod and respond to webfinger queries made towards the old closed identity. That sounds good otherwise but to make it work webfinger would have to pick up the message as a response. I think that might be a bit outside spec?

3 and 4 relate to 1 - where I think we should just recommend reserving forever.

TODO: To avoid an extra endpoint, should we use a version of NodeInfo instead?
@jhass had strong objection against it, so probably his opinion is to be considered.

Well, I didn't mean the NodeInfo but a NodeInfo :P But, I think it is out of scope of this spec anyway, better keep it simple with a dedicated clean endpoint.

I guess it’s fine if I base the content schema on this?

Yes, but generalized for the spec, so we shouldn't include all the keys but a generic set. diaspora* can of course implement a larger set containing the full amount of data used in our profiles.

DS

Dennis Schubert Wed 10 Feb 2016 9:08AM

Yes, but generalized for the spec, so we shouldn't include all the keys but a generic set.

If you truely want to support "all" networks, you have to define a minimal set of keys. Otherwise, diaspora may use username where other networks might use user and everything becomes a complete mess.

CS

Comrade Senya Wed 10 Feb 2016 7:27PM

I believe we may define a minimal set of keys as required and everything else as optional, so other networks pick what they would like to support.

JR

Jason Robinson Wed 10 Feb 2016 7:36PM

Defining a single all-around schema with strict keys is hard, especially as we ourselves use pretty unique names like "aspects". Making our export not use our own key names just to support this would be odd.

What about aliases? The spec defines the basic expected keys using the most common key names that would be expected in a generic spec, and then we define a set of aliases, for example aspects could map to something that might be more generic like contact_groups.

These aliases can be expanded as seen fit in future spec versions.

CS

Comrade Senya Wed 10 Feb 2016 7:44PM

In the PR I introduced format by generating a schema basing on our exported archive json document using http://jsonschema.net/

Then I manually edited it to fix some issues and I removed most of the keys from the "required" section, leaving
"name",
"private_key",
"profile",
"contacts".

The latter two - "profile" and "contacts" beign objects themselves doesn't have required fields though. So we can either make them optional as well, or make some of their contents required.

DS

Dennis Schubert Wed 10 Feb 2016 7:44PM

Defining a single all-around schema with strict keys is hard, especially as we ourselves use pretty unique names like "aspects".

Ah c'mon. What do you want to achieve? Do you want something that looks fancy but nobody is able to use between implementations or do you want to have a spec that defines a standard way to exchange account data?

If you want a spec, then define keys. Define username, contacts, groups, birthday and stuff. No alias. One key has exactly one descriptor, no exceptions. How implementations call these things internally is completely irrelevant to your spec, since you should define a data format, not a set of rules on how social networks should behave internally. Diaspora may put diaspora_id in the username field, Friendca my call the same database column lookhowfancytheuseriscalled. The spec does not care. And the spec should not care.

JR

Jason Robinson Wed 10 Feb 2016 7:48PM

@dennisschubert so how do we use our JSON exports to restore then as they would be incompatible - or are you saying the diaspora JSON exports would be renamed?

DS

Dennis Schubert Wed 10 Feb 2016 7:50PM

If you write a spec that is well-written and suitable for diasporas need (that is, we can somehow export all fields), I'm sure it's an easy task adapting the spec...

DS

Dennis Schubert Wed 10 Feb 2016 7:53PM

Think about the other side, the people implementing specs. What would you do if you see a spec that literally says "username contains the users name. Be aware that this field also may be called profile_name, name or fancyfancy". How are you supposed to implement that? Rolling dices while implementing specs is surely not a good thing.

CS

Comrade Senya Wed 10 Feb 2016 8:20PM

Do we follow the federation protocol for the "Delivery package" and for the "I've moved" messages? If so, we must use XML instead of JSON.

CS

Comrade Senya Wed 10 Feb 2016 8:25PM

Also, if we reuse federation protocol for these messages, we don't have to define signing methods in our spec, since signing is already implemented on the salmon level.

CS

Comrade Senya Wed 10 Feb 2016 8:30PM

But there is a problem, if we want to rely on the federation protocol in the spec, we must refer to the federation protocol spec to stay formal, but AFAIK there is no formal specification document for the federation protocol.

CS

Comrade Senya Wed 10 Feb 2016 8:48PM

And in this case "Backup server receive route" is the usual public receive route for the server.

JR

Jason Robinson Wed 10 Feb 2016 8:50PM

I don't think we should mix this and diaspora federation together. At least the spec can't be written like that.

More comments tomorrow, zzzz...

CS

Comrade Senya Wed 10 Feb 2016 9:02PM

Well, I don't know, that is exactly how we can reuse endpoints and signing code also. I feel that this is optimal to reuse the federation protocol, but not to invent one more way to exchange messages. To write the spec we'll need some formal definition of the federation protocol then. At least some short version.

@dennisschubert, what do you think?

DS

Dennis Schubert Wed 10 Feb 2016 9:05PM

but AFAIK there is no formal specification document for the federation protocol.

True, and at this point, there is no point in writing one. After the federation-gem efforts are done, I intent to formalize what we have at that point so we have a reference documentation for further usage and other networks as well.

I'd like to add something for Jason here. You are trying to write a spec, which is much appreciated. "Spec" is short for specification, but I get the feeling that you are actually trying to be as vague as somehow possible. I see why you're trying to do this, but if you want to come out with something that people might actually use, you have to do the opposite. Be as specific as possible. Otherwise, people will never be able to adopt the spec, everyone is going to maintain their own set interpretation errors and nothing will ever be compatible.


Example: Defining terminology.

Bad:

username contains a string.

Good:

username contains an identifier that uniquely identifies an user across the network.


Example: Defining formats.

Bad:

groups contains an array of contact groups.

Good:

groups contains an array of objects specifying groups. Group objects should contain a title attribute as well as a members array. title should be the human readable identifier. members should be an array of usernames.


Do you understand what I'm trying to say?

JR

Jason Robinson Thu 11 Feb 2016 8:09PM

Regarding diaspora protocol. Yes, the initial version in the wiki that I wrote in October had the idea that the whole backup/restore would be implemented with minimal changes by hooking up to the current federation endpoints, signing, etc. This is definitely the easy route, I and to be honest I wouldn't of imagined any other way.

However, it is clear that there is heavy work going into the protocol itself and I'm not at all sure it makes sense to smash a concept like this that has absolutely nothing to do with social content in. This would weaken not only the protocol but also make the backup/restore feature unstable due to breaking changes in the protocol itself.

As such, I began looking at it from outside of diaspora. As a separate feature, not conflicting or requiring the diaspora, or any other, protocol, it would become much more stable to implement. The downsides are of course clear; more time spent on defining the spec (or specification, thanks Dennis) and more time spent on development (signing methods, endpoints, etc). The end result however would be cleaner and more stable not just for the backup/restore feature, but also the diaspora federation protocol.

I know it is hard for people here to see the federated web outside the context of diaspora, but that is exactly what I would like to do here.

So, there are two ways forward:

1) Implement using the diaspora protocol. Some of the code would then have to be implemented in the diaspora federation gem, making it a permanent feature there.
2) Continue writing a more generic specification and implement using that, with no changes at all to the diaspora federation gem.

I'm not too interested in 1) to be honest, but if that is chosen, then imho implementation can start whenever a basic flow is agreed upon, and it's already there written waiting for comments. There is no need to write fancy specifications because implementing it would require understanding something that has no specification and that needs reverse engineering. If anyone is able to do that they should be able to follow this half done spec already.

I'm really interested in 2) and that doesn't really even depend on diaspora implementing it. For that, there is much work to do and many comments needed, from outside of diaspora developers. I'll do what I can to get some comments.

Btw, regarding 2), I'd like to make use of as much of existing or upcoming standards as possible.

Using ActivityStreams2 for the JSON messages for example would enable using existing and upcoming parser libraries - the spec is currently a W3C working draft and is moving on nicely. Backup/restore would use extensions to it where needed as per AS2 spec.

For signatures, there are a few that have been mentioned within the W3C SocialWG group. The most important thing would be to use something that has support on implementation or standardization level. It is possible signatures will be a part of ActivityPub, the federation spec from the SocialWG, but this is not guaranteed unfortunately.

So, how should it be for diaspora - backup/restore on top of the diaspora protocol or backup/restore implemented as a feature nothing to do with the diaspora protocol?

Should I create a proposal now, seems a good thing to decide before moving on with specifics?

DS

Dennis Schubert Thu 11 Feb 2016 8:12PM

I still don't see the connection between a spec describing how data should be exported and a spec describing how servers should communicate with each other.

JR

Jason Robinson Thu 11 Feb 2016 8:16PM

I still don’t see the connection between a spec describing how data should be exported and a spec describing how servers should communicate with each other.

You're looking at only the export/import part - which is manual. I only added that by request from @jhass . The main idea of the spec is to make backups automatic, using the whole network as a way to protect user identities (= one pod goes down, only some backups are lost temporarily).

To make this kind of automated system, servers need to understand each other. I'm not sure how that can be done without defining how servers talk to each other in the spec.

DS

Dennis Schubert Thu 11 Feb 2016 8:27PM

I see. As usual, I like well-defined standards more than self-hacked stuff. If you feel like AS is your way of exchanging stuff, sure, go ahead.

But how does ActivityStreams solve your issue? Sure, you got a way to exchange json objects, but ActivityStreams is rather incomplete at defining the fields (which, obviously, is done on purpose to keep AS extendable). You still have to very carefully design the actual fields to export? Isn't that the main issue?

JR

Jason Robinson Thu 11 Feb 2016 8:51PM

You still have to very carefully design the actual fields to export? Isn’t that the main issue?

Yes that is one large issue. It's certainly not the main issue imho, I'd say the signing is very critical.

Anyway, assuming diaspora as a project would want a backup/restore feature, and let's assume for the sake of this discussion that the answer is yes, a decision needs to be made I think whether to implement it on top of the current federation layer or implement it as a separate feature not related to federation layer.

And if you want to discuss content further, I'd say implementing on top of the current diaspora protocol we should just go with the current fields as per export schema (more basic set for the automated backup archives, but from the same set). A clearly defined schema is needed however for 2).

Btw, it's nice you didn't even read the document before commenting on it ;)

DS

Dennis Schubert Thu 11 Feb 2016 8:53PM

Oh, don't worry, I have read it. I stopped at the point you tried to specify how the user has to sign in and what buttons he has to click as well as the point where you started to assume rules about how implementations should store their stuff in the database, which is why I wrote the initial comment. Nice try, though. Maybe it'll work next time.

Feel free to open a proposal.

JR

Jason Robinson Thu 11 Feb 2016 8:56PM

Thanks Dennis, as polite to other people as always :)

JR

Poll Created Thu 11 Feb 2016 9:30PM

If diaspora* implements the backup/restore specification, should it base it on the diaspora federation protocol? Closed Sun 28 Feb 2016 8:07PM

Assuming diaspora* might want to implement a backup/restore process (which enables migrating across pods), should it build this support into the diaspora protocol or base it on a separate specification, not modifying the diaspora protocol to support this feature?

Note! The backup/restore spec COULD be one like https://the-federation.info/specs/backup-restore/, but this proposal is NOT about whether to implement this specification or not.

Votes:
* YES - any backup/export type specification should be built on top of the diaspora federation protocol
* NO - any backup/export type specification should not be implemented on top of the diaspora federation protocol.

For clarifications, see for example comment https://www.loomio.org/d/qsVJ2K1t/backup-and-restore#comment-923083

Results

Results Option % of points Voters
Agree 40.0% 2 H CS
Abstain 60.0% 3 RZ B C
Disagree 0.0% 0  
Block 0.0% 0  
Undecided 0% 262 BK ST FS MS TS AA S CB HF BO JH DM GC JH JR F RF M EG G

5 of 267 people have voted (1%)

CS

Comrade Senya
Abstain
Thu 11 Feb 2016 10:26PM

Well, I like both ideas. Can't decide yet. See my comment.

RZ

Renato Zippert
Disagree
Fri 12 Feb 2016 12:20AM

Looks safer to me to avoid allowing user information traveling on the network in case of any kind of vulnerability that could trigger the process, entirely or partially, without user consent, to a malicious destination or with a man-in-the-middle.

RZ

Renato Zippert
Abstain
Fri 12 Feb 2016 12:33AM

Looks safer to me to avoid allowing user information traveling on the network in case of any kind of vulnerability that could trigger the process, entirely or partially, without user consent, to a malicious destination or with a man-in-the-middle.

RZ

Renato Zippert
Abstain
Fri 12 Feb 2016 8:48PM

Can't see clearly the benefits or issues of both options.

B

Bady
Abstain
Tue 23 Feb 2016 4:47AM

It'll be great if someone can explain the difference between two (backup/restore based on and not based on federation protocol) in layman terms so that people like me can take a better decision about it!

B

Bady
Abstain
Tue 23 Feb 2016 4:48AM

I see lot of comments here, but it'll be great if someone can explain the difference between two (backup/restore based on and not based on federation protocol) in layman terms so that people like me can take a better decision about it!

B

Bady
Abstain
Tue 23 Feb 2016 4:49AM

I can see a lot of comments here, but it'll be great if someone can explain the difference between two (backup/restore based on and not based on federation protocol) in layman terms so that people like me can take a better decision about it!

CS

Comrade Senya
Agree
Tue 23 Feb 2016 11:10AM

I believe, reusability is valuable and the acc. b/r spec may be written in a neutral way (no links to federation, so may be used outside) while staying compatible with the federation protocol.

H

hewiak
Agree
Wed 24 Feb 2016 8:38PM

usability trumps just about all other arguments, of which security remains among the most important

C

CSammy
Abstain
Thu 25 Feb 2016 1:12AM

I'm against voting exclusively for one or the other solution. To me, it is not clear whether this vote is on single accounts or entire pods. I also don't want to vote over anything in regards to federation before the new federation gem is finished.

CS

Comrade Senya Thu 11 Feb 2016 10:40PM

Well, I like both ideas. The first option I like for its simplicity of implementation. The second is good as it is a completely separate piece of functionality, and the generalization would make it possible to use the spec even in some applications where we don't have social media at all. Like integrating backup/restore in some completely different platform, like, for example, Loomio. It is a big, complex and interesting task. The main question is whether it makes much sense to implement this spec somewhere outside the diaspora* federation world? If one could name a few decent applications of the spec outside the world of the software which supports the diaspora* federation protocol, I would probably support the latter proposition.

I believe that the issues with diaspora* federation protocol instability is not a big deal for the topic, because there will be transition periods which will guarantee backward compatibility for sensible periods of time, so it'll be completely transparent for the feature because we operate on the upper level and the transport level is provided to us.

DS

Dennis Schubert Fri 12 Feb 2016 12:23AM

@renatozippert "disagree" does not mean the data will not be sent over the network. Apparently, that's not even a point open for discussion here. The question is if the spec should be based on diasporas protocol or something different like ActivityStreams.

RZ

Renato Zippert Fri 12 Feb 2016 12:32AM

Thanks @dennisschubert... I'll abstain for now then and read more about it. I need to understand this better.

JR

Jason Robinson Fri 12 Feb 2016 6:00PM

If it helps, we can first vote whether to work on this feature at all - if that is required to continue. I feel it would be better to decide how to do it first and then vote to approve it. It would not be fair to vote on approving something that isn't finished as a plan.

@renatozippert the whole backup/restore idea here is about automatic encrypted backups over the network. If you are wary of this please vote disagree when there is a vote to approve the feature for implementation - as said the current proposal is not approving the feature for implementation.

@dennisschubert - ActivityStreams2 is not a protocol, it is a content serialization specification. I doubt it can be used to federate anything...

RZ

Renato Zippert Fri 12 Feb 2016 8:54PM

The problem of basing this on the protocol would be having to change the protocol that is related to other projects?
Right now I tend to agree with basing on the protocol, as creating the backups and moving profiles would be much more elegantly implemented with protocol support, not as something "external".

FS

Florian Staudacher Sat 13 Feb 2016 12:19PM

Ultimately I would love to see our federation protocol getting used for all kinds of stuff, but right now I think it wouldn't make sense to take it much further than "social media" content.
For me, an account backup-replication protocol is out of scope of the original federation. I'd imagine in a far away future you could sync your account backup to some not-even-related-to-diaspora service (e.g. owncloud?) and that shouldn't depend on implementing anything else other than the backup endpoint, imho.

CS

Comrade Senya Sun 14 Feb 2016 11:30PM

Ok, how about the following idea.

We agree on the backup/restore message set and implement it with both federation and non-federation based transport protocol? This is redundant work, but not that big, and it will make happy everyone. After a while, we'll see, which version got integrated to the third party services (that's why we standartize it at all) and drop the one, that is redundant. I read that this is how decisions were made in Linux kernel sometimes - by including two competing technologies.

So, as for the specification, it must then be made agnostic to the transport protocol used.

P.S. Does XMPP have anything on the topic?

E

Elm Fri 19 Feb 2016 11:43AM

Hi ! I do not understand what is technically going on here, but it is great to hear that this fundamental feature is on its way. Thanks !

G

goob Tue 23 Feb 2016 2:24PM

I don't consider myself competent to vote on this, so will leave it up to those of you with the technical knowledge required.

CS

Poll Created Mon 29 Feb 2016 2:50PM

Shall we encypt the backup archive when a user exports it manually? Closed Tue 8 Mar 2016 1:07AM

Currently we have the archive export feature and it is being exported unencrypted. It is convenient since a user can browse the contents of the archive himself. On the other hand, if the user by his mistake (or some malicious software) passes the archive to a third party, it would lead to bad consequences. We might encrypt the archive with a password like it was proposed for the automatic backup feature. (Here is a possible rough implementation of the archive encyprtion presented.) This way it gonna be more safe, but unreadable without some extra tools (Here is a snippet for decrypting archives produced by the modified export feature).

Results

Results Option % of points Voters
Agree 15.4% 2 E MDB
Abstain 23.1% 3 R FL C
Disagree 61.5% 8 G DS SVB S PS BK G C
Block 0.0% 0  
Undecided 0% 257 BK ST FS MS TS AA S CB HF BO JH DM GC JH JR F RF M EG G

13 of 270 people have voted (4%)

DS

Dennis Schubert
Disagree
Mon 29 Feb 2016 7:46PM

If the user is not able to keep the manual export safe, he is also not able to keep the key/passphrase used for decryption save, so there is no loss and no win.

G

goob
Disagree
Tue 1 Mar 2016 6:51PM

If a user wants to encrypt their backup they can do so once they have downloaded it. I think we should have 'encryption' when restoring a backup to a new account, by using a private key produced by the originating pod, or something like that.

C

Chris
Abstain
Tue 1 Mar 2016 7:38PM

Isn’t this why we use https? So that files are transferred over encrypted lines? I think it would just introduce another layer of complexity, for not much benefit.

G

Globulle
Disagree
Fri 4 Mar 2016 6:41AM

It is useful to strongly warn the user to keep the archive in a safe (and potentially encrypted) place, but this is not diaspora*'s responsibility to do it.

PS

Pavithran S
Disagree
Sat 5 Mar 2016 1:21PM

Please don't overdo things and increase complexity. Its users duty to encrypt or keep his data save. Since it was mentioned by Chris that we anyways use https for transfers shouldn't that be enough? :open_mouth:

MDB

Mathijs de Bruin
Agree
Mon 7 Mar 2016 8:33AM

Voting behaviour of community members should be private at all times. When using OpenSSL as in the snippet decryption is simple enough. When in doubt, make it opt-out.

E

Elm
Agree
Mon 7 Mar 2016 12:59PM

I’m OK for encryption if this feature can be coded afterwards (since it is not first on the priority list) and is made “opt-outable” (for people who does not need such privacy security).

FL

Frode Lindeijer
Abstain
Mon 7 Mar 2016 4:19PM

How about making it optional through an "Encrypt exported data" checkbox? :monkey:

C

CSammy
Disagree
Mon 7 Mar 2016 9:12PM

It adds unnecessary complexity for a feature that's ultimately the user's responsibility. Export should not be possible without using https though. If I'd want my backup to be encrypted, I'd decrypt the pre-encryp. backup and encrypt it myself again.

S

SuperTux88
Disagree
Tue 8 Mar 2016 12:47AM

This doesn't add security, because the password/key would be sent over the same connection as the backup-archive. It only adds complexity on the server side and on the user-side, if the user wants to open the archive.

BK

Brad Koehn Mon 29 Feb 2016 3:07PM

I would recommend not encrypting the data; it adds additional complexity to the system that the developer and user ultimately needs to keep track of, introduces additional failure points, while not providing much in the way of actual benefit to the user.

Any user can encrypt the data him/herself using a variety of tools once its downloaded.

DU

[deactivated account] Mon 29 Feb 2016 9:48PM

I also would recommend not to encrypt the data; however it would be good to tell users, that the data is unencrypted and should be kept in a safe place before downloading.

DS

Dennis Schubert Tue 1 Mar 2016 6:52PM

@bradkoehn Blocking proposals is only valid if the proposal itself is invalid, which it is not. Please don't block, but disagree.

C

Chris Tue 1 Mar 2016 7:37PM

I'm going to abstain from voting because I've had next to zero participation in the whole backup/restore conversation. And I feel like there are people here who know better than me what should happen. But I have this to say:

Isn't this why we use https? So that files are transferred over encrypted lines? I think it would just introduce another layer of complexity, for not much benefit.

And correct me if I'm wrong, but if you were going to send an encrypted archive, it would have to be encrypted on the server side...using keys generated....somewhere; either locally or on the server and they'd have to be sent over the same (again, already encrypted) lines we're worried about someone tapping. I'd rather just do all of it locally.

I've just repeated what other people have already said, and probably poorly. So, my current recommendation is no, but I'm not an expert so I'm not voting.

Maybe encryption should be optional?

G

goob Thu 3 Mar 2016 8:01PM

Blocking proposals is only valid if the proposal itself is invalid, which it is not. Please don’t block, but disagree.

When was that decided, Dennis? My memory is that the decision was that a 'block' was considered a 'strong disagreement' and no more. At the moment the only proposal regarding this that I can find is this one. I'd be grateful if you'd point me to the proposal which changed this so that I know the current situation. Thanks.

DS

Dennis Schubert Fri 4 Mar 2016 8:16AM

When was that decided, Dennis?

Actually, good question. That's how we handled in the past. Do you feel like we should write an official documentation?

CS

Comrade Senya Fri 4 Mar 2016 8:45AM

This says:

Block does not prevent the proposal from passing; it is simply a strong message that more discussion might be needed.

DS

Dennis Schubert Fri 4 Mar 2016 8:45AM

So I shall be wrong! Sorry. :)

G

goob Fri 4 Mar 2016 3:46PM

Oh, thanks for finding that, Senya. I knew I'd read it somewhere!

CS

Poll Created Tue 10 May 2016 10:20PM

Include comments of other people to the user's data archive Closed Sun 22 May 2016 10:02PM

Outcome
by Comrade Senya Tue 25 Apr 2017 5:44AM

Proposal passes, comments will be included in the user's archive.

We have the feature request for this. The user's data archive which we export now includes only user's own comments. But if we want to import posts with comments from the archive, the comments won't look consistent unless we also include other people's comments for that posts to the archive. However it maybe viewed as data safety violation. Whether we should add other people's comments to the user data archive?

Results

Results Option % of points Voters
Agree 66.7% 12 EG R E EK S KAK BK G CS MO DD EG
Abstain 16.7% 3 A B C
Disagree 16.7% 3 JR DS W
Block 0.0% 0  
Undecided 0% 253 BK ST FS MS TS AA S CB HF BO JH DM GC JH F RF M G AX PC

18 of 271 people have voted (6%)

CS

Comrade Senya
Agree
Tue 10 May 2016 10:26PM

The user who exports his data already has access to the other people's comments on his posts. So the export feature will just represent the data which is already available in a different manner, it won't expose anything new, so it isn't unsafe.

DS

Dennis Schubert
Disagree
Tue 10 May 2016 10:29PM

When a user comments under a limited post, the commenter expects that the comment will not leave the scope of given comment. During exporting, data can be lost (for example by people not caring about their stuff, uploading them to public sources...)

S

SuperTux88
Agree
Wed 11 May 2016 1:10AM

We can't control it anyway (everybody can copy/paste). Also, if the post-author moves to another pod, the new pod needs to know all comment, so that comment-authors still can delete their comments (which is relayed over the post-author)

G

Globulle
Agree
Wed 11 May 2016 7:20AM

When you post on the profile of someone else, you know that you are not completely in control. And as long as a limited post remains limited, I assume the risk of undesirable leak is very weak.

DD

Dominic Duffin
Agree
Wed 11 May 2016 5:37PM

Comments on a social network are in the public sphere, so I'd say that any slight privacy risk would be worthwhile to make an important feature work properly.

KAK

Karthikeyan A K
Agree
Thu 12 May 2016 2:09PM

Though privacy is a concern, I think its good to have comment backup.

JR

Jason Robinson
Disagree
Wed 18 May 2016 6:35PM

I believe we shouldn't make it easier to violate other peoples privacy.

C

CSammy
Abstain
Sat 21 May 2016 1:06PM

The backup may include comments of other people since it belongs to the user's history, but it should not be possible to restore someone else's comments without their consent.

B

Bady
Abstain
Sat 21 May 2016 9:28PM

I agree that backups are incomplete with others' comments, but it should be allowed only with the consent of the users who commented. Otherwise it will be a violation of Diaspora's core philosophical values regarding privacy.

R

RAM518
Agree
Sat 21 May 2016 9:46PM

One privacy issue is that a person's comment one some else's thread and a foreign pod could become searchable to web spiders, and then difficult for them to ever erase. Some pods have looser robots.txt rule and more stuff leaks out also.

A

aj
Abstain
Sun 22 May 2016 12:41PM

As a podmin i don't need backup/restore within D.

A

aj
Abstain
Sun 22 May 2016 12:44PM

As a podmin i don't need backup/restore within D, but i think it would be reassuring to users on larger pods to have a backup including the comments. I don't consider it a privacy violation if a user wants to backup their limited posts with comments.

MO

mozilla@blausand.net
Agree
Sun 22 May 2016 7:08PM

Comments are statements in the privacy realm of the original poster and thus outside personal control. If you want to comment in a personal way, use the "post" technology and link to your post.
I really expect everybody to /think/ before submitting.

F

Flaburgan Wed 11 May 2016 8:08AM

To comment on a limited post means you trust your friend and his podmin. With this proposition, a user can change his pod, and you will be forced to trust the new podmin for the old data you posted. On another hand, you haven't access to your friend contacts anyway so you don't know with which pod the comment is shared initially, and the whole model of diaspora* is based on the assumption that you trust your friend, and your friend trust his podmin, so the situation seems not worse here than before. The only case left is how the user will store the archive. I'm wondering what Facebook does? Does it include friends comments in the export?

G

Globulle Wed 11 May 2016 8:19AM

I'm wondering what Facebook does? Does it include friends comments in the export?

It may have changed since I did it (more than 1 year ago) but it didn't saved any comment (either your comments on other's messages nor friends comments on your messages).
Note that Facebook export was not intended to import the profile elsewhere later.

C

CSammy Sat 21 May 2016 1:09PM

Restoration of comments is a re-publication. Publication of another user's content and/or publication in the name of another user, both being the case when restoring comments, needs the consent of the user in question. Comments of users not consenting to this may be replaced by a content-less placeholder indicating that there was a comment, but it is no longer available.

B

Bady Sat 21 May 2016 9:48PM

one suggestion, for including others' comments in the backup without any privacy violation, is to add the following option to Diaspora:

  • when we want to publish a new post on diaspora there's an option to limit its visibility. in the same way introduce a similar option while commenting to allow the user to declare the comment as "Public" (which says that the OP is free to download/export/restore this comment) or "Private" (which says that this comment should not be downloaded/exported/restored).

the above option only works for the newer comments after such a feature is introduced. to deal with the existing comments, i think one should be allowed to backup others' comments but should not be allowed to restore the same without the consent of everyone included.

CS

Poll Created Mon 23 May 2016 5:16PM

For other peoples comments include guid and author's pod, but not the text and diaspora ID Closed Tue 24 May 2016 11:34AM

Outcome
by Comrade Senya Tue 25 Apr 2017 5:44AM

I was wrong opening this proposal, because the previous one actually had passed. If you feel disagreement with the decision, please say it in the discussion.

Since people might not be happy about including their comments in the export archive, let's not do it! Lets include the guids of the comments and the pod where the author lives. With this information it will be possible to fetch the comments from the other pod if such thing is allowed. Thus the decision whether it is allowed or not to have the comments out is decided by the author's pod and is out of the backup/restore feature scope. For example, fetch of the public comments is allowed without any conditions, and of private comments is allowed only if you are on the pod which has imported the archive of the previous user and can prove it. Moreover, having the guids may be used to render stubs in the discussions like "there are 4 comments in this discussion that are not available to you". This will make a difference between the case where there are no comments and the comments are missing for some reason (weren't fetched or were retracted).

Results

Results Option % of points Voters
Agree 37.5% 3 JR DU DD
Abstain 25.0% 2 EK A
Disagree 37.5% 3 E S CS
Block 0.0% 0  
Undecided 0% 259 BK ST FS MS TS AA S CB HF BO JH DM GC JH F RF M EG G AX

8 of 267 people have voted (2%)

DD

Dominic Duffin
Agree
Mon 23 May 2016 5:40PM

That would seem to be a good idea.

S

SuperTux88
Disagree
Mon 23 May 2016 10:08PM

That would create really broken conversations after someone migrated to another pod under his own posts. I would rather create a new account and keep the old one, so I can still read the old posts and don't break everything ...

E

Elm
Disagree
Tue 24 May 2016 6:15AM

Seems a good idea but it may be add complexity to the network and it may not be worth it and I agree with Super Tux88.

EK

Emmanouel Kapernaros
Abstain
Tue 24 May 2016 9:23AM

I don't think its a good idead to accept the risk of breaking comments when there is a better way. Podmins right now have the power to harm their users by having their data and their keys. This proposal is theoritically an illusion of being safer.

EK

Emmanouel Kapernaros
Abstain
Tue 24 May 2016 9:25AM

I don't think its a good idea to accept the risk of breaking comments when there is a better way. Podmins right now have the power to harm their users by having their data and their keys. This proposal is theoritically an illusion of being safer.

A

aj
Abstain
Tue 24 May 2016 10:11AM

a good suggestion that addresses the privacy and size issues for the archive, but the problems of fully recreating/restoring/cloning an account under federation* still exist so Where is the use case where the proposed backup would actually get used?

JR

Jason Robinson
Agree
Tue 24 May 2016 10:19AM

This sounds like a good compromise. Also still think it is not a good idea to include content into the migration archive. This just makes the whole process more complicated than it needs to be and provides little on top of the most important task.

S

SuperTux88 Mon 23 May 2016 10:50PM

The migration should be an easy way to move to a new home. It should provide the same experience (as much as possible) on the new pod, as on the old one (hopefully better performance/stability/uptime). But if I break all my own posts (and conversations in comments on my posts), it is really "useless".

Also fetching is not a solution:
1. I would create really high traffic if you fetch every comment from one of your posts.
2. It is not possible now to fetch private stuff. (and it is also not possible to fetch public posts know, but this could be added easily)

Why do we need to prevent someone to export data, that he is allowed to see? Everywhere else we handle it the way that the post author is the "owner" of the comments (and likes and everything else attached to the post). He decides who can see this stuff. Why handle this different at export? Everyone can take a screenshot of the comments or copy/paste them. So why should we maim a great feature
only to promise privacy, which is not there? If there is an API someday, it would be really easy to create a backup of all comments, but also now I can create a script that backups all comments (and if my new pod is my own server, then I can even import everything in the database).

This whole stuff only makes things worse and harder for those who simply want to move to a new home and don't want anything bad. But it prevents nothing that someone who wants to misuse it can do bad stuff with data of others. There are still many possibilities to do bad stuff, and we can't do anything against it. It's the same as with the whole copy protection and DRM, the good ones get the disadvantages, but it can't do anything against the bad people.

If someone really has a problem that the post author (and everybody else who can read the post/comment) can do "whatever he wants" with the comment, he should rather not post stuff on the internet!

CS

Comrade Senya Mon 23 May 2016 11:00PM

I mostly agree with you, but there is a little difference between copy-paste (or downloading with API) and exporting, since in the archive there must be the comment's author's signature, and by having a signature you can prove to a 3rd party that the person actually said these very words. And you can't prove the authenticity of words with screenshot or copy-paste.

S

SuperTux88 Mon 23 May 2016 11:17PM

If I have my own pod, I can also prove that the person actually said this (export signature from the DB), or if you enable fetching for comments, it also contains the signature. You can not prevent this without removing the signature (which would be a really bad idea :wink: )

But I think this is an advantage, because you can only export/import the exact same words that someone wrote (except that a podmin can always manipulate stuff on his own pod, but this has nothing to do with this topic).

Again, if someone has a problem with that and can't stand to his words, he should rather not post stuff on the internet ...

On a distributed network, where you send everything to other servers, you never ever have full control on that data again, and we should not promise that here. There are multiple other ways to export and prove stuff, which are by design.

CS

Comrade Senya Tue 24 May 2016 12:28AM

Okay, I wasn't right about opening a new proposal, since "A proposal is passed if there are more Yes votes than No and Block votes together". But it would be nice to discuss another option anyway.

E

Elm Tue 24 May 2016 6:23AM

Hi, including the comments are all right for me as there is many other ways to save publication and its comments. However the questions I'd like now to be discuss here are the following (but maybe I do not undersand how things works and these question will be irrelevant ?):

  • If I have commented on a post, and the owner of the post migrates, will I be able to delete manually (or by closing account) my comments on the same publication after its migration ?

  • it raise another question to me : will I be able to find the post easily ? (say, if I have bookmarked it, will the adress be still connected to the new place of the publication (with a redirect maybe ?) Will the GUID of the publication be changed (if I had bookmarked the adresse with the GUID instead of the local number) ?

CS

Comrade Senya Tue 24 May 2016 11:06AM

If I have commented on a post, and the owner of the post migrates, will I be able to delete manually (or by closing account) my comments on the same publication after its migration ?

As I said there comments deletion is not very precise wording for the federated web. You can't be sure the foreign pod is not modified to ignore retraction messages. It's better to say comments retractions. Considering that, yes, retractions will be routed to the new pod.

it raise another question to me : will I be able to find the post easily ? (say, if I have bookmarked it, will the adress be still connected to the new place of the publication (with a redirect maybe ?) Will the GUID of the publication be changed (if I had bookmarked the adresse with the GUID instead of the local number) ?

There is no need to change neither GUID nor local id of posts in the migration process. It just updates owner reference.

This just makes the whole process more complicated than it needs to be and provides little on top of the most important task.

Well, I guess breaking conversations is not what people expect from the migration process. You may say it's not important, but I'm sure it would result in issues reported like "I migrated and lost all comments to my posts! How can I unmigrate back, please?"

JR

Jason Robinson Tue 24 May 2016 11:11AM

You may say it's not important, but I'm sure it would result in issues reported like "I migrated and lost all comments to my posts! How can I unmigrate back, please?"

Better than them currently saying "I lost my account due to the pod I'm on disappearing, how do I restore my identity?".

Start off small, expand later...

G

goob Tue 24 May 2016 6:52PM

Would it be technically feasible, when someone migrates to a new pod, to send a message to people who have participated in conversations with them asking whether they are happy for their comments to be stored on this new pod?

It could work something like this:

  • Alice, Bob and Cath have participated in a private conversation.
  • Alice migrates from Pod X to Pod Y.
  • Once migration is complete, the data is deleted from Pod X.
  • Bob and Cath receive notifications 'Alice has moved to Pod X. Are you happy for your private interactions with Alice to be stored on Pod X?'
  • Bob is happy for this to happen so clicks OK. His comments in the private conversation are then sent from his home pod to Pod Y.
  • Cath is not happy as she has heard bad things about security on Pod Y. She clicks no. Her comments are not transferred to Pod X.

This means that in this case some comments are not available to Alice on her new pod, but I'd imagine that this case where someone denies their comments to the new pod will be rare as it would probably only happen if there was a pod with a bad reputation. So this would, in most cases at least, enable complete conversations to be hosted and available on Alice's new pod while respecting the privacy and agency of the other participants in those conversations.

Would that a) solve the concerns about privacy and b) be technically feasible?

S

SuperTux88 Tue 24 May 2016 11:44PM

I think we need to make clear for what we want to use this archive?

  • Local-Copy of all my data: then we can leave many meta-data and data of other users out of the archive
  • Backup of an account to restore on another server: then we need to include everything needed to restore the complete account on a new server. So that the user can continue on the new pod as he stopped on the old one.

@jasonrobinson

Also still think it is not a good idea to include content into the migration archive.
and
Better than them currently saying "I lost my account due to the pod I'm on disappearing, how do I restore my identity?".

How would you restore the account out of an archive without content if the pod doesn't exist anymore?

@goob

Would it be technically feasible, when someone migrates to a new pod, to send a message to people who have participated in conversations with them asking whether they are happy for their comments to be stored on this new pod?

  • where would you store the comments before they are stored on the new pod? should the user import the archive a second time, after the users all accepted that the comments can be stored?
  • the discussion was about comments on posts, that could be hundreds of users that need to be asked?
  • what should happen for users that are not active anymore and can not answer your question?
  • how should a person answer, if their pod doesn't exist anymore?

There is so much that doesn't work properly with your "solution", in my opinion this creates more problems than it solves. We need a simple solution that everybody can simply move to a new pod, without extra barriers.

If I have my own pod, I can move it (and with it my profile and all posts and comments from all users) to wherever I want and I don't need to ask everybody. But if I don't have my own pod (but want to move to my own pod), then I need to ask everybody?

Also: If you write a comment on another users post, he don't ask you, if he can federate your comment to all other users that can see that post.

I think we need to stop the illusion, that you still have full control over everything after your data left your pod. When other users can see your posts/comments, then they can do whatever they want with it (does not mean that they do so, but the could). And yes, bad guys can do bad things (publish your private posts/comments), but I think that is no reason to create a broken unusable migration-feature for all good guys.

E

Elm Wed 25 May 2016 10:32AM

Another way could be (but that's just an idea and I do not know if it is a good one…) : add a binary info to each comments depending on an option in the setup page of the person who comments, saying «I agree to migrate my comments on a new pod if a contact migrate to this new pod». When doing the migration only post with the flag should migrate the others being retracted and replaced by "comment missing here". Would this be technically feasable and do not add too much complexity ?

JR

Jason Robinson Wed 25 May 2016 10:43AM

@supertux88

How would you restore the account out of an archive without content if the pod doesn't exist anymore?

It's all part of the spec that I wrote that Senya based his crowd funding on :) I have no knowledge what the current situation is, whether the mentioned automatic backup is even planned to be implemented, but it's totally doable.

Btw, just saw two more pods announce going offline with a very short warning notice. The whole migration feature will not help at all if there is no automated identity backup. Only 1% of people will regularly download their archive just to backup a service they are using. I know I never do. Why? Because service providers do it for me. Diaspora can be thought as a big service provider composed of many nodes.

But this discussion has been taken already, I'd rather not start down that road again.. :)

G

goob Wed 25 May 2016 11:16AM

  • where would you store the comments before they are stored on the new pod? should the user import the archive a second time, after the users all accepted that the comments can be stored?

As I said, 'Bob is happy for this to happen so clicks OK. His comments in the private conversation are then sent from his home pod to Pod Y.' His comments are stored on his home pod, and sent direct from there.

  • the discussion was about comments on posts, that could be hundreds of users that need to be asked?

Apologies; I'm sure I had read that this related specifically to conversations.

Also: If you write a comment on another users post, he don't ask you, if he can federate your comment to all other users that can see that post.

That's a choice that I make when I comment on a post; I implicitly authorise it to be shared with those with whom it has been shared. However, it shouldn't subsequently be shared with further people, which is why it is not possible to retrospectively make a limited post public.

All I have done is shared an idea that might help enable migration of remote users' comments while meeting the concerns of those who voted against the proposal. This procedure I suggested with comments is, as I understand it, the same as what would happen if Bob and Cath were in Alice's contact list (a notification asking for confirmation that they are happy for their user details to be hosted on Alice's new pod). If that is not how the migration feature is being implemented, then I would be less happy about it being implemented.

G

goob Wed 25 May 2016 11:19AM

add a binary info to each comments depending on an option in the setup page of the person who comments, saying «I agree to migrate my comments on a new pod if a contact migrate to this new pod».

This wouldn't work from my point of view. I would, for instance, be happy for my comments to be migrated to a pod run by someone I knew, but less happy if Alice was moving to diaspora.dodgy-identity-thieves.com

It's the choice of whether to allow your comments to be moved to a particular pod which is at issue.

E

Elm Wed 25 May 2016 11:29AM

It's the choice of whether to allow your comments to be moved to a particular pod which is at issue.

Not necessarily. If you do not trust you would select that you do not agree to migrate you're comments to another pod (be it trustable or not), if you do not care you would agree.

S

SuperTux88 Wed 25 May 2016 12:46PM

@jasonrobinson

It's all part of the spec that I wrote that Senya based his crowd funding on :) I have no knowledge what the current situation is, whether the mentioned automatic backup is even planned to be implemented, but it's totally doable.

But this automatic backup should contain the same data as the manual backup, so it also needs the comments (and other additional metadata).

@goob

As I said, 'Bob is happy for this to happen so clicks OK. His comments in the private conversation are then sent from his home pod to Pod Y.' His comments are stored on his home pod, and sent direct from there.

And if the pod of this person doesn't exist anymore (because maybe the person was on the same pod as you, when your pod disappeared?) , it is not possible to restore the data?

That's a choice that I make when I comment on a post; I implicitly authorise it to be shared with those with whom it has been shared.

But you don't know with whom the post has been shared? Maybe your comment is already on diaspora.dodgy-identity-thieves.com because the post author shared it with a person on this pod. You don't know it, you don't see it, you can't control it. So you are trying to control something, where you don't even have control when you write your comment.

If you write a comment on a post from alice, you trust alice, that alice shared the post (and your comment) only to trustworthy pods (you can't control it). So why don't you trust alice when she wants to move to another pod?

@all

why do you try to control something where there is no control? This only makes this feature less usable, but doesn't stop anybody to do bad things if he wants to. If you write a comment on a Post on Alice:
* You trust alice, that she has shared the post only to trustworthy persons
* You trust alice, that she don't leak your comment to someone else (now and in the future and in the backup)
* You trust every other person (the person with whom alice shared the post), that they do the same ...

You don't have control on anything when you write the comment (you need to trust alice) ... why do you try to imagine to control what alice could backup? If alice has her own pod, you also don't have control on how alice backups here database and where she stores the backup and what and how she restores the database-backup on another server?

Everything you try to imagine control where there is no control only makes this cool and useful feature:
* more complex
* more error prone
* takes longer to implement
* less useful for the users

JR

Jason Robinson Wed 25 May 2016 12:49PM

@supertux88

But this automatic backup should contain the same data as the manual backup, so it also needs the comments (and other additional metadata).

No it doesn't, at least not in what I wrote :) I never wanted content included in the automated backup archives - that would make the whole system too heavy.

But as said, that ship has sailed, for the current plans I don't know.

CS

Comrade Senya Fri 27 May 2016 6:46PM

Ok, the next important point to raise. What do we do to the polls on export/import/migration?

Just ignoring a poll is not an option since this will result to the broken UX. After migration, user would see his posts where previously were polls without them or with messed voting.

First option is to include everything to the export archive required for polls to keep working after archive import: polls themselves, poll answeres, poll participations. But there is a nuance: I don't know if it is a design decision or just random, but now it's impossible to check who voted for which answer. If we export the archive there will be each vote in the archive with the ID of the voter. So it's gonna be less private this way. On the other hand, an owner of an own pod is able to see this info anyway, so it's not generally protecting privacy.

Pros: polls are alive after users migration
Cons: much metadata must be included to the archive resulting in possible bloat

Second approach is to lock votings when user migrates. Correct me if I'm wrong, currently it's impossible to close a poll for new votes. If we introduce such a feature, we can include only number of voters to the archive and close voting on migration.

Pros: votings still looks nice; archives are lighter
Cons: it's impossible to participate in old polls anymore after a migration

Please share your opinions before I start any proposal.

E

Elm Fri 27 May 2016 7:35PM

Hi, migrations will be useful but rare events. So I'd vote for the simple and lighter second approach.

S

SuperTux88 Sat 28 May 2016 3:55AM

I think it's better to include everything to the archive.

Cons: much metadata must be included to the archive resulting in possible bloat

I don't think that that is a problem, because only a few posts contain polls, and also polls are not really big.

CS

Comrade Senya Sat 28 May 2016 9:57AM

Biggest thing is not polls themselves, but poll participations, each one must include GUID and author_signature.

One poll participation is ~300 bytes, for 1000 participants it'll be ~300kb and compressed notably lighter.

S

SuperTux88 Sat 28 May 2016 8:07PM

but we don't have polls with 1000 participants ;) And we have also signatures in comments and likes, and compared to polls, the polls are smaller ;)

So if I'm an active user with many posts (long history ;) ) with many participations (comments, likes and polls), then the archive could be some MB, but this should not be a problem. If I have uploaded some photos, I think the photo-archive is bigger.

CS

Poll Created Mon 30 May 2016 3:12PM

Include polls & poll participations to users archive? Closed Mon 6 Jun 2016 1:02AM

See the comment and further discussion.

Results

Results Option % of points Voters
Agree 100.0% 9 F G DM EK S FL B CS DD
Abstain 0.0% 0  
Disagree 0.0% 0  
Block 0.0% 0  
Undecided 0% 258 BK ST FS MS TS AA S CB HF BO JH DM GC JH JR RF M EG G AX

9 of 267 people have voted (3%)

G

goob
Agree
Mon 30 May 2016 4:50PM

I vote for option 1, but also vote for closing down what's federated in the polls feature code, as comment above. This should enable polls to be active for a user after migrating, while keeping exports lighter.

F

Flaburgan
Agree
Tue 31 May 2016 8:53PM

I'm I'm favor of including everything needed to reproduce the exact same behavior on the new pod than the user had on the previous pod.

G

goob Mon 30 May 2016 4:49PM

But there is a nuance: I don't know if it is a design decision or just random, but now it's impossible to check who voted for which answer. If we export the archive there will be each vote in the archive with the ID of the voter.

I'm surprised to hear that this info is federated by the polls feature. I don't think there's any need for anyone except the poll creator to be able to see who voted for which option. I would vote for closing this down so that only poll question, poll answers and number of votes for each answer were federated. But this should be shut down within the polls feature code rather than in the export code. That way exports will be lighter!

S

SuperTux88 Tue 31 May 2016 12:59AM

@goob: the author of the poll-participation is federated to validate the signature and also to prevent duplicate votes by the same person. And if only the poll-creator sees who voted and sends only numbers to everybody else, he would be able to fake the result. Maybe we should simply make the poll-participants public visible to everybody (like votes here on loomio). But this needs its own discussion and doesn't belong to the export. :)

Also the proposal here is only about including the poll and answers to the poll-creators export. The poll (and answers) is not included to other persons export.

G

goob Tue 31 May 2016 11:15AM

@supertux88, thanks for your reply. I couldn't (and still can't) find any indication that this vote relates only to migration by a poll creator, but if so, I think it is important to migrate all data necessary for a poll to work, so that a poll (if still open) can still be work.

It would be good to refine the polls feature with things such as the ability to close polls (and to create them to be open for a limited time only), and to look at ways to reduce the federation of user data when interacting with a poll, if this is appropriate an useful. All this would make the migration of an account including polls more streamlined, but this work doesn't belong to the migration feature. These refinements would be best added to, and discussed within, the Enhancements for polls feature issue in Github.

Thanks again.

U

unity100 Fri 26 Aug 2016 6:39PM

Does backup and restore concept include Pod Migration?