Loomio
Thu 26 Feb 2015 12:11PM

Back end design - how it does it

H Harper Public Seen by 124

This is for discussing the actual development. A place to ask for help when code doesn't work? This is not my (Adam's) area, so I don't know quite what to put here.

@fiatjaf (Giovanni) said (in Design thread):
I am working on a prototype here that treats IOUs as first-class citizens. In the way I am modelling the data, each user should have a dashboard with not various balances, but with various independent IOUs, issued at different dates and each one capable of yielding interest at different rates. (I don’t know if this is scalable, but I like it and if it is not scalable we can think about it later).

With my approach, it is easy, for example, to

a. Implement IOU markets, of any way you like.
b. Manage, mix and merge different IOUs with different interest rates (for example, you can lend a small quantity of money to a friend for 0 interest, but then later when he asks you for a bigger amount, you decide that you will lend only at a bigger interest rate).
c. Change credit limits granted to other users.
d. Visualize paths, tell the user to whom he is linked and how much he can pay (ok, not pay, but create a chain of IOUs) to each person at each time, let him choose paths based on the total interest rate of the path, pay through different paths.
e. Discard IOUs.
f. Revoke IOUs.

Nothing said here is impossible to do in Ripplepay, I believe (in fact some of these are actually done there), but anyway they easy to do in my prototype.

However, I am still struggling to find the proper way of doing currency conversion and proper IOU cancellation and I would like some light on these topics.

Ok, let me clarify. Imagine we have the following IOU:

{
      "bearer": "ana", 
      "issued_at": "2014-05-02T00:00:00", 
      "issuer": "geraldo", 
      "rate": 10.0, 
      "value": 300
}

Then “geraldo” enters a payment of 200 to “ana” (or to someone else, but with the payment path passing through “ana”). What should happen?

This IOU should stay? Probably not. But should it be deleted and replaced with a newer IOU, with value of 100 (300 - 200)? We should create the two IOUs, one from “ana” to “geraldo” and one from “geraldo” to “ana”, then later merge them and generate a new one?

What if we have the following (like explained in the comment above):

{
      "bearer": "ana", 
      "issued_at": "2014-05-02T00:00:00", 
      "issuer": "geraldo", 
      "rate": 10.0, 
      "value": 300
}
{
      "bearer": "ana", 
      "issued_at": "2014-05-03T00:00:00", 
      "issuer": "geraldo", 
      "rate": 0, 
      "value": 5
}

Then, when “geraldo” enters a payment of 20 to “ana”, from which IOU we should subtract these 20 (note that the two IOUs have different interest rates)?

H

Harper Thu 26 Feb 2015 12:44PM

What is "rate"?
Also, why is there a date?

In my vision, there is no such thing as "an IOU". Weird huh?
There are users.
Each user has any number of IOU balances.
Each IOU balance is either a credit balance (for holding IOUs) or a debt balance (for issuing IOUs). A credit balance cannot become a debt balance, and a debt balance cannot become a credit balance.

An example debt balance contains the following data:
three letter currency: GBP
upper limit: 2000
lower limit: 600
compound interest rate: 4%
creditors need your permission to hold these IOUs: FALSE
terms: vague, I mean, whatever dude, it'll probably be ok.
short name: car loan

An example credit balance contains the following data:
three letter currency: GBP
issuer: ana
buy price: 0.982
upper limit: 150
sell price: 0.994
lower limit: 140
amount: 73.56
database of entire balance history (all transactions with times and dates) for purposes of calculating compound interest.
short name: savings

AS

Alessio Stalla Thu 26 Feb 2015 1:07PM

My thoughts about the topics discussed, in an inordinate fashion.

I prefer to talk of nodes rather than users; it's more general. A node might represent an external entity rather than a human user.

About IOUs, I have a mindset similar to Adam's - in thinking about Ripple I've always thought of IOUs as "connections" between objects, rather than objects in themselves. It's the same thing really, but with a different perspective.

I agree that debt and payment/transfer are complementary. If A pays B $10 cash, A is transferring credit in the form of a $10 bill. That $10 bill is an IOU from "the system", i.e. everyone participating in the economy. "The system" has basically infinite trust (as long as it works).

As I see it, in the context of Ripple, a payment is not a transfer of a IOU, rather it is a transfer of value through IOUs. IOUs might be emitted, extinguished, and altered in the flow of value from one node to the other. The concept of "holding" a IOU seems strange to me, but it just means that you configure the system not to use a certain amount of that IOU for value transfers. I can see that as a useful feature.
Similarly, Adam's "market" for IOUs in my view is just a configuration of the network where navigating edges has a non-zero cost. Again a useful feature, even if the algorithmic/complexity impact scares me, as you already know.
Same thing with interest. Interest = the network changes automatically over time even if no transfers are made. It is simply a form of automation in my eyes.

Of course to account for all the complications above there must be the possibility for two nodes to have multiple connections with different configurations. E.g. one with 1% interest, another one with 0. One with buy price 0.9, another with buy price 1. And so on. Ideally, for transactions the system should choose the best path - however the implementation complexity of that must be evaluated.

Currency conversion - I don't think the graph should deal with this. It's an unnecessary complication, another of the things I don't like about ripple.com. I think a single virtual currency should be used internally, and only displayed differently to different users according to their preferences and current conversion rates.

Having a prototype is incredibly valuable. The ability to experiment is gold. However, I would refrain from talking about actual code until we have a solid design.

That's all for now. I hope some of what I wrote makes sense :)

MGG

Mario G. Gurgel Thu 26 Feb 2015 2:59PM

In my example, "rate" is the interest rate. The "issued_at" date is only the date in which that IOU specifically was emitted.

I don't see what is unclear on my examples. I think the concept of IOU may be getting in our way, but I've already gave up that idea I was advocating in the email thread.

Let's try to clarify things here. In Adam's explanation:

In wuush, there is no such thing as a negative balance. IOUs held and IOUs issued are displayed separately.

and

In my vision, there is no such thing as “an IOU”. Weird huh?
There are users.
Each user has any number of IOU balances.
Each IOU balance is either a credit balance (for holding IOUs) or a debt balance (for issuing IOUs). A credit balance cannot become a debt balance, and a debt balance cannot become a credit balance.

So, what do you mean by "each user has any number of IOU balances"? Each balance must be between two users, right? An "IOU balance" is only the sum of all loans and repayments made between two users, with a specific interest rate. Right?

The two same users can have multiple IOU balances between then? Ot just two (one in each direction)?

Does it make sense to have two IOU balances between "ana" and "geraldo" if in one of them "ana" owes $10 at an interest rate of 1% and in the other "geraldo" owes $10 at an interest rate of 2%? Shouldn't these two cancel themselves (and stop accumulation of compound interest?).

MGG

Mario G. Gurgel Thu 26 Feb 2015 3:02PM

Alessio, I don't understand what you mean by

The concept of “holding” a IOU seems strange to me, but it just means that you configure the system not to use a certain amount of that IOU for value transfers.

I think there is a problem with the concept of an IOU. So I will ask, in the most general terms possible, what is an IOU?

H

Harper Thu 26 Feb 2015 3:09PM

@alessiostalla:

I prefer to talk of nodes rather than users; it’s more general. A node might represent an external entity rather than a human user.

Yes, when I say a "user" I really mean a node. For most intents and purposes I think of them as human users because it helps me imagine real scenarios, but it is nodes that I'm talking about, not users.

The concept of “holding” a IOU seems strange to me

I use the word "holding" to mean that a node has some of the specified IOU. Holding, as in, in your hand. When a node holds IOUs, it doesn't mean that they can't be spent in a payment or a through payment, it just means that they have some of those IOUs, their balance for that set of IOUs is higher than 0.

The ability to set lower limits, preventing IOUs from being spent (either by that node directly, or with a through payment) is unrelated.

Similarly, Adam’s “market” for IOUs in my view is just a configuration of the network where navigating edges has a non-zero cost. Again a useful feature, even if the algorithmic/complexity impact scares me, as you already know.

Not sure I quite understand everything you said there.
There is no "market" as such (in my wuush). The balance data listed in my previous post is the market. Perhaps that's what you meant.

Interest = the network changes automatically over time even if no transfers are made. It is simply a form of automation in my eyes.

I really think we shouldn't try to implement interest at this stage; I only included it in the list for the sake of completeness. Focus on the bold things.

Ideally, for transactions the system should choose the best path - however the implementation complexity of that must be evaluated.

I could probably write a really inefficient (but functional) example algorithm for that in python if really necessary. It would take a lot out of me, and my energy is very limited, so this should be avoided if at all possible. It might be easier with some help.

Currency conversion - I don’t think the graph should deal with this. It’s an unnecessary complication, another of the things I don’t like about ripple.com.

I disagree, completely. Currency conversion doesn't add too much complication to pathfinding (to me, none of it is complicated, it's all really simple and obvious), and the pathfinding function I would write could include this.

I think a single virtual currency should be used internally, and only displayed differently to different users according to their preferences and current conversion rates.

That would add complexity (to my vision of wuush) and be unnecessary. That's XRP all over again. I see the problem you are trying to solve, and that is one way to solve it, but there's a better way. I just haven't been very good at describing it or explaining why/how it's better/simpler.

I don't think we should be trying to implement currency conversion at this stage. The reason being, there's a lot of functionality we can make, that doesn't depend on currency conversion, and currency conversion won't be possible till we've done almost everything else. I've imagined the stages as something like this:
1. all transactions in the current reserve currency (USD).
2. allow nodes to issue and accept IOUs in any three letter currency they wish, but not do any conversion between them, so that each currency has its own completely separate network of debts that don't interact with each other.
3. add currency conversion.
4. compound interest
5. other things.

A node's database will contain:
debt balances
credit balances
any number of currency conversion orders

An example currency conversion order contains:
buy/sell: buy
base currency: USD
counter currency: GBP
price: 0.64 (GBP)

Having a prototype is incredibly valuable. The ability to experiment is gold. However, I would refrain from talking about actual code until we have a solid design.

Very wise. We haven't yet reached a consensus on what wuush is. That said, Giovanni's experimentation has led to discussion which means we are all a bit closer to understanding each other.

AS

Alessio Stalla Thu 26 Feb 2015 3:14PM

In my understanding, a IOU is a specific piece of a credit relationship between two subjects; in the context of that IOU, the Debtor and the Creditor.

In the context of Ripple/Wuush, I've always thought of IOUs as an internal implementation detail that wasn't directly visible to users. E.g. a Node4j edge, with properties, from a node to another. So the idea of people "holding" IOUs reifies them - it gives them a concrete feeling they didn't have in my mind before, that's why it seemed strange to me. I see nothing wrong with that, it's just a different way of framing the same concept.

The two same users can have multiple IOU balances between then? Ot just two (one in each direction)?

Multiple. Because they may have non-mergeable characteristics, such as different interest rates, or different prices.

Does it make sense to have two IOU balances between “ana” and “geraldo” if in one of them “ana” owes $10 at an interest rate of 1% and in the other “geraldo” owes $10 at an interest rate of 2%? Shouldn’t these two cancel themselves (and stop accumulation of compound interest?).

I think they should cancel themselves, yes. The system should always prefer cancelling out existing debts to creating new ones.

H

Harper Thu 26 Feb 2015 4:11PM

@fiatjaf (Giovanni):

“rate” is the interest rate

what you mean by "interest" and what I mean by "interest" are very different things. When I say 4% interest, I mean continuous compound interest at a rate of 4% per year. When you say interest at a rate of 10, what does that mean? Can you tell me what 10 of your interest is (approximately) in terms of % per year continuous compound interest?

The “issued_at” date is only the date in which that IOU specifically was emitted.

It sounds like what you are calling an IOU is (in terms of Adam's wuush) a single payment transaction between one node and another node, with no intermediaries. Is this correct?

If so, you're just going to end up with more and more of these IOUs and they will never be destroyed. That would explain why you're having trouble figuring out how they should cancel out. In Adam's wuush, IOUs are not actually issued. There are just balances that get modified, like in Ripplepay. IOUs don't need to be cancelled out because the same balances both send and recieve payments and adjust their amounts accordingly.

Each balance must be between two users, right?

Nope (at least, that's not a way I find helpful to think about it). Each balance is contained withn a node (usually associated with a user).
unlike Ripplepay and Ripple, credit balances and debt balances are not at all symmetrical. They behave very differently.

A debt balance doesn't contain the amount of the balance. The amount the owner of the node sees is the sum of all the amounts from all the corresponding credit balances in other nodes.

So, if you have 30 of Jane's USD IOUs, that doesn't mean that Jane owes 30 USD IOUs. Jane's corresponding debt balance might be 80 USD. See, it's not the same balance.

But, you could look at it in another way and say that Jane has an invisible balance with you, to the value of 30 USD, in which case yes, you're quite right, each balance is between two nodes. from this point of view, the amount of a debt balance is composed of the amounts of all the balances with each user that accepts your IOUs.

The two same users can have multiple IOU balances between then? Ot just two (one in each direction)?

any number. And it doesn't have to be mutual.
To replicate Ripplepay functionality, there would have to be a balance in each direction, even if the limits were set to zero and were not used.

Does it make sense to have two IOU balances between “ana” and “geraldo” if in one of them “ana” owes $10 at an interest rate of 1% and in the other “geraldo” owes $10 at an interest rate of 2%? Shouldn’t these two cancel themselves (and stop accumulation of compound interest?).

YES! I mean, no it doesn't make sense for two nodes to owe each other. First, remember that a "balance" is not the same as a debt. A balance can have an amount (a value) of zero. It doesn't go away just because it's empty. It stays there because it contains other information like prices and limits. Second, how would geraldo get into the state of owing $10 to ana, when ana owes geraldo $10? when geraldo attempted to pay ana $10, the pathfinding algorithm would find that (assuming there are no cheaper paths involving intermediaries) the cheapest way for geraldo to pay ana would be to redeem the $10 that she owes him.

the interest rate is irrelevant to this example.

Setting a hard upper limit of "1" on prices and having the pathfinding algorithm prefer redemption over issuance (when dealing with equal prices) means that existing debts get cancelled first. If geraldo was trying to pay ana $15 (and ana accepts geraldo's IOUs) then geraldo's node would spend the $10 IOUs from ana before issuing his own IOUs to ana.

Wheew, this is getting long.

AS

Alessio Stalla Thu 26 Feb 2015 4:59PM

I think we're all making confusion :P

An IOU is definitely not a payment; it might get issued or extinguished or updated due to a transfer.

The "rate" Giovanni mentions might well be the interest rate in continuous compound interest. Also, an issue date is necessary in order to actually calculate interest, since you need to know how much time has passed since the IOU was issued. So an IOU will have initial amount A, issue date D, interest rate R, and the system will be able to show its actual amount as of now to the user (which is A times e to the power of (now - D) in years times R). Every time that IOU is modified (due to a transfer), the initial amount must be altered by applying the interest formula and the payment, and the issue date must be set to now. That's how I understand it.

"Balance" as I see it is an abstract concept which might be the immediate result of a calculation or a datum physically stored somewhere (e.g. on a node, or in a cache) and periodically updated in order to save computing power. That's an implementation detail.

So, if you have 30 of Jane’s USD IOUs, that doesn’t mean that Jane owes 30 USD IOUs. Jane’s corresponding debt balance might be 80 USD. See, it’s not the same balance.

Just to be sure - sorry for being dense - that is true solely because of buy/sell prices that are not 1, right? Whereas if they're 1, having 30USD worth of Jane's IOUs does mean that Jane owes me 30USD as of now, right? Else, I fail to see why you would say that.

Later on Adam uses the same term "balance" to indicate something else, it seems to me. I.e. credit lines. In my view, there are nodes with a bunch of info (such as login data if they're users), and there are connections between them. Connections can be of one of two types:
- credit lines: "A is willing to give at most X credit to B at interest rate R, buy price P, sell price S"
- IOUs: "A owes B an amount of X since date D, at interest rate R, buy price P, sell price S"

Credit lines are like "IOU templates". When an IOU is issued, its parameters are copied from a credit line. Later, that credit line might be modified and the IOU would not be affected, and vice-versa.

Then my "balance" is simply the sum of all outgoing and incoming IOUs to my node.

Does that sound right to you?

MGG

Mario G. Gurgel Thu 26 Feb 2015 5:42PM

Exactly, @alessiostalla. Exactly.

MGG

Mario G. Gurgel Thu 26 Feb 2015 5:55PM

About this @harpereaves point:

If so, you’re just going to end up with more and more of these IOUs and they will never be destroyed. That would explain why you’re having trouble figuring out how they should cancel out. In Adam’s wuush, IOUs are not actually issued. There are just balances that get modified, like in Ripplepay. IOUs don’t need to be cancelled out because the same balances both send and recieve payments and adjust their amounts accordingly.

I'm thinking about IOUs the way @alessiostalla described. The data we have are a bunch of IOUs issued by someone and held by someone else. They should cancel themselves whenever people pay each other in the real life or send a new payment through the system in the opposite direction. How exactly to do this was my question.

Having an IOU generated for each payment, or something like that, will, in fact, fill the system with a lot of IOUs, but I imagine that it is necessary to give all transactions a materialized existence on the database (with its date and value), otherwise it will be impossible to calculate interest.

I also imagine that this is the same as having a balance with references a lot of different payments, but maybe it makes more sense to model data in terms of actual loans (IOUs) and possibly repayments.

AS

Alessio Stalla Thu 26 Feb 2015 10:12PM

Ok, I've read the discussion more carefully this time. I have the feeling we're circling around fake issues that arise from mixing design and implementation details.

Tell me if I got it wrong.

Adam basically thinks in terms of relational database tables. There is a Node concept, a Debt Balance concept, a Credit Balance concept, a Transaction concept.
1. A Node has 0..n balances (potentially both debt and credit).
2. A Debt Balance specifies interest and limits. A Debt Balance has a reference to a holder Node and to multiple Transactions.
3. A Credit Balance specifies prices and limits, it has an amount and a Transaction history used to calculate interest (how exactly?). Transactions also refer to a Debt Balance (this is where interest comes from).
4. A Credit Balance has a reference to a holder Node (per point 1. above) and a reference to a issuer Node. As far as I understand it, the issuer is only useful when no Transactions have happened yet, in order to store the parameters of the relationship between the two Nodes.
In a relational database, travelling long chains of relationships is not idiomatic, it is costly and awkward. So important information must be kept "close" to a node. IOUs then are an implicit, emergent concept, while balances are core (because they're what you show to the user).

Alessio and Giovanni think in terms of a graph. So there are Nodes and edges (connections) between Nodes: Credit Line edges and IOU edges. Each edge has the properties: interest rate, sell price, buy price (as well as properties useful for display purposes, such as a description, that aren't considered by any algorithm).
Credit Lines also have a lower and upper limit.
IOUs also have an amount and an issue date.
Credit Line edges are just templates for IOUs. When an X amount of value is transferred from A to B, two things can happen:
1. a new IOU from A to B is created by copying all relevant parameters from a Credit Line from B to A, or
2. an existing IOU with the exact same parameters is updated.
In case 2., assuming an interest rate of R, the IOU changes from date D0 and amount A0 to date D1 = now and amount A1 = X + A0 * eR*(D1 - D0 as years). X can also be negative, however the IOU cannot become negative. If it would, then it is extinguished and another IOU with the remaining amount is issued from B to A if possible, else the payment is invalid and cannot happen.
Here I didn't consider buy and sell prices. If they're less than 1, then the amount of a payment is calculated accordingly, but all other calculations stay the same.
The graph is optimized towards typical traversals. So, credit lines and IOUs are core concepts, while balance is an implicit, emergent concept.

Did I get it somewhat right? Then, I think it's possible to reconcile the two models. But now I'm tired :P

H

Harper Mon 2 Mar 2015 2:32PM

@alessiostalla:

Ok, I’ve read the discussion more carefully this time. I have the feeling we’re circling around fake issues that arise from mixing design and implementation details.

Yup. My bad.

I had to look up "relational vs graph", and "graph edge". I know what you mean by "IOU" and "template" now, but these are all new concepts to me and I will need to think about them before I can comment further.

Thanks for your thorough analysis and explanation.

H

Harper Thu 5 Mar 2015 11:25AM

Why are credit line edges and IOU edges separated?

I favour

2 an existing IOU with the exact same parameters is updated.

Pathfinding
I imagine pathfinding going a like this:
instead of trying to find the cheapest path for a payment of X from A to B, it just looks for the cheapest path, full stop. Then it sees how much can be paid using that path. If the amount that can be paid through that path is insufficient, then it looks for the next cheapest path to use for the remainder, and so on, until either there are no paths at all or there is no remainder (in which case it offers the payment option (and the price) to the node making the payment).

The point here is that in the case of

X can also be negative, however the IOU cannot become negative. If it would, then it is extinguished and another IOU with the remaining amount is issued from B to A if possible, else the payment is invalid and cannot happen.

"another IOU with the remaining amount is issued from B to A if possible, else the payment is invalid and cannot happen." is wrong. If the path using As debt to B is insufficient then the next cheapest path must be found. It might be that a new IOU from B to A is the cheapest way, but it might not.

Here I didn’t consider buy and sell prices. If they’re less than 1, then the amount of a payment is calculated accordingly, but all other calculations stay the same.

Yes and no. buy and sell prices can't just be tacked on as an afterthought. When all prices are 1, any path will do, so the approach is focussed on finding a path. When prices may be less than 1, you have to find not just a path but the cheapest path.

There are many different ways of doing this. I've always imagined having pathfinding as a service that users pay for (or do themselves on their own computers) if they want to be thorough. At least we should (eventually) support alternative methods and depths of pathfinding.