sipa changed the topic of #bitcoin-wizards to: This channel is for discussing theoretical ideas with regard to cryptocurrencies, not about short-term Bitcoin development | http://bitcoin.ninja/ | This channel is logged. | For logs and more information, visit http://bitcoin.ninja
weez17 has quit [Remote host closed the connection]
weez17 has joined #bitcoin-wizards
shpx has joined #bitcoin-wizards
Murch has quit [Quit: Snoozing.]
shpx has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Samdney has quit [Quit: Verlassend]
rmwb has joined #bitcoin-wizards
shpx has joined #bitcoin-wizards
rmwb has quit [Ping timeout: 256 seconds]
Giszmo has quit [Ping timeout: 264 seconds]
dabura667 has joined #bitcoin-wizards
Murch has joined #bitcoin-wizards
CubicEarths has quit []
CubicEarths has joined #bitcoin-wizards
shpx has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
tromp has quit [Remote host closed the connection]
Aaronvan_ has joined #bitcoin-wizards
schmidty has joined #bitcoin-wizards
schmidty is now known as Guest51622
AaronvanW has quit [Ping timeout: 260 seconds]
Guest51622 has quit [Ping timeout: 264 seconds]
tromp has joined #bitcoin-wizards
Murch has quit [Quit: Snoozing.]
Aaronvan_ has quit [Read error: Connection reset by peer]
AaronvanW has joined #bitcoin-wizards
tromp has quit [Ping timeout: 264 seconds]
Aaronvan_ has joined #bitcoin-wizards
rmwb has joined #bitcoin-wizards
AaronvanW has quit [Ping timeout: 246 seconds]
rmwb has quit [Ping timeout: 248 seconds]
Belkaar has quit [Ping timeout: 256 seconds]
Belkaar has joined #bitcoin-wizards
Belkaar has joined #bitcoin-wizards
Chris_Stewart_5 has joined #bitcoin-wizards
tromp has joined #bitcoin-wizards
tromp has quit [Ping timeout: 260 seconds]
Chris_Stewart_5 has quit [Ping timeout: 276 seconds]
belcher_ has quit [Quit: Leaving]
TheoStorm has quit [Ping timeout: 240 seconds]
rmwb has joined #bitcoin-wizards
AaronvanW has joined #bitcoin-wizards
Aaronvan_ has quit [Ping timeout: 240 seconds]
rmwb has quit [Ping timeout: 240 seconds]
shpx has joined #bitcoin-wizards
tromp has joined #bitcoin-wizards
tromp has quit [Ping timeout: 246 seconds]
rmwb has joined #bitcoin-wizards
yongu has left #bitcoin-wizards [#bitcoin-wizards]
d_t has quit [Ping timeout: 276 seconds]
schmidty has joined #bitcoin-wizards
schmidty is now known as Guest44225
Guest44225 has quit [Ping timeout: 240 seconds]
AaronvanW has quit [Ping timeout: 246 seconds]
legogris has quit [Remote host closed the connection]
legogris has joined #bitcoin-wizards
shesek has quit [Ping timeout: 260 seconds]
tromp has joined #bitcoin-wizards
tromp has quit [Ping timeout: 264 seconds]
schmidty has joined #bitcoin-wizards
schmidty is now known as Guest51146
Guest51146 has quit [Ping timeout: 264 seconds]
Murch has joined #bitcoin-wizards
tromp has joined #bitcoin-wizards
Emcy has quit [Ping timeout: 276 seconds]
tromp has quit [Ping timeout: 246 seconds]
shesek has joined #bitcoin-wizards
shesek has joined #bitcoin-wizards
shesek has quit [Changing host]
dabura667 has quit [Quit: Leaving]
CheckDavid has joined #bitcoin-wizards
ghost43 has quit [Ping timeout: 268 seconds]
tromp has joined #bitcoin-wizards
tromp has quit [Ping timeout: 264 seconds]
Murch has quit [Quit: Snoozing.]
meshcollider has quit [Quit: Connection closed for inactivity]
ghost43 has joined #bitcoin-wizards
tromp has joined #bitcoin-wizards
tromp has quit [Ping timeout: 264 seconds]
go1111111 has joined #bitcoin-wizards
dx25 has quit [Ping timeout: 245 seconds]
dx25 has joined #bitcoin-wizards
schmidty has joined #bitcoin-wizards
schmidty is now known as Guest48221
LeMiner2 has joined #bitcoin-wizards
Guest48221 has quit [Ping timeout: 268 seconds]
LeMiner has quit [Ping timeout: 264 seconds]
LeMiner2 is now known as LeMiner
tromp has joined #bitcoin-wizards
Emcy has joined #bitcoin-wizards
meshcollider has joined #bitcoin-wizards
daszorz has joined #bitcoin-wizards
<CubicEarths>
Thinking about ways initial syncing could be made faster... validating everything form the beginning is a serial process. What if a semi-trusted provided information about various intermediate UTXO states, providing UTXO snapshots at different block heights. Then the load could be split up among different computers.
<CubicEarths>
If the information provided was accurate, everything should match up once they chain fragments were fully verified.
<CubicEarths>
If the semi-trusted source had lied, the fragments would not line up, and the lie would be revealed. So it would just extending trust to allow for parallel computations.
<CubicEarths>
( I am looking around at all of the computers surrounding me and wishing they could all work together to sync the chain faster)
cryptojanitor has joined #bitcoin-wizards
* CubicEarths
thinks he really has a good idea this time
thrmo_ has joined #bitcoin-wizards
thrmo has quit [Ping timeout: 248 seconds]
<kallewoof>
CubicEarths: you need to trust whoever provides the snapshots or they could stuff it with fake utxos and fool your node afterwards.
thrmo_ is now known as thrmo
<CubicEarths>
kallewoof: I don't think so. If you were provided bad intermediate UTXO sets, you would just fail to validate, because the fragments wouldn't line up. And since you wouldn't know why they didn't line up, you would have made no progress.
<kallewoof>
CubicEarths: actually I missed the fragment part. I think people have thought about that.
<kallewoof>
CubicEarths: Could be wrong though. It feels like it comes up every week on the mailing list :)
<CubicEarths>
I def. don't see everything, but I've not been aware of this take on the idea
<CubicEarths>
kallewoof: Do you understand what I am describing?
<kallewoof>
CubicEarths: I think so. You basically have people give a partial representation of the UTXO set that will only validate if the parts all show the same UTXO set (i.e. nothing added/removed).
<CubicEarths>
Well, lets say you just split the chain into two parts. I give you the UTXO set as of block 300,000. On computer A you validate from genesis to 300k (at which point you would verify that it matches the UTXO I provided you). On computer B, you would validate from the provided UTXO set until the current block.
<CubicEarths>
kallewoof: So they would be a full representations of the utxo sets.. just at various points in block history
CheckDavid has quit [Quit: Connection closed for inactivity]
spinza has quit [Quit: Coyote finally caught up with me...]
spinza has joined #bitcoin-wizards
shpx has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
_whitelogger_ has joined #bitcoin-wizards
vicenteH` has joined #bitcoin-wizards
arowser has quit [Ping timeout: 260 seconds]
punch has quit [Ping timeout: 260 seconds]
legogris has quit [Ping timeout: 260 seconds]
cannedprimates has joined #bitcoin-wizards
jl2012 has joined #bitcoin-wizards
asoltys_ has quit [Ping timeout: 260 seconds]
gazab has quit [Ping timeout: 260 seconds]
cannedprimates has quit [Changing host]
jl2012 has quit [Changing host]
_whitelogger has quit [Remote host closed the connection]
so has joined #bitcoin-wizards
jaromil has joined #bitcoin-wizards
jaromil has joined #bitcoin-wizards
jaromil has quit [Changing host]
AaronvanW has joined #bitcoin-wizards
deusexbeer has quit [Ping timeout: 264 seconds]
deusexbeer has joined #bitcoin-wizards
espes__ is now known as espes
TheoStorm has joined #bitcoin-wizards
bildramer1 has joined #bitcoin-wizards
thrmo has quit [Quit: Waiting for .007]
bildramer has quit [Ping timeout: 252 seconds]
Chris_Stewart_5 has joined #bitcoin-wizards
LeMiner has quit [Ping timeout: 264 seconds]
Belkaar has quit [Quit: bye]
belcher has joined #bitcoin-wizards
yoleaux has joined #bitcoin-wizards
schmidty has joined #bitcoin-wizards
schmidty is now known as Guest43997
BCBot_ has quit [Remote host closed the connection]
BCBot has joined #bitcoin-wizards
Chris_Stewart_5 has quit [Ping timeout: 246 seconds]
Samdney has joined #bitcoin-wizards
thrmo has joined #bitcoin-wizards
Giszmo has joined #bitcoin-wizards
Chris_Stewart_5 has joined #bitcoin-wizards
laurentmt has joined #bitcoin-wizards
meshcollider has quit [Quit: Connection closed for inactivity]
laurentmt has quit [Quit: laurentmt]
Monerosuxs has quit [Quit: Monerosuxs]
Monerosuxs has joined #bitcoin-wizards
Monerosuxs is now known as Olufunmilayo
Chris_Stewart_5 has quit [Ping timeout: 240 seconds]
dougsland has joined #bitcoin-wizards
douglas_ has joined #bitcoin-wizards
dougsland has quit [Ping timeout: 240 seconds]
SopaXorzTaker has joined #bitcoin-wizards
sammi` has quit [Quit: Lost terminal]
douglas__ has joined #bitcoin-wizards
douglas_ has quit [Ping timeout: 265 seconds]
weez17 has quit [Quit: leaving]
Chris_Stewart_5 has joined #bitcoin-wizards
Guyver2 has joined #bitcoin-wizards
Belkaar has joined #bitcoin-wizards
Belkaar has joined #bitcoin-wizards
Belkaar has quit [Changing host]
Belkaar has quit [Client Quit]
Belkaar has joined #bitcoin-wizards
Belkaar has joined #bitcoin-wizards
Belkaar has quit [Changing host]
Belkaar has quit [Ping timeout: 256 seconds]
Belkaar has joined #bitcoin-wizards
Belkaar has joined #bitcoin-wizards
Belkaar has quit [Changing host]
shpx has joined #bitcoin-wizards
AaronvanW has quit [Remote host closed the connection]
AaronvanW has joined #bitcoin-wizards
Murch has joined #bitcoin-wizards
AaronvanW has quit [Ping timeout: 240 seconds]
daszorz has quit [Read error: Connection reset by peer]
<maaku>
CubicEarths: validate state backwards. starting with the tip, validate pror blocks stepping backwards, with their inputs given alongside them
<stevenroose>
maaku: is that actually been researched? that's actually quite a nifty idea
<maaku>
it's just an old idea from this channel
<stevenroose>
if you reasonably trust the longest chain you have (based on work), you could validate backwards while accepting new blocks
<eklitzke>
how would you know which was the longest chain
<maaku>
eklitzke: block headers
<stevenroose>
nicething is that while you're validating backwards, you are building the actual UTXO set
<maaku>
yes, you could take an untrusted UTXO set and validate backwards
<eklitzke>
block header has the difficulty for a given block not the accumulated difficulty
<stevenroose>
maaku: I mean without a starting utxo set
<stevenroose>
eklitzke: you fetch all headers first, which is fast, and validate if it has enough work
<maaku>
stevenroose: then I'm not sure what the gain is? you'd have to do full IBD before you coudl do anything
<eklitzke>
i see
<stevenroose>
maaku: not entirely, based on the assumption that people are more likely to spend more recent outputs
<maaku>
if you get a UTXO snapshot from an untrusted source, you can do IBD while making progress on the tip and while doing normal full node activities
<maaku>
while making a personal choice about when you start trusting the chain (1 week back? 1 month back?)
<stevenroose>
while going backwards, all new outputs are automatically added to your final UTXO set and you take pending inputs with you backwards and use them to cancel out outputs
<maaku>
eh, most of our infrastructure assumes the peers you talk to have the utxo set. without that assumption you wouldn't know what inputs to relay. messages would have to be added.. it gets real messy real fast
<stevenroose>
would be so nice though if an outpoint would somehow give you an idea of how old it is
<stevenroose>
maaku: you can refrain from relaying until after IBD
<maaku>
stevenroose: pettycoin had per-block commitments of the spend's depth
<maaku>
or height of the input; i forget
<stevenroose>
you can do this: get headers and reasonably trust header chain
<maaku>
stevenroose: then again, what would be the point? you're not able to do full-node things
<stevenroose>
the go backwards and for every block:
<maaku>
i know you CAN do it. I've thought this through, like 5 years ago ;) I just don't see the point
<stevenroose>
add inputs to input set and for every output check if you saw an input, then validate, otherwise add to UTXO
<stevenroose>
it means that you are constructing the REAL UTXO set gradually
<stevenroose>
so the further you go, the more of the new blocks you can actually validate
<stevenroose>
while when going 0 to end, you know nothing, because the UTXO you build is not final and can be spent from in later blocks
<maaku>
I'm not sure why you have the emphasis on "real". Either approach validates the UTXO set.
<stevenroose>
real I mean the one at the best chain
<stevenroose>
you have an actual subset of the latest utxo set against which you can validate blocks
<stevenroose>
just adding unresolves inputs to your pending inputs set
<stevenroose>
it means that on the premise that the chain you are on ends up being the longest (which is increasinly likely if you are getting pow-valid blocks for it), you can already do validation of blocks and get a view into the UTXO set
<stevenroose>
maaku: I'm not saying it will be faster, but it might make your node more useful until you finish. currently an out-of-sync node is useless because even 10 blocks behind, you can't accept a tx spending an output in the UTXO set of your longest validated chain
<stevenroose>
in this alternative method, you can accept txs that spend from your UTXO set, just not the ones that spend outputs older than your progress, which are increasingly rare as you progress
<stevenroose>
I mean ofc getting a UTXO set snapshot is more efficient, but that seems to not gonna happen soon and still requires you to download 10GB of data
<stevenroose>
passing along UTXO snapshots also cause other problems, you will never be able to do it within one block period and most nodes only keep the latest UTXO set, so when your connection is slow (and you have a commitment to a set at a given block), you will find it increasingly harder to find nodes able to server the UTXO set at a given point
<stevenroose>
serve*
d_t has joined #bitcoin-wizards
TheoStorm has quit [Quit: Leaving]
AaronvanW has joined #bitcoin-wizards
samm_ has quit [Remote host closed the connection]
samm_ has joined #bitcoin-wizards
AaronvanW has quit [Ping timeout: 260 seconds]
<CubicEarths>
maaku: I was specifically thinking about using the idea to split work up amongst multiple computers. To make parallel a process that otherwise must be serial.
thrmo has quit [Quit: Waiting for .007]
<CubicEarths>
There downside is that for each additional parallel 'thread', you would need to download yet another UTXO snapshot. So there is clearly a limit (which is likely in the single digits) beyond which nothing is gained...
<maaku>
CubicEarths: to do that I think you just need relative age commitments
<maaku>
then validating any range of blocks would generate two UTXO sets -- those which are assumed-valid from prior to the range, and which are carried forward into the future
<CubicEarths>
What are 'relative age commitments'?
<CubicEarths>
The way I understand the validation process to work, you need to have the complete utxo set for given height to be able to check if all the transactions in the next block are valid. And after processing that next block, you are in turn left with an updated UTXO set.
samm_ has quit [Read error: Connection reset by peer]
<CubicEarths>
So if I want to validate staring at block number 300,000, and proceed until the chain tip, wouldn't I need the utxo set as of block 300,000 as a starting point?
LeMiner has joined #bitcoin-wizards
AaronvanW has joined #bitcoin-wizards
jtimon has joined #bitcoin-wizards
<CubicEarths>
maaku:
<maaku>
require the witness to specify the blockheight of the output being spent
<arubi>
doesn't this break monotonicity? if a chain reorg happens and the input gets into a different height
<CubicEarths>
arubi: The concept that I was describing?
<arubi>
no I mean requiring the witness to specify the height of the txo spent
<arubi>
maybe I'm missing some context so sorry about that
cryptojanitor has quit [Quit: Connection closed for inactivity]
<arubi>
what I mean, if there's a chain of three blocks where each txo spends a next one, then a reorg happens, suddenly when a reorg happens these three spends can't be included in a single block
<arubi>
without specifying the block height of the input, they could be
<arubi>
s/ then a reorg happens,// (writing two things at the same time :) )
laurentmt has joined #bitcoin-wizards
<CubicEarths>
arubi: I was describing a way to make IBD parallel among independent computers. I suggested the user needed to acquire several different UTXO sets, from *intermediate heights*, and then the can validate starting from- and up to those heights and make sure everything matches.
laurentmt has quit [Client Quit]
<sipa>
you can just only store/verify UTXOs with a hash in a certain range
<sipa>
if you want to shard validation
meshcollider has joined #bitcoin-wizards
<CubicEarths>
sipa: How can you make any sense of an arbitrary block without a frame of reference? (which I thought the UTXO was)
TheoStorm has joined #bitcoin-wizards
<sipa>
CubicEarths: just assume that all inputs that spend a UTXO outside your range are valid
<maaku>
arubi: the witness would be generated by the miner when the block is created
<maaku>
no transactions are invalidated on a reorg
<arubi>
maaku, ah, that's the context I was missing.
<CubicEarths>
sipa: I think I understand. So you can just shard it arbitrarily, and then check that the UTXO sets have no conflicts?
<maaku>
yes
<CubicEarths>
So then, why not done? :)
shpx has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<maaku>
you want to write a patch?
<sipa>
the UTXO set is tiny
<sipa>
(for now)
<sipa>
and there are some logisitical issues, like needing a protocol by which the different shards can tell eachother about failure
<sipa>
and wallets needing to contact all the shard to make sure a tx is valid, etc
shpx has joined #bitcoin-wizards
thrmo has joined #bitcoin-wizards
<CubicEarths>
maaku: I was trying to understand if there were difficult theoretical issues...
<maaku>
It's a massive amount of work and unlikely to actually fix the bottleneck to IBD for most people, and unusable by most people (who don't have clusters of machines to IBD), etc.
<maaku>
So for the most part it's a solution in search of a problem.
<CubicEarths>
I can see those points... but why do you think it wouldn't fix the bottleneck (if used)?
<CubicEarths>
(what do you see the bottle as being?)
<CubicEarths>
(bottleNECK)*
<sipa>
initial sync, bandwidth, storage, cpu cost, ...
<CubicEarths>
So it is so well balanced, there is no particular weak point... which seems to ironically be a counter-argument to progress :D
<maaku>
More like it's not broken
<maaku>
It'd only speed up IBD. IBD is fine
<maaku>
Validation of IBD only takes a few hours -- if this was actually the bottleneck.
<sipa>
it would speed up IBD at the cost of multiplying _all_ resource costs (except db size related ones)
<CubicEarths>
I was thinking it would just be a temporary situation: Only used for IBD, and once synced, it would just collapse back to a single machine
<CubicEarths>
maaku: You say that, but am always reading about people complaining, or suggesting it is the reason why blocks ought to be smaller
onabreak has quit [Ping timeout: 260 seconds]
SopaXorzTaker has quit [Remote host closed the connection]
gwollon is now known as gwillen
CheckDavid has joined #bitcoin-wizards
TheoStorm has quit [Quit: Leaving]
thrmo_ has joined #bitcoin-wizards
thrmo has quit [Ping timeout: 260 seconds]
cryptojanitor has joined #bitcoin-wizards
AaronvanW has quit [Remote host closed the connection]
dgenr8 has quit [Ping timeout: 264 seconds]
dgenr8 has joined #bitcoin-wizards
Krellan has joined #bitcoin-wizards
onabreak has joined #bitcoin-wizards
Chris_Stewart_5 has quit [Ping timeout: 260 seconds]
thrmo_ is now known as thrmo
samm_ has joined #bitcoin-wizards
TheoStorm has joined #bitcoin-wizards
douglas__ has quit [Ping timeout: 264 seconds]
Murch has quit [Quit: Snoozing.]
Murch has joined #bitcoin-wizards
CheckDavid has quit [Quit: Connection closed for inactivity]
AaronvanW has joined #bitcoin-wizards
Untapped has joined #bitcoin-wizards
intcat has quit [Remote host closed the connection]
son0p_ has joined #bitcoin-wizards
intcat has joined #bitcoin-wizards
Giszmo has quit [Ping timeout: 256 seconds]
son0p_ is now known as son0p
shpx has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
shpx has joined #bitcoin-wizards
deusexbeer has quit [Quit: Konversation terminated!]
Murch has quit [Quit: Snoozing.]
Murch has joined #bitcoin-wizards
deusexbeer has joined #bitcoin-wizards
Chris_Stewart_5 has joined #bitcoin-wizards
midnightmagic has quit [Ping timeout: 256 seconds]
son0p has quit [Quit: Lost terminal]
shpx has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Chris_Stewart_5 has quit [Ping timeout: 240 seconds]