<jbenet>
daviddias: you can use git clone, too. though i guess you dont need the whole thing-- chriscool has taught me a lot about using git like wget. it's cool. kind of like how we use "ipfs get"
<daviddias>
I liked using the tags cause I could query the tags available and download directly from there. But if there is a better lighter way, let's do it :)
<jbenet>
yeah sounds good
<jbenet>
nah nah, it's all good. (this stuff barely matters at all)
Leer10 has joined #ipfs
inconshreveable has joined #ipfs
<jbenet>
daviddias: we really really should be pulling all the crazy amount of binaries from ipfs
<jbenet>
making the ipfs api is downloading go-ipfs app a bunch of times.
<jbenet>
daviddias: ALSO! i think this will make broken builds, because the `go-ipfs` package installs the go-ipfs for YOUR arch. which i think wont work for the other dists?
Eudaimonstro has quit [Ping timeout: 264 seconds]
<jbenet>
so am not sure npm dist-all will work :(
<daviddias>
re: putting everything on IPFS - every single time, so much bandwidth and dup data
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
daviddias: yeah. its crazy. im running dist and it's downloading the same stuff from gobuilder and github a bunch of times.
<jbenet>
daviddias: `ipget` will go _a long way_ towards this, because it avoids the complexity of a local daemon to manager.
<jbenet>
and it's purely meant as a thing like "wget".
<daviddias>
re: broken builds - oh, haven't thought of that! We need to add to ipfsd-ctl an option to select the arch, and we can check that through electron env
<jbenet>
yeah likely.
<M-davidar>
jbenet: UTC+10, you all seen to be active while I'm asleep :)
<M-davidar>
Probably still is, eta 48h last time I checked
<jbenet>
M-davidar: i added filters, so it will go faster (no network ops)
<daviddias>
One quick thing we could do is asking pinbot to ipfs add all the tags and automatically add the hash to the tag description, so that ipfs/dist can look for the hash and through ipget, get the code
<jbenet>
yeah
<jbenet>
daviddias: would be sweet.
<M-davidar>
Cool, would be nice for ipfs add to have similar stored as untar :)
<jbenet>
probably what we need is a repo like "go-ipfs-dist" which pulls go-ipfs, does all the magic of building it, and then publishes all the pieces to ipfs.
<jbenet>
then all the the distributions repo has to do is: ipfs object add-link $releases $dist $dist-ipfs-root
Leer10 has joined #ipfs
Eudaimonstro has joined #ipfs
<daviddias>
Why not take the builds from gobuilder directly ?
<jbenet>
daviddias: oh in the future we'll want to ensure we get the same bits as gobuilder, so we can sign them.
<daviddias>
Got it! :)
<jbenet>
go 1.5 makes cross-compile easier, but not sure about reproducibility.
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Leer10 has quit [Ping timeout: 250 seconds]
<M-hash>
go 1.4 gets a gold star for reproducability; haven't checked with 1.5 yet, but i've got repeatr building itself perfectly deterministically right now on 1.4.
<jbenet>
M-hash on all archs?
* jbenet
bb reloc
<M-hash>
mmm i've only been bothering with linux, so no comment, sorry
<drathir>
what is correct way of comment in config?
rschulman has left #ipfs [#ipfs]
rschulman has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<M-hash>
i don't currently have any projects with cross compilation set up to test it on, but i did also just check that go-1.5-linux-amd64 definitely does still have deterministic output fwiw
Eudaimonstro has quit [Ping timeout: 240 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<lgierth>
M-hash: can you tell me more about your cross setup? i want to try cross-compiling for openwrt
Eudaimonstro has joined #ipfs
<jbenet>
M-hash what we need to test is an [ (os-arch) x (os-arch) ] matrix where we confirm that every os-arch pair is compiled by every os-arch pair exactly the same.
compleatang has quit [Ping timeout: 260 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
compleatang has joined #ipfs
compleatang has quit [Ping timeout: 244 seconds]
compleatang has joined #ipfs
compleatang has quit [Read error: Connection reset by peer]
compleatang has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Eudaimonstro has quit [Ping timeout: 272 seconds]
zabirauf has joined #ipfs
Eudaimonstro has joined #ipfs
bedeho_ has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Eudaimonstro has quit [Remote host closed the connection]
simonv3 has quit [Quit: Connection closed for inactivity]
Eudaimonstro has joined #ipfs
substack has quit [Ping timeout: 250 seconds]
substack has joined #ipfs
<ipfsbot>
[go-ipfs] rht created gpe-35ab1e347b8c081b2a2e449a8a6df69244e7cc86 at 35ab1e3 (+0 new commits): http://git.io/vs6Vv
<ipfsbot>
[go-ipfs] rht created gpe-dfa0351df93042973774ea66dd19aee487d7d77d at dfa0351 (+0 new commits): http://git.io/vs6VG
<ipfsbot>
[go-ipfs] rht created gpe-9f0c8134cb4c6a1a769437c685b39b233759d172 at 9f0c813 (+0 new commits): http://git.io/vs6Vl
<ipfsbot>
[go-ipfs] rht created gpe-d32177cda06d1ae5988f657711d8fa8d0887c0ab (+1 new commit): http://git.io/vs6V4
<ipfsbot>
go-ipfs/gpe-d32177cda06d1ae5988f657711d8fa8d0887c0ab d32177c rht: Make sure ctx in commands are derived from req.Context...
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<ipfsbot>
[go-ipfs] rht created gpe-5d8e15042f361e59f729d2c8f11724d58f2c7b1e (+1 new commit): http://git.io/vs6Va
<ipfsbot>
go-ipfs/gpe-5d8e15042f361e59f729d2c8f11724d58f2c7b1e 5d8e150 rht: Replace WithTimeout with WithCancel whenever possible...
M-davidar is now known as davidar
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<ipfsbot>
[go-ipfs] rht created gpe-34e06f6c95a5475c00743f72b5b5e56ddf747116 (+1 new commit): http://git.io/vs66s
<ipfsbot>
go-ipfs/gpe-34e06f6c95a5475c00743f72b5b5e56ddf747116 34e06f6 rht: Wire a context down to (n *helpers.UnixfsNode) GetChild...
zabirauf has quit [Quit: My Mac has gone to sleep. ZZZzzz…]
<ipfsbot>
[go-ipfs] rht created gpe-a7202fa94cbb57689940873c19aac9ad786d9f29 (+1 new commit): http://git.io/vs66E
<ipfsbot>
go-ipfs/gpe-a7202fa94cbb57689940873c19aac9ad786d9f29 a7202fa rht: Fix 'ctx, _' to have explicit cancel...
<ipfsbot>
[go-ipfs] rht created gpe-fcf915fc4712566973cff2adac39e6f33ec56c84 (+1 new commit): http://git.io/vs66V
<ipfsbot>
go-ipfs/gpe-fcf915fc4712566973cff2adac39e6f33ec56c84 fcf915f rht: Define discoveryConnTimeout and set it to 30s...
amstocker has quit [Ping timeout: 255 seconds]
<ipfsbot>
[go-ipfs] rht created gpe-30e03b2da610b158ff061769080594dacc39df2b (+1 new commit): http://git.io/vs66i
<ipfsbot>
go-ipfs/gpe-30e03b2da610b158ff061769080594dacc39df2b 30e03b2 rht: Replace context.TODO in test files with context.Background...
<ipfsbot>
[go-ipfs] rht created gpe-dc2153574cc69dbf519f718369eba3a609799c2f (+1 new commit): http://git.io/vs66y
<ipfsbot>
go-ipfs/gpe-dc2153574cc69dbf519f718369eba3a609799c2f dc21535 rht: Remove ctx timeout in unixfsNode GetChild...
<ipfsbot>
[go-ipfs] rht created gpe-ccd06e5b66621568f7b66d9867779eaef843a75f (+1 new commit): http://git.io/vs669
<ipfsbot>
go-ipfs/gpe-ccd06e5b66621568f7b66d9867779eaef843a75f ccd06e5 rht: Fix ctx used in pub.Publish...
<ipfsbot>
[go-ipfs] rht created gpe-de5c0ceff0857ee3ee4be85785b3253c0ef98c28 (+1 new commit): http://git.io/vs667
<ipfsbot>
go-ipfs/gpe-de5c0ceff0857ee3ee4be85785b3253c0ef98c28 de5c0ce rht: Localize the scope of context.WithCancel for every DAG.Get...
<ipfsbot>
[go-ipfs] rht created gpe-6325e6c75e144bcf9a177f5f6f854c5776abee4f (+1 new commit): http://git.io/vs66A
<ipfsbot>
go-ipfs/gpe-6325e6c75e144bcf9a177f5f6f854c5776abee4f 6325e6c Tommi Virtanen: sharness: Use sed in a cross-platform safe way...
<ipfsbot>
[go-ipfs] rht created gpe-50860f0d5ceb8580b4b44dd8f9fddd71f0c5815f (+1 new commit): http://git.io/vs6ie
<ipfsbot>
go-ipfs/gpe-50860f0d5ceb8580b4b44dd8f9fddd71f0c5815f 50860f0 Tommi Virtanen: sharness: `fusermount -u` is the documented way to unmount FUSE on Linux...
pfraze has joined #ipfs
inconshreveable has quit [Remote host closed the connection]
amstocker has joined #ipfs
inconshreveable has joined #ipfs
pfraze has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
inconshreveable has quit [Remote host closed the connection]
Leer10 has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
sharky has quit [Ping timeout: 250 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
sharky has joined #ipfs
Tv` has quit [Quit: Connection closed for inactivity]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Eudaimonstro has quit [Remote host closed the connection]
Eudaimonstro has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
infinity0 has quit [Remote host closed the connection]
infinity0 has joined #ipfs
Leer10 has quit [Read error: Connection reset by peer]
zabirauf has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Leer10 has joined #ipfs
<ipfsbot>
[go-ipfs] whyrusleeping pushed 1 new commit to private-blocks: http://git.io/vs6j6
<ipfsbot>
go-ipfs/private-blocks b622bfb Jeromy: rename blockstores to Data and State...
<ipfsbot>
[go-ipfs] whyrusleeping force-pushed node-construct-v2 from 373d334 to 3e2b7ae: http://git.io/v3PkK
<ipfsbot>
go-ipfs/node-construct-v2 889049c Jeromy: replace nodebuilder with a nicer interface...
<ipfsbot>
go-ipfs/node-construct-v2 40a794d Jeromy: use NewNode instead of NewIPFSNode in most of the codebase...
<ipfsbot>
go-ipfs/node-construct-v2 f1c1832 Jeromy: make mocknet work with node constructor better...
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Leer10 has quit [Read error: Connection reset by peer]
<whyrusleeping>
| req.Context... │ akhavr
<whyrusleeping>
whoops...
<whyrusleeping>
thats what i get for leaning on my trackpad
Leer10 has joined #ipfs
mildred has joined #ipfs
voxelot has quit [Ping timeout: 246 seconds]
amstocker has quit [Ping timeout: 246 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
mildred has quit [Quit: Leaving.]
mildred has joined #ipfs
amstocker has joined #ipfs
mildred has quit [Ping timeout: 250 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
mildred has joined #ipfs
chriscool has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<ipfsbot>
[go-ipfs] jbenet closed pull request #1585: Make sure ctx in commands are derived from req.Context (master...cleanup-context) http://git.io/v3bTG
amstocker has quit [Ping timeout: 246 seconds]
gunn has quit [Ping timeout: 244 seconds]
gunn has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
keroberos has quit [Max SendQ exceeded]
sharky has quit [Ping timeout: 264 seconds]
marianoguerra has joined #ipfs
keroberos has joined #ipfs
bedeho_ has quit [Ping timeout: 246 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<M-hash>
this only covers linux/amd64 as the start for the moment, but if you run it twice, you should certainly get a tasty table of the same output hashes over and over :)
<M-hash>
ofc, replacing all those inputs with ipfs is a todo ;)
<cryptix>
gmorning ipfs
zabirauf has quit [Max SendQ exceeded]
Encrypt has joined #ipfs
<lgierth>
M-hash: very cool!
<lgierth>
:):)
<lgierth>
and good morning everyone
mkarrer_ has quit []
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<cryptix>
M-hash: windows should work with the -tags nofuse
<davidar>
morning lgierth, how much dedup did you end up getting for openwrt?
HastaJun has quit [Ping timeout: 246 seconds]
inconshreveable has quit [Remote host closed the connection]
<lgierth>
davidar: i didn't bother continuing to add it
<davidar>
Ah, what was the eta?
<lgierth>
sooo slow
<lgierth>
i don't know
<lgierth>
a week?
<lgierth>
it was slower that the download
<lgierth>
two things that we should have before these mirrors really make sense: 1) concurrent add, and 2) unixfs files with timestamps
<lgierth>
then we can wget --mirror directly into a fuse mount
<davidar>
Hmm, yeah, I'm seeing the same issue with the arxiv data
<lgierth>
sorry i was blocking you with the access btw, juan took care of it in the end
<davidar>
yeah, that would be awesome
<davidar>
No worries
<lgierth>
you should have davidar@(pollux|sirius).i.ipfs.io by now
<lgierth>
could you try it out? it would be good to use root only when we have to
<lgierth>
jbenet: btw, re: our brief discussion about slow adds, if the blocks are already stored, it's about as fast as i'd expect it to be -- it might be the writes that are slowing it down
<davidar>
cool, I was a bit uncomfortable doing everything as root :)
<davidar>
mind if I leave the current process running though?
<lgierth>
yeah can move it to your $HOME for the next run
<lgierth>
(you still have root@)
<davidar>
OK, will do
<lgierth>
if you install stuff, make a PR against ipfs/infrastructure, so that we can keep track if infrastructure changes
inconshreveable has joined #ipfs
<jbenet>
lgierth: yeah I think so too. It's a bunch of fsyncs-- the concurrent add pr will improve things dramatically
<davidar>
but hopefully resident mem for ipfs can be reduced :)
<davidar>
should I submit an issue?
<slothbag>
anyone tried updating to go 1.5 yet? i'm getting "no such tool "6g" trying to compile ipfs
<daviddias>
jbenet: still awake ? Available to talk about Record Spec?
<davidar>
slothbag: I believe there's been some issues upgrading from 1.4 to 1.5
<slothbag>
yeah, i'd say so.. time to downgrade back to 1.4 i guess
sharky has joined #ipfs
<jbenet>
daviddias give me 30m
<daviddias>
ok :) (just have in mind that in 1 hour I've to leave for about 1:15h)
<cryptix>
davidar: "undefined: sync.Pool" indicates you have an older go version (pre 1.4 iirc)
<cryptix>
slothbag: "no such tool "6g" .." sounds like a dirty go installation - mixing up 1.4 and 1.5
<davidar>
cryptix: yeah, just realised that :)
<jbenet>
daviddias ok i can talk now
<daviddias>
rad
<daviddias>
hangouts?
<slothbag>
cryptix: I had a single go directory with 1.4, i deleted it and replaced it with the same go folder from 1.5 zip.. cant imagine it'd be dirtu
<slothbag>
dirty
ei-slackbot-ipfs has quit [Remote host closed the connection]
<jbenet>
daviddias: ahh cant hangout atm. can type now, hang out in 25
<daviddias>
type can be ok
ei-slackbot-ipfs has joined #ipfs
<daviddias>
I essentially want to know what's your head (and preferably in a spec) about Records and Record Store
<daviddias>
want to implement that for libp2p :)
<cryptix>
slothbag: '$ whereis go' ? , did you maybe setup $GOROOT to another location
<cryptix>
slothbag: otherwise try deleting your $GOPATH/pkg and /bin, but that error should look differently
<cryptix>
slothbag: "no such tool "6g" is what happens when a 1.4 go tool tries to work with an 1.5 toolchain (6g, 8g, are gone there)
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
zorun has quit [Ping timeout: 246 seconds]
<jbenet>
records is already in a spec-- https://github.com/ipfs/specs/tree/master/records (layers on top of merkledag) and up to now, RecordStore is only a datastore (local get/put for records). it's not a networked protocol. we've discussing making it one, but we haven't made a concrete proposal.
<jbenet>
for distributing records -- records are merkledag objects, and so an be retrieved with bitswap, but most cases you want to make sure people serve them to each other (e.g. on the dht)
<jbenet>
we discussed with whyrusleeping in the past taking the "get/put" part of the dht and making it into a small little protocol
<daviddias>
right. So now, I've the DHT(peer routing) giving me the PeerInfo the the peers responsible for a given key
<daviddias>
so I open a stream to their 'record store/network'ed service' and get/put the records I want
oleavr has quit [Ping timeout: 246 seconds]
machrider has quit [Ping timeout: 246 seconds]
Luzifer has quit [Ping timeout: 246 seconds]
machrider has joined #ipfs
<slothbag>
getting this error on windows: error while writing stream WSASend tcp4 127.0.0.1:5001: An established connection was aborted by the software in your host machine. module=commands/http
bigbluehat has quit [Ping timeout: 246 seconds]
mg- has quit [Ping timeout: 246 seconds]
Luzifer has joined #ipfs
<jbenet>
why does "'record store/network'ed service'" need to be a thing? the "get/put" relationship is governed by whatever protocol sits on top-- for isntance, in a dht, you'd put records to specific peers according to the dht lookup strategy. in mdns, you probably would not put records at all, and only respond to gets. (or if put, then have to put everything-- but
<jbenet>
this is a different sort of protocol: gossip)
otherchas__ has quit [Ping timeout: 246 seconds]
sugarpuff has quit [Ping timeout: 246 seconds]
daviddias has quit [Read error: Connection reset by peer]
<jbenet>
so-- the gist is that i'm not seeing yet (or maybe i lost it since we last talked) what we gain by having the record store be a networked service itself, in addition to (or instead of) the dht proper.
<davidar>
slothbag: I don't see any existing issues about the daemon memory usage, so feel free to submit one
<Luzifer>
(dunno why every channel needs to have some fancy auto-welcome message…)
<daviddias>
decoupling the get/put from the DHT gives us the opportunity to have a common interface to talk with the record store that can be used independently of the peer-routing mechanism
<daviddias>
peer-routing would only serve to identify which peers we should talk to
<lgierth>
daviddias: and get you a couple of multiaddrs for each, right?
<daviddias>
lgierth: yep
<daviddias>
you could even use multiple peer-routing mechanisms at the same time
<daviddias>
and then use all of the PeerInfos they return to contact their record stores
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
daviddias: what i mean is that the decision of "whom to put to" and even "whether to put at all" is governed by the {dht, mdns, etc} mechanics, not by another protocol
<lgierth>
so peer-routing is wants-aware peer discovery, eh
Encrypt has quit [Quit: Quitte]
<jbenet>
daviddias: so a get/put protocol would really just be a blank get/put, which has to be mediated by the dht anyway. (you do not want to allow random peers putting random data to you nonstop, or using you as their cache (i.e. always getting from you)). and the protocols we might chose (i.e. whether to rate limit, by how much, etc) are all dependendent on the
<jbenet>
{dht, mdns, gossip, ...} protocols
<jbenet>
as in, how do i know i should store a put from you? because i'm part of a dht and you just sent me a dht message.
<daviddias>
right :)
<daviddias>
in order to know if I should put/get from you, I should first learn from the DHT (or another peer-routing mechanism)
<jbenet>
and how do i know to ignore puts from you? because i'm not engaged in a "record sharing" protocol with you, or because you've gone over its limits, or not following its rules in some way.
<daviddias>
I think in the same page
<jbenet>
no, what i mean is, i should not have a wide open handler for data that lets you store stuff. it should be rate limited by the proper protocol (eg dht)
<daviddias>
I think we are in the same page
<daviddias>
We can have the rate limiting on the Record Store itself
<jbenet>
hmm dont think so. i think you're saying you want a separate protocol for storing / retrieving records, and i'm saying we cannot have that without intimate knowledge of the protocol that give you a right to put records,
<jbenet>
i dont think so-- it depends entirely on what protocol lies on top: different dhts will have different rules, things like mdns might just not let you put at all, gossip protocols might let you store a lot. something like ssb replication would require a check to the social links
<jbenet>
we could abstract out the get/put mechanics, but it would have to either be replicated, or heavily configured by the parent protocol
<jbenet>
like, if i'm part of dht1 and dht2, which have different rules as to who gets to put to me, or how much, i'd either need two get/put things, or have one that delegates its decisions to the other two.
<jbenet>
(i'm usually all about abstracting out all the things), im not seeing what is gained here though--- do you have a concrete example that reconciles what i discuss here o/ -- maybe we should look at concretely to see which part we're discussing
<daviddias>
The way I was seeing it (after the whiteboard discussion) is that peer-routing would never see the data that is going to be fetched or stored in other peers, peer-routing would only give us the PeerInfo (addrs, pubkeys and Ids) of the peers we should talk for a given key
<daviddias>
get(key) -> []PeerInfo
<daviddias>
and then libp2p would do swarm.openStream('/record-store/1.0.0', PeerInfo, function (err, stream) {<do all the fetch or stores>})
therealplato has quit [Ping timeout: 264 seconds]
<daviddias>
an mDNS peer routing would not have any intuitively scheme to pick which of the Peers should be responsible for a given key, so it makes sense if I broadcast a message over mDNS of whom has 'key', only those who have the key, reply to me.
<jbenet>
(lets ignore sync/sloppy distinctions for a moment) -- "kadDHT.get(key)" and "mdnsHT.get(key)" would do different things. in the former case, "kadDHT.get(key)" would query a kademlia dht for a key, and return the values received. "mdnsBroadcast.get(key)" might send out an mdns packet asking for "key", receive various responses, and return them, a different
<jbenet>
protocol yet ("pushGossip.Get(key)" would do no network io, and only read local storage (i.e. gossip only on push)
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
so the interface "get(key) -> []PeerInfo" does not fit this. --- let's try suffling the interfaces for a sec-- suppose we have a "Distributed Record Store" thing, which takes a "distribution protocol" with a simple "get(key) val, put(key, val)" interface. then, "DRS" implements all the record specific stuff, and uses the "distribution protocol" to do its
<jbenet>
get/puts. DRS doesn't care what peers are talked to, or even if peers are talked to at all. it just cares that the data is stored + retrivable according to the guarantees provided by "distribution protocol"
<jbenet>
where "distribution protocol" could be any of: a local hash table, a kademlia dht, mdns, a social network (like ssb), any datastore (s3, redis, etc)"
<jbenet>
so the interface would be: "dp = DHT(); drs = RecordStore(dp); drs.put(key, val, cb) /* which in turn calls */ dp.put(key, val, cb)"
<daviddias>
ok, so clearly there was some idea mutation since that discussion. Although it does feel to me that your conception of DHT is what I call peer routing + record store + logic done inside the libp2p module (which also exports the interface for layers above) and that is why I think in the end we are in the same page
<daviddias>
in my head I have:
<jbenet>
(yeah probably, sorry if im changing things up on you, i dont mean to. i'm trying to find the interfaces that make the most sense now, as once we go down this path, turning back is really really hard)
<daviddias>
libp2p.get = function (key) {
<daviddias>
var peers = kadRouter.get(key)
<daviddias>
peers.forEach(function (peer) {
<daviddias>
peer.openStream('/record-store/1.0.0', peer, function (stream) {
<daviddias>
// get the record I want from this peer
<daviddias>
})
<daviddias>
})
<daviddias>
// once I have the records retrieve
<daviddias>
return records
<daviddias>
}
<daviddias>
If I understand correctly, you call RecordStore what I call libp2p and you see the DHT transmitting the data on the DHT messages as where I (from the whiteboard discussion) got that would be a two step thing 1) get peers to talk 2) store/get from those peers
<daviddias>
I need to run, will be able to talk in the phone for next 20 minutes and then come back at full speed at 14:20 (Lisbon Time)
<daviddias>
definitely want to continue the convo, getting good abstractions is really important, in the end I'm good with whatever makes more sense, I don't mind rewriting my stuff
<jbenet>
ok sounds good, let's talk at 14:20. idea: let's flesh out two examples of alternative uses, and align our interfaces that way
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<daviddias>
Ok :)
voxelot has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<davidar>
what is the use case for iprs btw?
<jbenet>
davidar: "records" on the {DHT, mdns, whatever you want}.
<jbenet>
davidar: the point is that we'd like a common interface on how to distribute important cryptographic distributed-system metadata with certain properties (like required signatures, or temporal validity, etc)
<davidar>
what's it likely to be used for in practice?
<davidar>
(ie. is it something I should be interested in as a user? :)
<davidar>
or is it just internal bookkeeping stuff for ipfs
<jbenet>
yeah for the dht provide records, and ipns records
<jbenet>
(those are all "records" on "iprs")
<davidar>
ah, OK
<davidar>
so, the equivalent of DNS records, roughly speaking
Encrypt has joined #ipfs
slothbag has quit [Quit: Leaving.]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
yep
M-mistake has quit [Remote host closed the connection]
davidar has quit [Remote host closed the connection]
M-staplemac has quit [Remote host closed the connection]
M-prosodyContext has quit [Remote host closed the connection]
M-hash has quit [Remote host closed the connection]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
lgierth: installing go on pollux--- what's the best strat?
krl has joined #ipfs
M-staplemac has joined #ipfs
M-hash has joined #ipfs
M-prosodyContext has joined #ipfs
M-mistake has joined #ipfs
M-davidar has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
silotis has quit [Quit: No Ping reply in 180 seconds.]
silotis has joined #ipfs
danslo has quit [Remote host closed the connection]
zorun has quit [Ping timeout: 246 seconds]
zorun has joined #ipfs
<krl>
o/
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<krl>
lgierth: poing
<lgierth>
krl: here
<lgierth>
i'm on the cors thing
<krl>
lgierth: <3
<lgierth>
sorry i'm blocking you so long
<lgierth>
badly need to increase the bus factor for infrastructure.git
<krl>
lgierth: not really blocking anything except gateway demo
<lgierth>
(also i don't wanna be the infrastructure person)
<krl>
but that would be nice to have
<lgierth>
jbenet: maybe $HOME == $GOROOT
<lgierth>
jbenet: requires the least setup, with $HOME/bin already being in the path, etc.
<lgierth>
jbenet: also easiest to erase later if we have to
<lgierth>
i wasn't particularly smart about partitioning on pollux... not having a data partition makes it unneccessarily hard to rebuild the host
<cryptix>
$HOME==$GOPATH i can see. shouldnt fiddle with $GOROOT
<lgierth>
eh yeah gopath
<lgierth>
thanks
<reit>
jbenet: that worked great, thanks
danslo has joined #ipfs
therealplato has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<lgierth>
krl: grml have to head out again. what's seems really annoying is that access-control-allow-origin can't contain multiple hostnames
<lgierth>
i can set it to gateway.ipfs.io only for now
danslo has quit [Remote host closed the connection]
danslo has joined #ipfs
danslo has quit [Client Quit]
danslo has joined #ipfs
besenwesen has quit [Ping timeout: 246 seconds]
step21 has quit [Ping timeout: 246 seconds]
<jbenet>
lgierth: what part? the gateway's code should handle multiple hostnames fine (else it's a bug-- feature is intended)
<lgierth>
stackoverflow suggests setting *, then responding with the request origin only if there is one in the request
<lgierth>
yeah?
<lgierth>
ok
<lgierth>
i'll have to try that later today
<lgierth>
meh doesn't work yet
<lgierth>
gotta run
<lgierth>
bbiab
<jbenet>
kk later
<jbenet>
hmmm that's for the API-- maybe "Gateway.HTTPHeaders" instead of "API.HTTPHeaders"? (not sure what config it's grabbing the headers from-- i merged the CORS stuff before the readable gateway
besenwesen has joined #ipfs
besenwesen has joined #ipfs
<jbenet>
maybe gateway handler is only checking, but not setting headers, unlike API.
drathir has quit [Ping timeout: 246 seconds]
danslo has quit [Remote host closed the connection]
<cryptix>
short recap: git-remote-ipfs now theoretically solved. M-davidar pointed me to git-remote-dropbox which works exactly like i want it for -ipfs - porting right now
* cryptix
is cooking right now - l8ters
Eudaimonstro has joined #ipfs
<drathir>
hi all...
marianoguerra has quit [Ping timeout: 250 seconds]
<krl>
lgierth: o/
* whyrusleeping
here
<whyrusleeping>
not much to report. Gonna work through my open PRs this week.
Eudaimonstro has quit [Ping timeout: 245 seconds]
Eudaimonstro has joined #ipfs
<lgierth>
ok daviddias wanna go first?
<daviddias>
sure
* daviddias
SPRINT CHECKING
<daviddias>
This past week was +- productive. Coming back home after 53 of travelling meant that I had to take care of a lot of stuff in the first couple of days.
<daviddias>
- [x] Wrote more tests to the kad-router (which covered some more edge cases)
<daviddias>
- [x] Reviewed HTTP-P2P proposal
<daviddias>
- [x] Worked a lot of revamping the network spec, now known as libp2p. It still requires more though, specially after having a good conversation today with @jbenet where we learned that we weren't completely in sync of how the Record Store would work, did some progress, hopefully it will be nailed down by today node-ipfs meeting
<daviddias>
- [x] continue pursuing my role in the W3C WebRTC WG, documented railing process and still digesting all of the info available
<krl>
Starlog can now do infinite scroll, publishing and viewing of entries. Webcomponents turn out to be a really nice match to ipfs, from starlog repo you can now do 'npm run local', and 'npm run gateway' to use vulcanize to pack everything up into one html file, and view it on 5001 or gateway (still fails on CORS though)
<krl>
I would like to take next week to properly package the ipfs-api webcomponent, and make it part of the build-step of node-ipfs-api (maybe rename it too?). Things left to be done for starlog is editing entries.
<daviddias>
lgierth: since you are on top of this, can we also have the issue open for next sprint, so we can fill it in during discussions today
<krl>
have been bitten by untested edge cases in the append only log, and spent quite some time fixing that and covering all things we need
Encrypt has quit [Quit: Quitte]
<lgierth>
daviddias: yep will take care of it in a bit
<lgierth>
krl: i'll get cors on the gateway fixed tonight, i'll be at c-base from 9-ish
caseorga_ has quit [Remote host closed the connection]
<lgierth>
krl: cool, anything else?
<krl>
not reallys
<lgierth>
whyrusleeping: you wanna go next\
<lgierth>
ok, i'll go next cause i need to move in a few minutes
<lgierth>
cp inc
<lgierth>
hosts
<lgierth>
- applied proper naming to the utility hosts scruffy and willie, they're now deimos and phobos
<lgierth>
- built out the storage hosts sirius and pollux
<lgierth>
- gave davidar a login for sirius and pollux
<lgierth>
- mirrored downloads.openwrt.org to pollux
<lgierth>
- investigating ipfs add bottlenecks
<lgierth>
gateways
<lgierth>
- still using a self-signed cert, waiting for startssl.com's process
<lgierth>
- made IPNSHostname work (kind of), for self-hosting ipfs.io
<lgierth>
- needs a bit more work regarding stylesheets in directory listings
<lgierth>
- ongoing fixes to the readonly api, we need CORS there
<lgierth>
- erased secrets from the infrastructure.git history (i.e. rewrite history)
<lgierth>
not-done
<lgierth>
- dist.ipfs.io deployment
<lgierth>
- gateway-dmca-denylist
<lgierth>
- discovery.cjdns
<lgierth>
general things
<lgierth>
- picked up sprint sync
<lgierth>
- badly need to increase infrastructure bus factor, i'm the only one that can deploy at the moment and that sucks. it makes me a bottleneck, and keeps me from working on go-ipfs and cjdns stuff
<lgierth>
this week i've been blocking 4 people including myself ;)
<lgierth>
jbenet: i'm gonna add highlights in front of my lines
<chriscool>
It says: "Git comes with a "curl" family of remote helpers, that handle various transport protocols, such as git-remote-http, git-remote-https, git-remote-ftp and git-remote-ftps. They implement the capabilities fetch, option, and push."
<jbenet>
lgierth: i have 10min because i have an impt call at 21:30 (unless other person is late)
<lgierth>
ok
infinity0 has quit [Remote host closed the connection]
infinity0 has joined #ipfs
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
Eudaimonstro has quit [Ping timeout: 265 seconds]
caseorga_ has joined #ipfs
patcon has joined #ipfs
caseorga_ has quit [Ping timeout: 256 seconds]
notduncansmith has joined #ipfs
notduncansmith has quit [Read error: Connection reset by peer]
<jbenet>
ok im back
<jbenet>
lgierth: i can talk more about infra if there's other things? will be chatting with krl about apps soon, and im sure daviddias wants to talk about node-ipfs / libp2p things, and so on.
<jbenet>
whyrusleeping: are you around for discussions?
<jbenet>
chriscool if you have a bit of time would be great to discuss testing infrastructure -- we've some things we'd love to be able to do, and we have some machines we can use to do it. (maybe i should write a github issue about it first)
<lgierth>
jbenet: that was about it re: infrastructure, the rest can be handled in gh issues
<lgierth>
i'd like to chime in re: libp2p
<chriscool>
I was going to eat right now, but it can wait a bit
<krl>
jbenet: o/
caseorga_ has joined #ipfs
<jbenet>
chriscool: no no go ahead, chat later
<jbenet>
(or on github another time)
caseorga_ has quit [Remote host closed the connection]
<jbenet>
krl: ok sweet, same question as to lgierth-- want to discuss via text? (either here in chat, or a hangout?)
<chriscool>
ok thanks
<krl>
i'm fine either way, slight text preference
<daviddias>
I'm here and available with good Internet, just let me know when you are ready :)
<jbenet>
krl: have a link for the blog? would love to see it.