pomegranatedaddy has quit [Ping timeout: 240 seconds]
jonnycrunch has joined #ipfs
gkbrk has joined #ipfs
upperdeck has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
gts has joined #ipfs
espadrine has quit [Ping timeout: 264 seconds]
radiogenic has quit [Ping timeout: 252 seconds]
gts has quit [Ping timeout: 258 seconds]
upperdeck has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
gts has joined #ipfs
gts has quit [Ping timeout: 255 seconds]
cris_thor` has quit []
Alpha64 has joined #ipfs
upperdeck has quit [Ping timeout: 240 seconds]
Tootoot222 has joined #ipfs
upperdeck has joined #ipfs
gts has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
fluffypurplebunn has quit [Ping timeout: 255 seconds]
gts has quit [Ping timeout: 252 seconds]
upperdeck has quit [Ping timeout: 246 seconds]
upperdeck has joined #ipfs
gts has joined #ipfs
upperdeck has quit [Ping timeout: 240 seconds]
gts has quit [Ping timeout: 246 seconds]
upperdeck has joined #ipfs
bdTaylor has joined #ipfs
dconroy has joined #ipfs
cris_thor has joined #ipfs
cris_thor is now known as cris
gkbrk has quit [Quit: Leaving]
<dryajov>
pvh: thats right, but there is also https://github.com/libp2p/js-libp2p-webrtc-star but requires a signaling server, this has the advantage that it will allow establishing browser<->signlnig server<->browser connections...
ircler has joined #ipfs
upperdeck has quit [Ping timeout: 240 seconds]
gts has joined #ipfs
upperdeck has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
Jesin has joined #ipfs
kaotisk has quit [Read error: Connection reset by peer]
kaotisk has joined #ipfs
ircler has quit [Read error: No route to host]
dconroy has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
upperdeck has quit [Ping timeout: 248 seconds]
dimitarvp` has quit [Quit: Bye]
ircler has joined #ipfs
upperdeck has joined #ipfs
_whitelogger has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
gts has joined #ipfs
matoro has quit [Ping timeout: 255 seconds]
xmammut492 has joined #ipfs
ircler_ has joined #ipfs
gts has quit [Ping timeout: 246 seconds]
ircler has quit [Ping timeout: 248 seconds]
ircler has joined #ipfs
ircler_ has quit [Ping timeout: 255 seconds]
gts has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
qgnox has joined #ipfs
<pvh>
dryajov: i like webrtc-direct because it makes the nodes symmetrical but i'd really hoped there would be a way to carry the signalling traffic on the DHT somehow
<pvh>
i suppose that's what you're working on with circuit?
dconroy has joined #ipfs
trn has quit [Remote host closed the connection]
dconroy has quit [Client Quit]
matoro has joined #ipfs
upperdeck has quit [Ping timeout: 260 seconds]
upperdeck has joined #ipfs
gts has joined #ipfs
gts has quit [Ping timeout: 258 seconds]
Alpha64_ has joined #ipfs
Alpha64 has quit [Ping timeout: 248 seconds]
Alpha64_ has quit [Ping timeout: 248 seconds]
trn has joined #ipfs
upperdeck has quit [Ping timeout: 246 seconds]
upperdeck has joined #ipfs
Alpha64 has joined #ipfs
lanzafame has joined #ipfs
lanzafame has quit [Client Quit]
lanzafame has joined #ipfs
gts has joined #ipfs
gts has quit [Ping timeout: 258 seconds]
Mateon2 has quit [Ping timeout: 240 seconds]
r3c0d3x has quit [Ping timeout: 255 seconds]
Choyes[m] has joined #ipfs
gts has joined #ipfs
r3c0d3x has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
e0f has quit [Ping timeout: 246 seconds]
Aranjedeath has quit [Quit: Three sheets to the wind]
e0f has joined #ipfs
e0f has quit [Changing host]
e0f has joined #ipfs
qgnox has quit [Quit: WeeChat 1.9.1]
_whitelogger has joined #ipfs
gts has joined #ipfs
gts has quit [Ping timeout: 255 seconds]
ircler has quit [Quit: My Mac Pro has gone to sleep. ZZZzzz…]
gts has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
OneBigMes has joined #ipfs
gts has joined #ipfs
qgnox has joined #ipfs
ccii1 has joined #ipfs
upperdeck has quit [Ping timeout: 260 seconds]
ccii has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 255 seconds]
upperdeck has joined #ipfs
gts has quit [Remote host closed the connection]
gts has joined #ipfs
onikuma has joined #ipfs
gts has quit [Ping timeout: 240 seconds]
jaboja has quit [Remote host closed the connection]
kaotisk has quit [Read error: Connection reset by peer]
kaotisk has joined #ipfs
upperdeck has joined #ipfs
mentos1386 has quit [Read error: Connection reset by peer]
mentos1386 has joined #ipfs
justintv90 has joined #ipfs
AgenttiX has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
nikivi has joined #ipfs
arpu_ has joined #ipfs
mentos1386 has quit [Quit: mentos1386]
arpu has quit [Ping timeout: 248 seconds]
mentos1386 has joined #ipfs
mentos1386 has quit [Remote host closed the connection]
mentos1386 has joined #ipfs
mentos1386 has quit [Client Quit]
gts has quit [Remote host closed the connection]
upperdeck has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
justintv90 has quit [Ping timeout: 240 seconds]
upperdeck has quit [Ping timeout: 258 seconds]
mentos1386 has joined #ipfs
upperdeck has joined #ipfs
gts has joined #ipfs
_whitelogger has joined #ipfs
maxlath has quit [Ping timeout: 240 seconds]
maxlath has joined #ipfs
erictapen has joined #ipfs
erictapen has quit [Remote host closed the connection]
erictapen has joined #ipfs
erictapen has quit [Remote host closed the connection]
erictapen has joined #ipfs
gts has quit [Remote host closed the connection]
gts has joined #ipfs
gts has quit [Remote host closed the connection]
gts has joined #ipfs
gts has quit [Remote host closed the connection]
upperdeck has quit [Ping timeout: 240 seconds]
justintv90 has joined #ipfs
upperdeck has joined #ipfs
bdTaylor has quit [Ping timeout: 248 seconds]
gts has joined #ipfs
upperdeck has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
bdTaylor has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
bdTaylor has quit [Ping timeout: 248 seconds]
bdTaylor has joined #ipfs
erictapen has quit [Ping timeout: 240 seconds]
gts has joined #ipfs
<whyrusleeping>
redfish: implementing logic to 'limit' memory usage would end up just being us recommend people use systemd to limit the proceses memory usage
mentos1386 has quit [Quit: mentos1386]
gts has quit [Ping timeout: 252 seconds]
<whyrusleeping>
the only thing we can do is address the causes of the excess memory usage
<whyrusleeping>
which we are working on, pretty much as fast as we can
<r0kk3rz>
whyrusleeping: whats the current plan of attack ooi?
<whyrusleeping>
well, we've got connection closing implemented and merged into master
<whyrusleeping>
and i'm working on figuring out sane defaults so that older nodes don't have to manually modify their config
<whyrusleeping>
then we also recently identified a memory leak in the dht, this was fixed and shipped in 0.4.11
erictapen has joined #ipfs
erictapen has quit [Remote host closed the connection]
erictapen has joined #ipfs
erictapen has quit [Remote host closed the connection]
<whyrusleeping>
and now i'm looking at some extra usage by the peerstore
mrjn has joined #ipfs
erictapen has joined #ipfs
<whyrusleeping>
its not exactly a leak, but it needs to get put on disk or something
upperdeck has quit [Ping timeout: 260 seconds]
<whyrusleeping>
past that, we need to address memory spikes caused by connection notification backup.
<whyrusleeping>
this ones really tricky, and requires a lot of deep changes in go-libp2p
upperdeck has joined #ipfs
maxlath has quit [Ping timeout: 248 seconds]
<r0kk3rz>
sounds cool, is there anything major scaling wise planned? or just optimising the various parts
upperdeck has quit [Ping timeout: 255 seconds]
<whyrusleeping>
just optimizing the various parts. beyond these memory issues and connection closing, not much should get in the way of scaling the network
jaboja has joined #ipfs
upperdeck has joined #ipfs
<r0kk3rz>
i guess time will tell, onwards to 10k nodes!
dimitarvp has joined #ipfs
fredthomsen has quit [Ping timeout: 248 seconds]
gts has joined #ipfs
gts has quit [Ping timeout: 258 seconds]
<whyrusleeping>
one of our bootstrappers has seen 40k nodes in the past 20 hours
<r0kk3rz>
thats a tad insane, wasnt it 4k only a week ago?
<r0kk3rz>
or was the true number being masked by the connection closing issue
<whyrusleeping>
i think the true number was being masked yeah
<whyrusleeping>
and that 40k counts ephemeral nodes
<whyrusleeping>
i have no idea what the total 'nodes running right now' is
<r0kk3rz>
for sure, but its the best indication of network size we have. which wont count anyone with their own bootstrap nodes or private networks .etc
<whyrusleeping>
right
nocent has joined #ipfs
rcat has joined #ipfs
mrjn has quit [Quit: Page closed]
gmoro has joined #ipfs
maxlath has joined #ipfs
jaboja has quit [Ping timeout: 255 seconds]
erictapen has quit [Read error: No route to host]
<voker57>
Error: write /home/ipfs/.ipfs/datastore/009215.ldb: file too large
<voker57>
is my DB corrupted?
gts has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
lanzafame has quit [Quit: lanzafame]
espadrine has quit [Ping timeout: 258 seconds]
santamanno has quit [Ping timeout: 248 seconds]
gts has joined #ipfs
lanzafame has joined #ipfs
dconroy has joined #ipfs
dconroy has quit [Client Quit]
lanzafame has quit [Quit: lanzafame]
lanzafame has joined #ipfs
lanzafame has quit [Client Quit]
lanzafame has joined #ipfs
lanzafame has quit [Client Quit]
lanzafame has joined #ipfs
lanzafame has quit [Client Quit]
lanzafame has joined #ipfs
lanzafame has quit [Client Quit]
lanzafame has joined #ipfs
lanzafame has quit [Client Quit]
gts has quit [Remote host closed the connection]
gts has joined #ipfs
jaboja has joined #ipfs
erictapen has joined #ipfs
erictapen has joined #ipfs
gts has quit [Remote host closed the connection]
LeerSpace has joined #ipfs
pvh has quit [Quit: Connection closed for inactivity]
gts has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
gts has quit [Ping timeout: 258 seconds]
upperdeck has joined #ipfs
}ls{ has joined #ipfs
rcat has quit [Ping timeout: 255 seconds]
<LeerSpace>
Is there a preferred numbering scheme (t****) to use when creating a new sharness test in go-ipfs/test/sharness/? Or just use a number that's not taken?
<whyrusleeping>
LeerSpace: yeah, use a number thats not taken, try to pick a number close to one that relates to your test
<whyrusleeping>
or add your test into another existing test that is testing the same command
erictapen has quit [Read error: Connection reset by peer]
<whyrusleeping>
LeerSpace: in your case, one of the t008* might fit
<LeerSpace>
whyrusleeping: great; thanks!
gts has joined #ipfs
ircler has joined #ipfs
gts has quit [Ping timeout: 246 seconds]
<voker57>
yeah, repaired it with python module.
flypigcow[m] has left #ipfs ["User left"]
<voker57>
hmm, i'm getting this error again with flatfs
jaboja has quit [Ping timeout: 240 seconds]
<voker57>
Error: write /home/ipfs/.ipfs/blocks/QS/put-846948524: file too large
<whyrusleeping>
voker57: thats one i've never seen before
<voker57>
32-bit pitfalls?
<whyrusleeping>
LeerSpace: while you're at it, could you make sure to run gofmt on your code?
<LeerSpace>
whyrusleeping: sure thing
erictapen has joined #ipfs
erictapen has quit [Remote host closed the connection]
erictapen has joined #ipfs
PyHedgehog has joined #ipfs
anagh has joined #ipfs
gts has joined #ipfs
<anagh>
Can someone help me. How do i connect multiple ipfs nodes running on different vms in cloud
gts has quit [Remote host closed the connection]
gts has joined #ipfs
<whyrusleeping>
anagh: use `ipfs id` on one of them to get its multiaddr
<whyrusleeping>
then run `ipfs swarm connect <multiaddr>` on the other to connect to it
<whyrusleeping>
you can use `ipfs ping <peerID>` to verify connectivity
ircler has quit [Quit: My Mac Pro has gone to sleep. ZZZzzz…]
<anagh>
@whyrusleeping ok thank you. But how do i know which is the master node?
<anagh>
Is there anything like genesis in ipfs ?
<whyrusleeping>
master node?
<whyrusleeping>
you mean like a genesis block? ipfs isnt a blockchain
<anagh>
whyrusleeping yes. So suppose if i want to acess the data that i have uploaded earlier
<anagh>
Which nodes ip should i need to use ?
<whyrusleeping>
you just request the data
<whyrusleeping>
your node will find who has it, and request it from them automatically
ircler has joined #ipfs
<anagh>
ok the reason i am asking is because there 2 vms which has different ip and i am running 1 node in each of them. So i am confused. I can request any of this node for the data ?
<whyrusleeping>
yeah
xnbya has quit [Ping timeout: 248 seconds]
<anagh>
whyrusleeping thank you so much for the help
<whyrusleeping>
not a problem
<whyrusleeping>
anagh: let us know if you run into any issues
<whyrusleeping>
voker57: sorry, when do you get that issue?
<whyrusleeping>
is it when you start up the node?
<whyrusleeping>
are you out of disk space?
xnbya has joined #ipfs
<anagh>
whyrusleeping sure. I will get back to this channel if i face any issues
gts has quit [Remote host closed the connection]
ircler has quit [Quit: My Mac Pro has gone to sleep. ZZZzzz…]
<voker57>
whyrusleeping: I ran out of disk space, then I killed the node, restarted it and now getting these issues
<whyrusleeping>
voker57: hrm... thats odd
<voker57>
first error was on daemon start, second on pin rm
<whyrusleeping>
is it always the same filename?
droman has joined #ipfs
<voker57>
no, different
<voker57>
ipfs add fails now too
kirby__ has joined #ipfs
<voker57>
not sure if it worked pre-fulldisk
<voker57>
also I have 3.8gb in blocks despite having 1gb at most pinned content (and ~5g filestored)
gts has joined #ipfs
eater has quit [Ping timeout: 252 seconds]
eater has joined #ipfs
gts has quit [Ping timeout: 258 seconds]
joocain2_ has quit [Write error: Connection reset by peer]
upperdeck has quit [Ping timeout: 258 seconds]
joocain2 has joined #ipfs
eater has quit [Ping timeout: 260 seconds]
upperdeck has joined #ipfs
ircler has joined #ipfs
eater has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
eater has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
gts has joined #ipfs
eater has joined #ipfs
kaotisk-irc has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
kaotisk has quit [Ping timeout: 240 seconds]
ircler has quit [Quit: My Mac Pro has gone to sleep. ZZZzzz…]
ircler has joined #ipfs
upperdeck has quit [Ping timeout: 252 seconds]
upperdeck has joined #ipfs
cmbrnt has quit [Remote host closed the connection]
erictapen has quit [Read error: Connection reset by peer]
gts has joined #ipfs
ccii1 has quit [Ping timeout: 255 seconds]
gts has quit [Ping timeout: 252 seconds]
KrisJE has joined #ipfs
KrisJE has quit [Client Quit]
PyHedgehog has quit [Quit: Connection closed for inactivity]
ccii has joined #ipfs
erictapen has joined #ipfs
espadrine has joined #ipfs
gts has joined #ipfs
upperdeck has quit [Ping timeout: 258 seconds]
upperdeck has joined #ipfs
gts has quit [Ping timeout: 255 seconds]
ircler_ has quit [Quit: ZZZzzz…]
<voker57>
whyrusleeping: already ran, it finishes fine
<whyrusleeping>
and then after that, you still get those issues?
<whyrusleeping>
the leveldb one i can understand
<whyrusleeping>
but getting it for flatfs makes me think you have some sort of deeper filesystem issue
<voker57>
yes
ircler has joined #ipfs
gde33 has joined #ipfs
tobiasBora has joined #ipfs
<tobiasBora>
Hello,
<tobiasBora>
I just uploaded a file, I can access it locally, but when I try from another server, it fails
<whyrusleeping>
tobiasBora: is your daemon running?
<voker57>
if yes, are any external IPs in the output of `ipfs id`?
<tobiasBora>
whyrusleeping: when the daemon was not running, I got an error about the merkeldeg, and with the daemon, it just hang forever
<whyrusleeping>
tobiasBora: so you have two ipfs nodes, running in different places, and you added a file on one, and are trying to fetch it on another?
<tobiasBora>
voker57: I think that some ip are external yes
<tobiasBora>
whyrusleeping: exactly
<tobiasBora>
I push on my local machine, and try to fetch it from my ssh server
<whyrusleeping>
tobiasBora: okay, and *both* daemons are running?
gts has joined #ipfs
<tobiasBora>
whyrusleeping: yes
<tobiasBora>
maybe it can help, can you reach this file: ipfs cat QmdjUWtxTDXbrogzai37phs7i3MY3w1htrBW7nngavpPGi/qemu_packed.tar.gz ?
upperdeck has quit [Ping timeout: 258 seconds]
upperdeck has joined #ipfs
<whyrusleeping>
tobiasBora: yeah, i can see that file on a public ipfs node
<whyrusleeping>
it could be NAT traversal issues
<tobiasBora>
Hum, now it's working, I don't understand...
<whyrusleeping>
its likely working now because the node i just checked on now has the data
<tobiasBora>
the first time it spent 30s to download, and now I get it in 1s
<whyrusleeping>
and your server was connected to my node
<whyrusleeping>
but your server likely couldnt get a connection to your local machine
<tobiasBora>
hum...
gts has quit [Ping timeout: 252 seconds]
<tobiasBora>
Just a question, the file are supposed to be hosted on our personnal computers, or it's supposed to be shared between servers, so that I can shutdown my computer ?
<whyrusleeping>
its hosted on whatever computers fetch it or add it
<whyrusleeping>
if you fetch it to your server, then your server is hosting it too, and you could shut down your local machine
<tobiasBora>
so basically, if no node wants to host me... I can't do anything right ?
<whyrusleeping>
right
<tobiasBora>
ok
<whyrusleeping>
its like torrents in that regard
jaboja has joined #ipfs
<tobiasBora>
and how could I check if a server accepted me ?
<whyrusleeping>
you cant reliably know for certain, but you can check `ipfs dht findprovs <hash>` to get a listing of all nodes that claim to have a given hash
<tobiasBora>
whyrusleeping: the result I get is a list of 3 hash, so it means that 3 servers hosts me ?
<whyrusleeping>
yeap, thats likely your local machine, your server, and the machine that I used to check for your content earlier
<whyrusleeping>
you can check your nodes `ipfs id` and see if it matches one from that list
<tobiasBora>
Ok thanks ! and when the reference is a folder, they host the whole folder or they can choose only one file in the folder ?
<whyrusleeping>
when the reference is the folder, they might not necessarily be hosting the content
<tobiasBora>
ok thanks
<tobiasBora>
and your server got my file because he is nice, or because you ask him to do so ?
LeerSpace has quit [Quit: Page closed]
gkbrk has joined #ipfs
OneBigMes has joined #ipfs
<redfish>
whyrusleeping: limiting externally (via systemd) would be sensible -- as long as the app logic is aware of that. Normally, I'd expect apps to fail completely if malloc fails (primarily, because there are optional vs. vital allocations that the app needs to do without which it just can't run)
<redfish>
if your app gracefully tolerates failure in any alloc (so failure of any library call anywhere, pretty much), then external limiting seems sensible
<whyrusleeping>
right, and go programs will panic (SIGQUIT) if they run out of memory
<redfish>
yeah, your comment about keeping things on disk vs in-mem is what i mostly had in mind
<redfish>
it's easier to keep everything in memory, but then you have to expand effort to actually introduce disk+memory, with all the implications on keeping the disk structures consistent etc
<whyrusleeping>
yeap
<whyrusleeping>
especially when the in memory thing gets updated *frequently*
<whyrusleeping>
it becomes quite the effort to shove that thing to disk
<redfish>
yeah, hopefully leveldb (or lmdb) would handle much.
<whyrusleeping>
yeah... leveldb is okay at this
<redfish>
and "outsource" the paging to the OS. (that's lmdb's design principle -- very nice)
<whyrusleeping>
but you really want to do some application level coalescing of things
<redfish>
makes sense. some app awareness might be unavoidable.
<whyrusleeping>
its not so much the cost of disk writes that is the issue, its deciding when to serialize data
<redfish>
yeah, so about serialization. the nice appraoch in lmdb: the mmap is at a fixed address, so you can actually store pointer structures in the DB. No marshaling overhead. It's so simple, yet so powerful.
<Kythyria[m]>
That's portable?
<redfish>
Kythyria[m]: you don't need the DB to be portable in this use case..
}ls{ has quit [Ping timeout: 240 seconds]
<whyrusleeping>
redfish: that wouldnt work at all in cases where the application has to restart
<Kythyria[m]>
Heck, it maps things into memory at precisely consistent locations and never moves them? That's even more surprising.
<Kythyria[m]>
I mean, it's surprising you can rely on mmap to actually put things where you asked and the OS to not put something in the way.
<redfish>
Kythyria[m]: there's a knob the OS gives you precisely for this control
<whyrusleeping>
Kythyria[m]: the address space is entirely virtualized
<Kythyria[m]>
I'm still surprised you can rely on every OS lmdb runs on to never map something over the top of the address space you'll need before you get there.
<redfish>
whyrusleeping: sure, heaps won't be compatible with this. this is for lower level approach to allocating the objects in the first place
<redfish>
Kythyria[m]: you bring up portability? *one* OS won't change such asspects about the processe's addr space
gkbrk has quit [Read error: Connection reset by peer]
<Kythyria[m]>
Which one is that?
ccii has quit [Ping timeout: 240 seconds]
<redfish>
i mean any *one* OS. mem regions won't change between runs of the same process -- you can certainly control that with the linking script.
<Kythyria[m]>
Ah, so it requires you disable ASLR.
}ls{ has joined #ipfs
<redfish>
i would bet you disable security obfuscation for specific elf sections
<redfish>
otherwise, it would be impossible to use memory-mapped hardware registers
<Kythyria[m]>
Does that let you reserve an area of address space to ensure nothing gets mapped there until you ask?
<redfish>
you can specify that you want the elf section at addr X in the elf binary to be loaded into memory at addr Y
<Kythyria[m]>
And mmap is fine with clobbering that?
<redfish>
but to get a fixed-addr mmap, i don't think you even need to reserve anything. there are probably well-defined ranges for what can get mapped where on an OS. Like libs will go into 0x4... on 32-bit, they won't end up where your stack is, ever. even with obfuscation, etc.
gts has joined #ipfs
<redfish>
Kythyria[m]: well, i think mmap will fail if you try to map into an already allocated region, particulary if that reagion is RO. But, I'm not sure about this level of detail. So, don't listen to me too much.
<Kythyria[m]>
Also note that it's explicitly stated to use copy-on-write, so your data will move anyway!
<tobiasBora>
grrr... whyrusleeping, did you remove my content from your node ? I can't access it anymore
<whyrusleeping>
tobiasBora: it might have run a garbage collection
<redfish>
whyrusleeping: actually, heaps can work too. You just need to make sure the heap implemetation loads its volatile state from the same blob (e.g. the first block in the addr range) on process startup
<whyrusleeping>
tobiasBora: you can make a connection between your nodes manually
<tobiasBora>
whyrusleeping: how ?
<whyrusleeping>
on your local machine do `ipfs swarm connect <multiaddr of your server>`
<whyrusleeping>
you can get your server multiaddrs from running `ipfs id` on your server
gts has quit [Ping timeout: 258 seconds]
<tobiasBora>
hum I don't understand anything, now even ipfs id does not return
<whyrusleeping>
huh?
<whyrusleeping>
thats not supposed to happen
<whyrusleeping>
what version of ipfs are you using?
<tobiasBora>
whyrusleeping: 0.4.11. But I restarted the daemon and now it "id" returns !
<whyrusleeping>
tobiasBora: if that happens again, where `ipfs id` hangs, i would love to get a stack trace of your daemon (which you can get via curl localhost:5001/debug/pprof/goroutine\?debug=2 > goroutines.txt)
<redfish>
Kythyria[m]: isn't that function totally optional? like, if you want to remap something. Btw, we should ask hyc if he has an example of fixed-address use case.
<tobiasBora>
whyrusleeping: ok, no problem
<redfish>
Kythyria[m]: what i'm confused about is: lmdb claims to have a zero-copy API, but it can't be zero-copy on write, can it?
<Kythyria[m]>
`mdb_env_open`'s arguments imply it doesn't mmap at a fixed location by default, and indeed it says of the option to do so, "This option may not always work, depending on how the operating system has allocated memory to shared libraries and other uses. The feature is highly experimental. "
<redfish>
always doesn't refer to different runs of the process, though, of course.
<tobiasBora>
grrr, strange thing again, /ipfs dht findprovs <hash> does not return anything, but ipfs cat <hash> get me the result...
<redfish>
if you do get it to work once on your platform, it will work there, period.
<Kythyria[m]>
Um, no?
fredthomsen has joined #ipfs
onikuma has quit [Ping timeout: 240 seconds]
<Kythyria[m]>
So, no matter how non-deterministic the code preceding `mdb_env_open()` is, the OS will _never_, **ever** map anything into the space I plan to use?
<redfish>
Kythyria[m]: sure, i see your point. no, it has to be deterministic, for sure. we're talking much lower level programming than usual here.
<Kythyria[m]>
And modern dynamic loaders aren't deterministic.
<Kythyria[m]>
On purpose.
<redfish>
but, you have control over much of the loader's behavior
<redfish>
you can load the stuff yourself, at the end of the day.
<Kythyria[m]>
You have to completely disable ASLR for the process.
<redfish>
ASLR sounds like a very vague term. It highly depends what the ASLR actually rearranges. It doesn't ever flip heap with stack. If it does, I be damned.
<redfish>
addr spaces have structure.
<redfish>
it's not a free-for-all
<Icefoz_>
Hence "layout" randomization.
<Icefoz_>
afaik it shuffles where the bits of the structure end up.
<Icefoz_>
you could end up with the heap above the stack but the stack will still grow downwards.
<redfish>
under contstraints. not unconstrained shuffling of anything to anywhere.
kirby__ has quit [Remote host closed the connection]
<Icefoz_>
I dunno! Why couldn't it?
LoveCoinz has joined #ipfs
<Icefoz_>
I don't know much about ASLR, so.
kirby__ has joined #ipfs
<Kythyria[m]>
"When Randomized mmap() base is enabled, PaX randomly shifts the mmap() base, affecting the positioning of all libraries and other non-specific mmap() calls. This causes all dynamically linked code, i.e. shared objects, to be mapped at a different, randomly selected offset every time."
<redfish>
i bet it does doesn't in practice. that's just not what the implementation does. not because it's fundamentally inconceivable to shuffle everything in the addr space.
LoveCoinz has quit [K-Lined]
OneBigMes has quit [Ping timeout: 240 seconds]
<redfish>
randomly seelcted offset from what range?
<redfish>
what is Pax? what is randomized mmap?
merc88 has joined #ipfs
gts has joined #ipfs
<redfish>
this is all very platform specific. my point is very simple. there are ways to implement persistent storage in your application that has consistency and durability guarantees without marshalling overhead.
<redfish>
if you have to reconfigure the linker, re-write your own linker, reconfigure Go's heap, not use Go's GC, etc etc then so be it.
<Kythyria[m]>
And that's very different from "it will work, period"
merc88 has left #ipfs [#ipfs]
Alpha64 has joined #ipfs
<redfish>
no it is not. once you do what is needed to be done to create the setup, the setup will work.
<redfish>
*on every run of the process. otherwise, you having setup things is a meaningless.
<tobiasBora>
Is it possible to ask to some servers to pin my content ? Because my ssh server won't be always available...
<Icefoz_>
tobiasBora: I am making a program for that right now. ;-)
<Kythyria[m]>
redfish: The thing is, it'll look like it worked even if you didn't actually do all those things.
gts has quit [Ping timeout: 252 seconds]
<tobiasBora>
Icefoz_: great ;-) so now if I want to ask to someone to pin my content, I can only ask on this channel ? :P
<Kythyria[m]>
But actually it's non-deterministic and will only sometimes work.
<redfish>
that would not qualify "i figured out a way to store my objects in raw form in a paged DB". Whatever. I'm not here to convince or push anything. If you think you know of a faster way to achieve persistent, durable, consistent, memory-backed storage for app data, just do it.
upperdeck has quit [Ping timeout: 248 seconds]
ccii has joined #ipfs
OneBigMes has joined #ipfs
upperdeck has joined #ipfs
jaboja has quit [Ping timeout: 240 seconds]
gts has joined #ipfs
Teeed has quit [Ping timeout: 258 seconds]
gts has quit [Ping timeout: 246 seconds]
rendar has quit [Quit: std::lower_bound + std::less_equal *works* with a vector without duplicates!]
<Kythyria[m]>
Faster, maybe not. More reliable without a lot of extremely platform-specific fiddling? Practically everything.
Teeed has joined #ipfs
<whyrusleeping>
tobiasBora: you can ask, and depending on what the content is, some people here might decide to help you out
maxlath has quit [Ping timeout: 248 seconds]
<tobiasBora>
ok ^^
jaboja has joined #ipfs
gts has joined #ipfs
jaboja has quit [Ping timeout: 240 seconds]
koshii has quit [Ping timeout: 240 seconds]
koshii has joined #ipfs
<whyrusleeping>
tobiasBora: I just wrote up a little doc based on me helping you troubleshoot just now, mind reviewing it? https://github.com/ipfs/go-ipfs/pull/4332
gts has quit [Ping timeout: 255 seconds]
<whyrusleeping>
also anyone else who wants to be helpful, please let me know if theres anything i can change or add in that
upperdeck has quit [Ping timeout: 264 seconds]
upperdeck has joined #ipfs
ircler has quit [Quit: bye]
upperdeck has quit [Ping timeout: 255 seconds]
<tobiasBora>
whyrusleeping: great job ;-) Seems perfect I guess... Maybe just adding a line saying that the file has no reason to be hosted by others hosts if they don't manually accept it(at the beginning I thought that my file was spread on the whole network)
upperdeck has joined #ipfs
aw1n has joined #ipfs
upperdeck has quit [Ping timeout: 255 seconds]
upperdeck has joined #ipfs
maxlath has joined #ipfs
upperdeck has quit [Ping timeout: 258 seconds]
upperdeck has joined #ipfs
gts has joined #ipfs
erictapen has quit [Ping timeout: 240 seconds]
dconroy has joined #ipfs
jaboja has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
gts has quit [Ping timeout: 246 seconds]
erictapen has joined #ipfs
upperdeck has joined #ipfs
aw1n has left #ipfs ["Leaving"]
upperdeck has quit [Ping timeout: 260 seconds]
jaboja has quit [Ping timeout: 255 seconds]
upperdeck has joined #ipfs
<redfish>
Kythyria[m]: your use of "reliable" is misleading. There is nothing less reliable about a correctly implemented fixed-addr approach relative to any other approach.
ygrek_ has joined #ipfs
upperdeck has quit [Ping timeout: 255 seconds]
droman has quit [Quit: WeeChat 1.9.1]
guideline has quit [Ping timeout: 252 seconds]
OneBigMes has quit [Remote host closed the connection]
upperdeck has joined #ipfs
gts has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
upperdeck has quit [Ping timeout: 252 seconds]
upperdeck has joined #ipfs
<Kythyria[m]>
The difficulty is in correctly implementing it. Which you are going to have to do again for every platform. And cannot be packaged as a straightforward library: it heavily affects all code that precedes the mapping, and requires changes to how you build code that is related only by being in the same process. This is probably what the pointer fixup callback is for.
<Kythyria[m]>
So that you can fix the pointers up rather than try to pin them in the same place.
upperdeck has quit [Ping timeout: 248 seconds]
guideline has joined #ipfs
justintv90 has quit [Ping timeout: 264 seconds]
<Kythyria[m]>
Oh, and callback implemented or not you still have to deal with stuff changing position if it gets bigger or possibly is overwritten at all.
upperdeck has joined #ipfs
aw1n has joined #ipfs
<Kythyria[m]>
(and "every platform" may include different OS versions... and are you sure your libc is deterministic about allocation?)
<str1ngs>
I need to check file hashes without execing ipfs
gts has joined #ipfs
upperdeck has joined #ipfs
<lgierth>
str1ngs: you can use `ipfs add -n` without running the daemon
<nothingmuch>
str1ngs: i think hashes would be for the IPLD objects not for the contents as is, in most circumstances
<nothingmuch>
(due to chunking)
gts has quit [Ping timeout: 252 seconds]
upperdeck has quit [Ping timeout: 252 seconds]
<str1ngs>
ipfs add means I need to exec ipfs
<str1ngs>
but I think I can do this with repo.Mock
upperdeck has joined #ipfs
<str1ngs>
useing NilRepo
<str1ngs>
acutally i dont need repo.Mock either just use BuildCfg with NilRepo.. I think
Encrypt has joined #ipfs
<lgierth>
no
<lgierth>
try it
<lgierth>
ipfs add -n yourfile
<str1ngs>
yes but I need the ipfs binary to do that
<lgierth>
i have a year or three of experience with this ;)
<lgierth>
ah got it
<lgierth>
i'm afraid there's not really any other tool right now
<str1ngs>
-n uses NilRepo anyways
upperdeck has quit [Ping timeout: 240 seconds]
kaotisk-irc has joined #ipfs
<str1ngs>
I think I'll just use the webapi
<str1ngs>
it has a only-hash option
kaotisk has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
<lgierth>
yeah -n is the equivalent of only-hash
<lgierth>
ipfs add --only-hash and -n are the same
<str1ngs>
shell.AddNoPin is not quite the same as only-hash though right?
larpanet has joined #ipfs
<lgierth>
yeah not the same, that just adds without pinning
<str1ngs>
I know this seems anal retentive, but my program is very bare metal. it's quit possible that ipfs might not be running at all. in fact it's better if it's not for end users.
<ansuz>
hi lgierth
upperdeck has quit [Ping timeout: 240 seconds]
dconroy has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<lgierth>
hi ansuz
upperdeck has joined #ipfs
dconroy has joined #ipfs
dconroy has quit [Read error: Connection reset by peer]
<Alpha64>
looks easy, could it be that two peers report different sizes for a hash ?
<Alpha64>
is that kind of conflict resolved ?
<str1ngs>
I would think not
<Icefoz_>
Huh, I can't find any way to do it from command line.
<str1ngs>
size is recorded as a go unit64 IIRC
upperdeck has quit [Ping timeout: 260 seconds]
aw1n has joined #ipfs
<str1ngs>
err uint64
<Icefoz_>
If two peers give different sizes for a hash then I would imagine one of them is broken.
tskrrr has joined #ipfs
upperdeck has joined #ipfs
<Alpha64>
i'm designing a distributed system and i want to prevent that attck
<Alpha64>
peers reporting random sizes for a file
<Alpha64>
just a hunch
<Alpha64>
maybe the protocol takes care of that
<Icefoz_>
As far as I know (which isn't much) there's not really any defense against malicious peers, but they can't do anything that anyone talking to them can't trivially verify and find incorrect.
<Icefoz_>
You should wait for someone who knows what they're talking about to show up though. :-)
tskrrr has quit [Client Quit]
aw1n has quit [Client Quit]
aw1n_ has quit [Quit: Leaving]
aw1n has joined #ipfs
aw1nn has joined #ipfs
aw1nn has quit [Client Quit]
upperdeck has quit [Ping timeout: 252 seconds]
upperdeck has joined #ipfs
<whyrusleeping>
Alpha64: peers don't report size for a file
<whyrusleeping>
they give you the data you ask for, and you can tell what size it is
gts has joined #ipfs
<Alpha64>
ok so you have to download the file first?
<whyrusleeping>
only the root block, but yes
<Alpha64>
is there a preset size for the root block =
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
gts has quit [Ping timeout: 252 seconds]
bdTaylor has quit [Ping timeout: 240 seconds]
<whyrusleeping>
No block is larger than 1MB
upperdeck has quit [Ping timeout: 248 seconds]
<nothingmuch>
whyrusleeping: does ipfs files use the normal pinning mechanism?
upperdeck has joined #ipfs
<whyrusleeping>
nothingmuch: no, it has its own mechanism of preventing stuff from being gc'ed
<nothingmuch>
whyrusleeping: what if I *want* to gc locally?
mentos1386 has quit [Remote host closed the connection]
<whyrusleeping>
you want to gc stuff thats in your `ipfs files` space?
<nothingmuch>
whyrusleeping: yeah the motivation for the bitcoin thing is to offload the blk*.dat files to another server
mentos1386 has joined #ipfs
<nothingmuch>
while still having bitcoind be able to access them at much higher latency, to relay to my spv wallet
<whyrusleeping>
ah, i see
<whyrusleeping>
so, thats a thing that I want
<whyrusleeping>
but we don't have a way to do it yet
<whyrusleeping>
we explicitly designed the mfs system so that you could copy giant datasets into it and work on them without having to have everything locally
<nothingmuch>
whyrusleeping: i should still be able to pin the immutable trees coming out of ipfs files stat /, right?
upperdeck has quit [Ping timeout: 248 seconds]
arpu_ has quit [Quit: Verlassend]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
str1ngs has quit [Remote host closed the connection]
upperdeck has joined #ipfs
mentos1386 has quit [Quit: mentos1386]
mentos1386 has joined #ipfs
espadrine has quit [Ping timeout: 240 seconds]
upperdeck has quit [Ping timeout: 246 seconds]
upperdeck has joined #ipfs
ircler has joined #ipfs
ircler has quit [Client Quit]
upperdeck has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 240 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 255 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 252 seconds]
upperdeck has joined #ipfs
robattila256 has quit [Quit: WeeChat 1.9.1]
upperdeck has quit [Ping timeout: 258 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
kaotisk-irc has joined #ipfs
kaotisk has quit [Ping timeout: 248 seconds]
upperdeck has quit [Ping timeout: 248 seconds]
str1ngs has joined #ipfs
upperdeck has joined #ipfs
mentos1386 has quit [Quit: mentos1386]
dconroy has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 240 seconds]
anagh has quit [Ping timeout: 260 seconds]
dconroy has joined #ipfs
upperdeck has joined #ipfs
_whitelogger has joined #ipfs
upperdeck has quit [Ping timeout: 248 seconds]
upperdeck has joined #ipfs
dconroy has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<str1ngs>
well I'm pretty stoked now. ported most of what I need in my package manager to using ipfs
<str1ngs>
I hope to add more ipfs features though in the future.
upperdeck has quit [Ping timeout: 240 seconds]
dconroy has joined #ipfs
kaotisk-irc has quit [Read error: No route to host]
upperdeck has joined #ipfs
kaotisk-irc has joined #ipfs
fredthomsen has joined #ipfs
upperdeck has quit [Ping timeout: 255 seconds]
upperdeck has joined #ipfs
upperdeck has quit [Ping timeout: 255 seconds]
upperdeck has joined #ipfs
dconroy has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]