Caterpillar has quit [Quit: You were not made to live as brutes, but to follow virtue and knowledge.]
Rusty78 has joined #ipfs
<MikeFair>
requested invite
Rusty78 has quit [Client Quit]
jesse22 has joined #ipfs
jasonyu1840[m] has joined #ipfs
jesse22 has quit [Client Quit]
jesse22 has joined #ipfs
borgtu_ has quit [Remote host closed the connection]
infinity0_ has joined #ipfs
infinity0_ has quit [Changing host]
infinity0_ has joined #ipfs
infinity0 is now known as Guest9568
Guest9568 has quit [Killed (tolkien.freenode.net (Nickname regained by services))]
infinity0_ is now known as infinity0
ONI_Ghost has joined #ipfs
<camb[m]>
I simply can't find an open source video search engine for IPFS with schema or microformats support and thumbnails. I'm having a hard time believing none exist. Does anyone have suggestions?
<Fessus>
camb[m], there really aren't a lot of good ways to search IPFS content atm, especially open source ones
<Fessus>
a majority of videos stored on IPFS are through dtube so I guess that's a nice start
espadrine_ has quit [Ping timeout: 240 seconds]
dgrisham has quit [Quit: WeeChat 2.0.1]
<MikeFair>
camb[m], You mean something like D-Tube?
* MikeFair
didn't read down far enough *chagrin*
<camb[m]>
I'm aware of Dtube.
<MikeFair>
camb[m], The challenge is that CAS is simply hard to search
<camb[m]>
as far as I understand it's just a simple way of uploading to ipfs and Steem.
<camb[m]>
for a price
<MikeFair>
The way it needs to be done is you distribute your search query out to all the nodes, which then provide feedback on a channel you're listening too
<MikeFair>
but the infrastructure for that isn't here
<Fessus>
camb[m], it's free to upload
<camb[m]>
the dtube search will only find videos that were uploaded on the dtube website as far as i understand.
<MikeFair>
I should say the applications aren't here; the infrastructure is
<camb[m]>
Fessus: they take a percentage of your steem rewards.
<camb[m]>
Semi free then.
<MikeFair>
camb[m], Yes, you can't "crawl" IPFS
<Fessus>
google seems to manage
emblazer has joined #ipfs
ONI_Ghost has quit [Ping timeout: 240 seconds]
<Fessus>
you can search fairly effectively by just using "site:"ipfs.io <stuff> -wiki"
<Fessus>
in google
<MikeFair>
Fessus, that's because Google is dealing with (1) registered root websites; (2) mutably linked data; (3) has built the distributed storage and processing power to suck the whole internet through its backbone of datacenters
<MikeFair>
IPFS doesn't have linked content
<Fessus>
no, but what they've cataloged is impressive
<MikeFair>
yacy looks exactly like what I described; you send your query out to lots of distributed nodes that execute your search and give you the feedback on a callback channel you're watching for
<camb[m]>
or even just any open source video search that I could try to tweek to IPFS
<MikeFair>
camb[m], I don't think you're looking for video search; you're looking for video meta data search
<MikeFair>
you're looking for videos with keywors in the descriptions I'm assuming
ONI_Ghost has joined #ipfs
<MikeFair>
Unless you're really expecting to do the video image analysis to determine "what" the video content contains
<MikeFair>
camb[m], If you can already search IPFS text via yacy, then that's the place for you to start
<camb[m]>
MikeFair: you are right. I'm looking looking for video meta data search
<MikeFair>
camb[m], the yacy indexing algorithm needs to be updated to understand references to videos; to recognize video binary files and suck out their metadata and closed caption, and whatever other "textual content" they have
<MikeFair>
the biggest challenge there imho is you have to suck the video across the wire to the indexer first
<camb[m]>
I will keep working with Yacy if there are no better suggestions.
* MikeFair
wonders why people seem to keep missing the "big problem" with doing indexed search this way; it's like spam on the network... ;)
<camb[m]>
What part is spam on the network?
<MikeFair>
If you really want to work on "better solution" then I'd advise you write an indexing service that can run on an IPFS host that can index only its local content
<MikeFair>
You have to keep sucking the data from the source to the indexer to keep the indexer current
shizukesa has joined #ipfs
<MikeFair>
even if that data is never requested
<MikeFair>
but indexing the data requests the data
<MikeFair>
therefore things become "most interesting" because they are being indexed; not because anyone actually cares
<MikeFair>
and the network bandwidth is being consumed to transer all these files from sources to indexers
<AphelionZ>
MikeFair: good call on the node-level indexing
<AphelionZ>
Thats exactly how it should work.
<MikeFair>
To avoid that, you have to index the data "at the source" not move it first
<AphelionZ>
It should report its index metadata out on HTTP and explicitly NOT bitswap
<MikeFair>
Personally I think we need to take it one step further, and allow each other to submit "search algorithm indexing code" to nodes
<AphelionZ>
Yeah, def
<MikeFair>
So the algorithms are published in IPFS to give them an identifier
<MikeFair>
I can then say search for "whatever some of my search data description is" using "/ipfs/CIDOFALGO"
dimitarvp has quit [Quit: Bye]
<MikeFair>
That way you make smart search algos that can identify the structured content your looking for quickly ; execute the search ; and report back the findings
<MikeFair>
I think PubSub would be find for posting results
randomstrangerb has quit [Ping timeout: 268 seconds]
<MikeFair>
s/find/fine/
randomstrangerb has joined #ipfs
<MikeFair>
Because you have N nodes answering in the affirmative; you'll have multiple sources posting their results
<AphelionZ>
Would be a fixed list of search algorithms or would you be able to write and submit your own
<AphelionZ>
Relatedly is that how computes works ChrisMatthieu
<MikeFair>
obviously submit new ones, probably to an IPNS address
<AphelionZ>
Functions passed via pubsub?
<MikeFair>
AphelionZ, I'm thinking posted to IPFS, then referenced via PubSub
<AphelionZ>
Right
<MikeFair>
If they're anything worth their salt, they'll reused often
<MikeFair>
And we'll want to audit and "approve" them
<camb[m]>
I only index links by trusted sources. I wouldn't call that spam.
<MikeFair>
camb[m], it's the bandwidth consumed to suck the data down to your local node for the sole purpose of indexing it
<MikeFair>
not that it's unwanted
<AphelionZ>
Well theoretically you should only have content by links from trusted sources
<AphelionZ>
Er... content from trusted sources
<MikeFair>
I disagree; unless "trusted sources" has a very broad "not blacklisted" conatation
<MikeFair>
You're searching for it specifically because you don't know where it is
shizukesa has quit [Ping timeout: 240 seconds]
<MikeFair>
I don't know what the sources for great cat videos are, so how could I have "preauthorized" them
ccii has joined #ipfs
<MikeFair>
There's certainly a usefulness to being able to constrain a search to certain sources (like a domain name) but it's not the general case imho
<MikeFair>
I definitely think we shouldn't be indexing data that wasn't registerd to be searched
emblazer has quit [Remote host closed the connection]
<MikeFair>
which is a bit of a different thing though
ccii1 has quit [Ping timeout: 240 seconds]
<MikeFair>
It's a tough nut to exactly crack obviously; and I definitely some kind of hybrid ground will develop where reputation and "trustworthiness" will elevate the order of results
<MikeFair>
but one person's "trusted source" is another person's "mortal attacker"
<MikeFair>
The Catholic Church is a great example of that
<MikeFair>
Obviously they'd have content to publish
<camb[m]>
lol
<MikeFair>
And some poeple would only want exclusively those results curated by them ; while other's would want to avoid anything they approved
braxaro_ has joined #ipfs
braxaro_ has left #ipfs [#ipfs]
* MikeFair
is a big beleiver in identifying "source" (not necessarily original author) + "data"
jesse22 has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<MikeFair>
they both are equally important to making a validity decision over
raynold has quit [Quit: Connection closed for inactivity]
<MikeFair>
Because I don't think I could handle a world where I get every result anyone has ever published on a few keywords
<MikeFair>
:)
<MikeFair>
camb[m], I think you should actually look at Computes too
<MikeFair>
camb[m], If we get the computes indexers to be physically close to the IPFS nodes; then I think it'd work well
<MikeFair>
camb[m], I think the index would be publishable in IPLD and available to everyone
<camb[m]>
sounds like a good idea.
ryjen has joined #ipfs
hexed[m] has joined #ipfs
chowie has joined #ipfs
ryjen has quit [Ping timeout: 256 seconds]
randomstrangerb has quit [Ping timeout: 256 seconds]
ryjen has joined #ipfs
randomstrangerb has joined #ipfs
droman has quit [Quit: WeeChat 2.0.1]
ryjen has quit [Ping timeout: 248 seconds]
sz0 has quit [Quit: Connection closed for inactivity]
ONI_Ghost has quit [Ping timeout: 256 seconds]
ericxtang has quit [Remote host closed the connection]
ericxtang has joined #ipfs
jaboja has quit [Remote host closed the connection]
ericxtan_ has joined #ipfs
ericxtang has quit [Ping timeout: 240 seconds]
lord| has quit [Ping timeout: 240 seconds]
ericxtan_ has quit [Ping timeout: 240 seconds]
ryjen has joined #ipfs
ryjen has quit [Ping timeout: 240 seconds]
ryjen has joined #ipfs
PyHedgehog has joined #ipfs
lord| has joined #ipfs
leebyron has joined #ipfs
colatkinson has joined #ipfs
ryjen has quit [Ping timeout: 248 seconds]
ONI_Ghost has joined #ipfs
ryjen has joined #ipfs
SunflowerSociety has joined #ipfs
ONI_Ghost has quit [Ping timeout: 248 seconds]
ONI_Ghost has joined #ipfs
ryjen has quit [Ping timeout: 268 seconds]
SunflowerSociety has quit [Ping timeout: 248 seconds]
ryjen has joined #ipfs
colatkinson has quit [Quit: colatkinson]
pcardune has joined #ipfs
Neomex has joined #ipfs
rngkll has quit [Read error: Connection reset by peer]
anewuser has joined #ipfs
ryjen has quit [Ping timeout: 248 seconds]
colatkinson has joined #ipfs
screensaver has joined #ipfs
screensaver14 has quit [Ping timeout: 276 seconds]
ONI_Ghost has quit [Ping timeout: 240 seconds]
chowie has quit [Ping timeout: 248 seconds]
}ls{ has quit [Quit: real life interrupt]
pcardune has quit [Remote host closed the connection]
clemo has quit [Ping timeout: 264 seconds]
leebyron has quit [Remote host closed the connection]
ericxtang has joined #ipfs
ericxtang has quit [Ping timeout: 252 seconds]
anewuser has quit [Ping timeout: 248 seconds]
ygrek has joined #ipfs
Mr0grog has joined #ipfs
Mr0grog has quit [Client Quit]
leebyron has joined #ipfs
lifeonalagoon[m] has joined #ipfs
engdesart has left #ipfs ["no"]
raynold has joined #ipfs
Graylan has joined #ipfs
Graylan has quit []
Graylan has joined #ipfs
fractex has quit [Ping timeout: 256 seconds]
leebyron has quit [Remote host closed the connection]
leebyron has joined #ipfs
leebyron has quit [Remote host closed the connection]
leebyron has joined #ipfs
leebyron has quit [Ping timeout: 248 seconds]
engdesart has joined #ipfs
dpl has joined #ipfs
ulrichard has joined #ipfs
dpl_ has joined #ipfs
dpl has quit [Ping timeout: 264 seconds]
Amitha has joined #ipfs
fractex has joined #ipfs
Amitha has quit [Ping timeout: 260 seconds]
colatkinson has quit [Quit: colatkinson]
colatkinson has joined #ipfs
leebyron has joined #ipfs
colatkinson has quit [Client Quit]
colatkinson has joined #ipfs
leebyron has quit [Ping timeout: 248 seconds]
markwylde[m] has joined #ipfs
ygrek has quit [Ping timeout: 268 seconds]
runde has quit [Ping timeout: 265 seconds]
mtodor has joined #ipfs
mtodor has quit [Remote host closed the connection]
espadrine_ has joined #ipfs
doubleorseven[m] has joined #ipfs
warner has joined #ipfs
upperdeck has quit [Ping timeout: 268 seconds]
inetic has joined #ipfs
colatkinson has quit [Quit: colatkinson]
colatkinson has joined #ipfs
dolphy has joined #ipfs
ecloud has quit [Read error: Connection reset by peer]
inetic has quit [Ping timeout: 256 seconds]
inetic has joined #ipfs
espadrine_ has quit [Ping timeout: 240 seconds]
cwahlers has joined #ipfs
cwahlers_ has quit [Ping timeout: 256 seconds]
ylp has joined #ipfs
pcardune has joined #ipfs
pcardune has quit [Ping timeout: 252 seconds]
ecloud has joined #ipfs
rendar has joined #ipfs
Neomex has quit [Remote host closed the connection]
Graylan has quit [Quit: Connection closed for inactivity]
bomb-on has quit [Quit: zzz]
xzha has joined #ipfs
fireblade[m] has joined #ipfs
runde has joined #ipfs
jungly has quit [Ping timeout: 240 seconds]
jungly has joined #ipfs
colatkinson has quit [Ping timeout: 240 seconds]
metric has quit [Ping timeout: 240 seconds]
bomb-on has joined #ipfs
dpl_ has quit [Quit: Lost terminal]
colatkinson has joined #ipfs
metric has joined #ipfs
metric is now known as Guest43409
saki has quit [Ping timeout: 268 seconds]
raynold has quit [Quit: Connection closed for inactivity]
<MikeFair>
I'm not sure if I need a real peerId or not
<MikeFair>
(or even want one)
<whyrusleeping>
whats that do?
<MikeFair>
It's a CID over HTTP tool
<MikeFair>
You can upload CIDs, request CIDs, upload files
<MikeFair>
It's designed to mimic the IPFS file routing protocol over HTTP
<MikeFair>
so that more websites on shared hosts can act as the CID block file repositories for IPFS
<MikeFair>
It's also to help my JS Browsers Pin stuff
<MikeFair>
It's trivial to get a JS Broswer to upload a file to a website that can then server as a source of "pinning" for other IPFS nodes
<MikeFair>
It doesn't speak the IPFS network protocol at all; just short lived HTTP request/response (like a web site does)
<MikeFair>
Using cron, it would be able to link to other HTTP servers and execute a "very slow" version of the CID routing algorithm
esph has quit [Ping timeout: 248 seconds]
<MikeFair>
(Culminating in turning this simple script into something like a WordPress plugin)
robattila256 has quit [Quit: WeeChat 2.0.1]
esph has joined #ipfs
<MikeFair>
(So all those WordPress sites can act as CID block file sharing websites)
robattila256 has joined #ipfs
<MikeFair>
whyrusleeping, Many folks here have faced this weird challenge where we unlike those maintaining ipfs.io; we don't have long lived Go Nodes; and the js-ipfs nodes are, well, not quite there yet
mtodor has joined #ipfs
mtodor has quit [Remote host closed the connection]
<whyrusleeping>
So why not run a go node
<whyrusleeping>
?
<MikeFair>
And while I don't a VPS, I do havea shared webshoting with lots of file storage space
<MikeFair>
I don't have a fixed place on the internet to run one
<whyrusleeping>
but you have a fixed place on the internet to run that tool?
<AphelionZ>
[DeNiro voice] I run a go node, focker. can you bitswap me?
<MikeFair>
Yes, a shread webhost behind an Apache server
<MikeFair>
err shared
<MikeFair>
I accidently left a Go node running there once; they killed it ;)
<whyrusleeping>
ah, you want lower resource consumption
<MikeFair>
(As they should have, because I was a moron)
<MikeFair>
I want shared HTTP server host
<AphelionZ>
whyrusleeping: while you're here is there anything special i need to enable to be able to point DNS at it and use dnslink?
<MikeFair>
You may not use BIND
<MikeFair>
You must use CRON
<whyrusleeping>
AphelionZ: to point dns at what exactly?
<AphelionZ>
My go node
<MikeFair>
You may not run for more than 5 minutes
<MikeFair>
that kind of thing
<AphelionZ>
Like i would ipfs.io's gateway
<whyrusleeping>
AphelionZ: nothing special, just make sure your gateway is publicly accessible, and it should work fine
<AphelionZ>
Huh ok
<MikeFair>
AphelionZ, a TEXT entry called dnslink
<AphelionZ>
Yeah i have that for mrh.io but its pointed at ipfs.io
<AphelionZ>
whyrusleeping: i will try harder and report back. My nginx reverse proxy might be messing with it
<MikeFair>
If you want a browser to be able to read it; it must point at a website
<MikeFair>
To my knowledge a Go Node doesn't speak HTTP at this time
<MikeFair>
(it might over the API, but not present a web page)
<MikeFair>
You need a website to catch the broswers http request and do the IPFS request
<MikeFair>
One that might be hosted at a shared hosting provider ;)
<MikeFair>
whyrusleeping, Am I wrong? Does a Go node provide a web page on any of its listenting ports?
<Icefoz>
MikeFair: I think it does.
<Icefoz>
Yeah there's a way to make it serve files via HTTP, and it will generate index pages and such.
<MikeFair>
I couldn't find one; there's the "api" endpoints but that's more REST JSON
<MikeFair>
I see my daemon says it launched the gateway on 8080; but the browser keeps returning 404s
<MikeFair>
:5001/webui brought up a node management page
<MikeFair>
Do I need to publish a /index.html or something to my local repo in order to make 8080 work?
<MikeFair>
Is there any way I can use the IPNS keys I developed as the "root" for port 8080?
* MikeFair
will have to rummage through the docs
<MikeFair>
AphelionZ, but it looks like what you want is most likely to serve out the gateway port (8080 by deafult) from the nginx reverse proxy
<MikeFair>
I'm not sure how to publish the web page content for the gateway to serve
<whyrusleeping>
MikeFair: the gateway on 8080 requires you give it an ipfs path
<whyrusleeping>
so localhost:8080/ipfs/QmFoo
<MikeFair>
Right, but that's not the same thing as what you'd want if you were to redirect a DNS query from a browser to your Go node
<MikeFair>
(to use as your web host server)
<MikeFair>
So there's no way currently for me to provide an "index.html" type file resource for the gateway
<MikeFair>
(for non /ipfs/ directory paths
<MikeFair>
It might be better to create third thing called "HTTP server" for this rather than overloading the Gateway, but since the Gateway already seems HTTP capable; and only seems to respond to ipfs/ipns namespaces; using the root space to serve html files directly seems to make sense
<MikeFair>
I'd put some "Gateway_Default_CID" in my config
<MikeFair>
maybe even the node's own IPNS address?
<MikeFair>
(we could choose a different directory name to store the files in, like /blocks/QmFoo or something, to avoid confusion with "real" ipfs nodes; the concept remains the same.
Caterpillar has quit [Ping timeout: 256 seconds]
erictapen has quit [Remote host closed the connection]
erictapen has joined #ipfs
colatkinson has joined #ipfs
pcardune has quit [Remote host closed the connection]
colatkinson has quit [Quit: colatkinson]
trqx has quit [Ping timeout: 255 seconds]
robattila256 has quit [Quit: WeeChat 2.0.1]
robattila256 has joined #ipfs
dolphy has joined #ipfs
dolphy has quit [Remote host closed the connection]
vivus has joined #ipfs
trqx has joined #ipfs
Neomex has quit [Ping timeout: 260 seconds]
Fessus has joined #ipfs
colatkinson has joined #ipfs
randomstrangerb has quit [Ping timeout: 268 seconds]
randomstrangerb has joined #ipfs
ygrek has joined #ipfs
lemongrass has joined #ipfs
clownpriest has joined #ipfs
MikeFair has quit [Ping timeout: 240 seconds]
<lemongrass>
quit
lemongrass has left #ipfs [#ipfs]
MDude has quit [Ping timeout: 256 seconds]
TheAssassin has quit [Read error: Connection reset by peer]
MDude has joined #ipfs
Encrypt has quit [Quit: Quit]
Mitar_ has joined #ipfs
Neomex has joined #ipfs
<Mitar_>
is there any documentation how I could run a Docker IPFS container on my server and use it as seed box? so that I can somehow submit a list of IDs that IPFS instance should keep cached and share it around?
pcardune has joined #ipfs
colatkinson has quit [Quit: colatkinson]
<whyrusleeping>
Mitar_: I don't think we have docs on that right now
<whyrusleeping>
but if you write up your usecase somewhere we can work on writing up some
<Mitar_>
oh, I just have some datasets I would like to share with my researchers and students
<Mitar_>
so I was thinking of using IPFS
<Mitar_>
and put a Docker container on one of servers
<Mitar_>
and then ideally I would add dataset on my local machine
<Mitar_>
go to server, and say to keep this ID around and seed it
<Mitar_>
so that then I can give students the IPFS link to the dataset
<Mitar_>
and know that there will be at least one machine around providing that
<Icefoz>
Mitar_: "ipfs pin" will do the "keep this ID around and seed it"
<Icefoz>
Setting up a docker container to do that should be pretty easy.
<Mitar_>
so you are saying I could just docker exec into the container and run the pin
<Mitar_>
hmm
colatkinson has joined #ipfs
<Mitar_>
yea, you are probably right
<Icefoz>
And there appears to be an official Docker image.
<Mitar_>
remind me, how does ipfs know which commands are local and which are coming from peers?
<Icefoz>
Mitar_: In what way? Using the HTTP API?
Jesin has joined #ipfs
<Icefoz>
By default the HTTP API only listens on localhost.
<Mitar_>
I mean, ipfs command line client is connecting over HTTP as well, no?
<Mitar_>
and HTTP API is different from P2P communication?
randomstrangerb has quit [Ping timeout: 248 seconds]
<Icefoz>
Yes, though I think the command line client might also do things like inspect the local database directly for some commands... I'm not sure.
randomstrangerb has joined #ipfs
<Icefoz>
By default the IPFS node only listens for commands on a local address.
<Mitar_>
thanks
K0HAX has quit [Ping timeout: 248 seconds]
K0HAX has joined #ipfs
leebyron has quit [Remote host closed the connection]
clownpriest has quit [Ping timeout: 240 seconds]
tombusby has quit [Ping timeout: 255 seconds]
rngkll has quit [Remote host closed the connection]
tombusby has joined #ipfs
leebyron has joined #ipfs
tenchooo has joined #ipfs
Mitar_ has quit [Quit: Page closed]
TheAssassin has joined #ipfs
Ecran has quit [Quit: Going offline, see ya! (www.adiirc.com)]
Neomex has quit [Ping timeout: 248 seconds]
zopsi has quit [Quit: Oops]
zopsi has joined #ipfs
tombusby has quit [Remote host closed the connection]
tombusby has joined #ipfs
KP_ has joined #ipfs
KP_ is now known as KPat
KPat is now known as KPat92
jesse22_ has joined #ipfs
esph has quit [Ping timeout: 260 seconds]
Hory has quit [Quit: Leaving]
jesse22 has quit [Ping timeout: 246 seconds]
lord| has joined #ipfs
esph has joined #ipfs
maxzor__ has quit [Remote host closed the connection]
colatkinson has quit [Ping timeout: 248 seconds]
jaboja has quit [Quit: Leaving]
jesse22_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
rngkll has joined #ipfs
rngkll_ has joined #ipfs
rngkll__ has joined #ipfs
rngkll has quit [Ping timeout: 240 seconds]
espadrine_ has joined #ipfs
ZaZ has quit [Read error: Connection reset by peer]
rngkll_ has quit [Read error: Connection reset by peer]
jesse22 has joined #ipfs
rngkll__ has quit [Ping timeout: 252 seconds]
vmx has quit [Ping timeout: 256 seconds]
Caterpillar has joined #ipfs
Caterpillar has quit [Client Quit]
rodolf0 has quit [Quit: Leaving]
vmx has joined #ipfs
rngkll has joined #ipfs
esph has quit [Ping timeout: 260 seconds]
rngkll has quit [Ping timeout: 240 seconds]
esph has joined #ipfs
chowie has joined #ipfs
leebyron has quit [Read error: Connection reset by peer]
leebyron has joined #ipfs
steveh7 has joined #ipfs
elevated_ has joined #ipfs
elevated has quit [Ping timeout: 240 seconds]
elevated_ is now known as elevated
colatkinson has joined #ipfs
tglman has quit [Ping timeout: 246 seconds]
<vmx>
gozala: i saw your pull request and wanted to have a look today, but only had brief look so far. i'll have a proper look tomorrow