<havenwood>
cloud69: Looks like they might. Consider using their API.
<cloud69>
great!
silverdust has joined #ruby
<cloud69>
i'm gonna have to scrape the web though, buncha different sites, newegg was just an example
<havenwood>
cloud69: Mechanize is fairly fancy. Use APIs when you can. :)
<havenwood>
cloud69: For things with JavaScript you may need a headless browser.
<nakilon>
Policy & Agreement 2. Use of the Site Whether on behalf of yourself or on behalf of any third-party, you agree not to: “Scrape”, duplicate, reproduce, copy, republish, license, sell, trade or resell the Site for any purpose;
<cloud69>
this is more than i bargained for, thanks guys
<havenwood>
cloud69: As nakilon hints, be aware of locally applicable laws and courtesy for your fellow devs.
<cloud69>
nakilon: all good
<cloud69>
yep.
<havenwood>
I'm generally in favor of a scrapeable web.
<havenwood>
Free the info.
<havenwood>
All information should be free.
<havenwood>
Give the information what it wants. Freedom, and to not be personified.
<nakilon>
tell it to 99.99% of people -- today they refuse to even chat and read news for free, only through authorization with phone number and so passport
SeepingN has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
cthulchu_ has quit [Read error: Connection reset by peer]
lucasb has quit [Quit: Connection closed for inactivity]
SeepingN has joined #ruby
silverdust has quit [Ping timeout: 256 seconds]
TCZ has quit [Quit: Leaving]
batisi[m] has joined #ruby
SeepingN has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
silverdust has joined #ruby
ur5us has quit [Ping timeout: 264 seconds]
silverdust has quit [Ping timeout: 260 seconds]
silverdust has joined #ruby
Emmanuel_Chanel has quit [Ping timeout: 268 seconds]
SeepingN has joined #ruby
ur5us has joined #ruby
Emmanuel_Chanel has joined #ruby
Technodrome has joined #ruby
adu has joined #ruby
SeepingN has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
SeepingN has joined #ruby
chouhoulis has quit [Remote host closed the connection]
cloud69 has quit [Quit: Connection closed for inactivity]
ur5us has quit [Ping timeout: 264 seconds]
ChmEarl has quit [Quit: Leaving]
adu has quit [Quit: adu]
helpa has quit [Remote host closed the connection]
helpa has joined #ruby
cloud69 has joined #ruby
fippy has quit [Quit: Leaving]
fippy has joined #ruby
SeepingN has quit [Disconnected by services]
SeepingN_ has joined #ruby
DTZUZU has joined #ruby
DTZUZU_ has quit [Ping timeout: 260 seconds]
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
<cloud69>
havenwood: good stuff, cheers!
<cloud69>
big data big money:)
orbyt_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
howdoi has quit [Quit: Connection closed for inactivity]
SeepingN_ has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
tpanarch1st has quit [Ping timeout: 240 seconds]
akem has quit [Ping timeout: 244 seconds]
duderonomy has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
helpa-bot has joined #ruby
helpa has quit [Read error: Connection reset by peer]
helpa-bot has quit [Remote host closed the connection]
helpa has joined #ruby
helpa has quit [Remote host closed the connection]
helpa has joined #ruby
justinmcp has quit [Quit: No Ping reply in 180 seconds.]
justinmcp has joined #ruby
duderonomy has joined #ruby
ur5us has joined #ruby
Glumetu has joined #ruby
cloud69 has quit [Quit: Connection closed for inactivity]
cnsvc has quit [Ping timeout: 240 seconds]
cloud69 has joined #ruby
cd has quit [Quit: cd]
burgestrand has joined #ruby
imode has quit [Ping timeout: 256 seconds]
silverdust has quit [Ping timeout: 256 seconds]
silverdust has joined #ruby
silverdust has quit [Ping timeout: 272 seconds]
clemens3 has quit [Ping timeout: 256 seconds]
clemens3 has joined #ruby
pwnd_sfw has quit [Ping timeout: 240 seconds]
ur5us has quit [Ping timeout: 264 seconds]
pwnd_sfw has joined #ruby
BSaboia has joined #ruby
_whitelogger has joined #ruby
fandre1986 has joined #ruby
burgestrand has quit [Quit: burgestrand]
_whitelogger has joined #ruby
Technodrome has joined #ruby
somedude has joined #ruby
savvasal has joined #ruby
bamdad has quit [Ping timeout: 240 seconds]
savvasal has quit [Ping timeout: 245 seconds]
pwnd_sfw has quit [Ping timeout: 260 seconds]
pwnd_sfw has joined #ruby
bamdad has joined #ruby
Ediz has quit [Ping timeout: 240 seconds]
Ediz has joined #ruby
burgestrand has joined #ruby
Rudd0 has quit [Ping timeout: 258 seconds]
somedude has quit [Quit: Connection closed]
akem has joined #ruby
BSaboia has quit [Quit: This computer has gone to sleep]
dionysus69 has joined #ruby
apoc has quit [Quit: WeeChat 2.8]
Retropikzel has joined #ruby
apoc has joined #ruby
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
impermanence has joined #ruby
impermanence has quit [Client Quit]
impermanence has joined #ruby
queip has quit [Ping timeout: 260 seconds]
Technodrome has joined #ruby
lucasb has joined #ruby
Retropikzel has quit [Quit: Vision[0.10.3]: i've been blurred!]
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
ellcs has joined #ruby
bmurt has joined #ruby
silverdust has joined #ruby
cloud69 has quit [Quit: Connection closed for inactivity]
Technodrome has joined #ruby
donofrio has joined #ruby
BrianWGray has joined #ruby
weaksauce has quit [Ping timeout: 258 seconds]
Technodrome has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
akem_ has joined #ruby
SeepingN has joined #ruby
Retropikzel has joined #ruby
akem has quit [Ping timeout: 246 seconds]
queip has joined #ruby
ua_ has quit [Quit: Leaving]
ua has joined #ruby
ChmEarl has joined #ruby
fandre1986 has quit [Quit: Connection closed]
ellcs has quit [Ping timeout: 260 seconds]
Technodrome has joined #ruby
TCZ has joined #ruby
Rudd0 has joined #ruby
chouhoulis has joined #ruby
rippa has joined #ruby
Glumetu has quit [Quit: Glumetu]
bmurt has quit [Quit: My MacBook has gone to sleep. ZZZzzz…]
znz_jp has quit [Quit: kill -QUIT $$]
znz_jp has joined #ruby
burgestrand has quit [Quit: burgestrand]
weaksauce has joined #ruby
fippy has quit [Quit: Leaving]
fippy has joined #ruby
bmurt has joined #ruby
bmurt has quit [Client Quit]
bmurt has joined #ruby
bmurt has quit [Client Quit]
orbyt_ has joined #ruby
greypack has quit [Ping timeout: 264 seconds]
cloud69 has joined #ruby
TCZ has quit [Quit: Leaving]
howdoi has joined #ruby
lmat has quit [Quit: leaving]
Tempesta has quit [Quit: See ya!]
<isene>
apotheon: VIM motion keys implemented
<isene>
Also tab completion for dirs and files in the bottom command window
chouhoulis has quit [Remote host closed the connection]
<apotheon>
wow
<apotheon>
isene: cool, will check it out
Tempesta has joined #ruby
chouhoulis has joined #ruby
VicMackey has joined #ruby
<isene>
apotheon: The tab completion (as well as the whole readline system for the bottom window) is written from scratch as the Curses module is severely lacking of features - it cannot read multi-byte characters from the screen and cannot integrate with Readline.
<fippy>
Ctrl+h Ctrl+w ?
<VicMackey>
Hello! Quick question: I have set up the magic comment #frozen_string_literal = true, but Ruby lets me """""modify""""" my string: https://dpaste.org/nCMn . I'm guessing that with "=" I am instantiating a new variable string instead of modifying the previous one?
<VicMackey>
That's why I've seen people using "<<" instead?
<isene>
fippy: You would need that?
<leftylink>
it is true that a new string was created and then assigned to the same name
<leftylink>
on the other hand, String#replace would be unavailable, as would String#clear, etc.
<VicMackey>
Alright leftylink, I understand. I think there was a specific name for that... now shadowing comes to my mind, but that's not it... Anyway, I've checked and using replace, indeed, gives me an error. Understood. Thank you very much! :)
ua has quit [Ping timeout: 258 seconds]
<fippy>
isene: Yes, I need that
<isene>
I could need a helping hand from some quoting guru here. I have code in RTFM to call w3mimgdisplay for showing images in the terminal. But I cannot for the love of potatoes manage to quote the command correctly, so I put the command to run into a bash script and call that from Ruby. The bash script is here: https://github.com/isene/RTFM/blob/main/imgw3m.sh - The Ruby code that calls it is here:
<isene>
https://github.com/isene/RTFM/blob/main/rtfm#L619 - The point: I would like to have the code inside RTFM rather than calling that external bash script (I consider that an ugly hack). Anyone?
<isene>
fippy: If I get the urge, I will add that, too
<ruurd>
isene why? scripts are first grade citizens.
<ruurd>
Nothing wrong with invoking another executable from within an application.
ua has joined #ruby
<adam12>
isene: I might explore storing the bash script as a string in your app, then opening bash and piping it into stdin.
<adam12>
isene: Or write to tempfile, execute tempfile, then cleanup. Seems a bit more indirect.
VicMackey has quit [Quit: Konversation terminated!]
cnsvc has joined #ruby
CrazyEddy has quit [Ping timeout: 256 seconds]
CrazyEddy has joined #ruby
bocaneri has quit [Read error: Connection reset by peer]
<isene>
The reason for my request is that I want RTFM to be a complete stand-alone Ruby script without relying on calling a bash script that could just as well have been quoted correctly inside Ruby. Because as it stands, Ruby is calling a bash script that calls w3mimgdisplay. = Ugly. Specifically - what I need to quote correctly is these two lines:
<isene>
fippy: Implemented ctrl-h and ctrl-w for command line
<isene>
fippy: TIL Open3
ur5us has joined #ruby
<isene>
fippy: I'm running 2.7.0 (Ubuntu 20.04) - not getting Open3 - any require needed for 2.7.0 or is it simply not available until 2.7.1?
<jhass>
isene: anything /stdlib/ (compated to /core/) needs a require, yes
<fippy>
isene: Do You try to require 'open3' ?
ruurd has joined #ruby
<isene>
fippy: That worked :-) But the solution seems to be buggy... I can make that command work in pry, but when doing the same in RTFM, the image somtimes flashes and then disappears, but often does not even flash...
<isene>
Specifically, I added this line instead of calling the bash script:
<isene>
fippy: The odd thing is that it does in fact show thumbnails for videos, but not images directly when browsed (even though it calls the same function ' image_show() '
bambanx has joined #ruby
<cloud69>
havenwood: to scrape JSON would you recommend open-uri, httparty or rest-client?
<balo>
Net:HTTP? do you need a 3rd party lib for that?
fippy_ has joined #ruby
<jhass>
no :)
<cloud69>
:D
fippy has quit [Ping timeout: 256 seconds]
<isene>
fippy_: The odd thing is that it does in fact show thumbnails for videos, but not images directly when browsed (even though it calls the same function ' image_show() '
<jhass>
httprb for a nice interface, faraday if you need to abstract over multiple implementations, typhoeus if you need maximum performance
burgestrand has joined #ruby
<isene>
fippy_: Got it to work - but without enclosing the image file name in " 'image' " - that stopped the command. But - with file names that contain spaces, I would need to enclose it in '' - any tips to fix that?
<jhass>
I think open-uri is deprecated and httparty and rest-client are about the oldest kids on the block as it gets :)
burgestrand has quit [Client Quit]
rippa has quit [Quit: {#`%${%&`+'${`%&NO CARRIER]
Liothen has quit [Ping timeout: 264 seconds]
Liothen has joined #ruby
<havenwood>
jhass: open-uri isn't deprecated, just Kernel#open. You've gotta use URI.open explicitly now. Seems nicer to be explicit anyways.
<havenwood>
HTTParty, heh. Party hard!
<cloud69>
hehe
<havenwood>
+1 for what jhass said. http.rb for nice interface or typhoeus for tried and true libcurl with great parallelism support.
<isene>
fippy_: I am now using "shellwords" to escape odd characters, including spaces, in file names - but as I do ' @selected_safe = Shellwords.escape(@selected) ' and then issue @selected_safe as the image into ' Open3.capture2 @w3mimgdisplay, stdin_data: "0;1;#{img_x};#{img_y};#{img_w};#{img_h};;;;;#{image}\n4;\n3;" ' - it shows images except those with spaces in the path
ruurd has quit [Quit: ZZZzzz…]
ruurd has joined #ruby
<cloud69>
havenwood: very nice :)
TCZ has quit [Quit: Leaving]
<isene>
havenwood: Wanna pitch in with my issue here?
<havenwood>
isene: I'm behind on backlog and just jumping into a meeting, but happy to look as soon as I have a window.
<isene>
havenwood: That would be great - the issue being showing images with spaces in the file name
<havenwood>
isene: Scanning backlog it seems you're using Shellwords escape, which I'd think would work.
<havenwood>
isene: Can you gist the relevant code?
<havenwood>
isene: I don't get where `@selected_safe` is used.
<havenwood>
About to be back in meetings, but I'll check back in in case nobody beats me to it.
<havenwood>
3 hours of solid meetings coming up. Yikes...
* havenwood
is tempted to go back to coding Ruby for a living
<leftylink>
hold on.
<leftylink>
I don't believe a shell is involved here.
<leftylink>
so, is Shellwords.escape needed?
<havenwood>
leftylink: I assumed the shelling out Open3 code was related, but I don't get how either.
swaggboi has quit [Ping timeout: 264 seconds]
Rudd0 has joined #ruby
cliluw has quit [Ping timeout: 240 seconds]
cliluw has joined #ruby
swaggboi has joined #ruby
zacts has joined #ruby
<isene>
Here's the sweet and short. Relevant code: https://github.com/isene/RTFM/blob/main/rtfm#L624 (to get rid of the next line of code). I am using @selected_safe as the shellescaped version of the image file name. This does work for showing images in terminal via w3mimgdisplay - but not when the file names includes spaces. But - if I did like before - send the file name enclosed in 'image.png' to the
<isene>
bash script then all files are displayed. Sending file names enclosed in '' to the Open3 command results in no images shown. What I need is to be able to show images with spaces in the file names.
<isene>
leftylink: & havenwood ^^
<leftylink>
my assertion remains that since a shell is not involved, I don't understand why shell escape is used. Since I don't have w3mimgdisplay installed, it appears that I will not be able to test it myself, so I regret that I am unable to comment further, and wish best of luck
<isene>
The program /usr/lib/w3m/w3mimgdisplay takes a certain string from the command line/stdin and the Open3 seems to serve that quite well - except for files with spaces in the file names
<isene>
The really odd stuff is that I _can_ enclose the file name in ' ' when issuing the Open3 command within pry, but not in RTFM
sagax has quit [Remote host closed the connection]
<riceandbeans>
if you have foo['bar']['baz'] = 'biff' and you always know that any instance of foo['bar'] will always have a 'baz' key with a value, is there any advantage to using the value by doing foo['bar'].fetch('baz') rather than just saying foo['bar']['baz'] ?
<jhass>
other than validating the assumption, no
pwnd_sfw has quit [Ping timeout: 264 seconds]
dionysus69 has quit [Remote host closed the connection]
<riceandbeans>
validating the assumption?
zacts has quit [Quit: leaving]
<riceandbeans>
I mean there's no block with it, it's just using the value
<riceandbeans>
In this theoretical
dionysus69 has joined #ruby
tpanarch1st has joined #ruby
weaksauce has joined #ruby
<jhass>
&>> {}.fetch(:a)
<rubydoc>
stderr: -e:4:in `fetch': key not found: :a (KeyError)... check link for more (https://carc.in/#/r/9xas)
<jhass>
it raises if the assumption doesn't hold that the key is always there
<isene>
Eh, forget my last comment, doesn't work in pry either
roshanavand has quit [Ping timeout: 240 seconds]
TCZ has joined #ruby
imode has joined #ruby
fippy__ has joined #ruby
<riceandbeans>
Ok, but in this case the key will always be there
fippy_ has quit [Ping timeout: 265 seconds]
<riceandbeans>
It's a safe assumption here
<havenwood>
riceandbeans: A #fetch ensures your assumption is valid.
<havenwood>
riceandbeans: For lax behavior: foo.dig('bar', 'baz')
<havenwood>
riceandbeans: My question would be why #[] paired with #fetch, rather than two fetches or one dig.
<havenwood>
riceandbeans: Two fetches is explicit and strictly raises for a missing key. One dig allows both keys to be missing without raising.
TCZ is now known as herbstluft
<riceandbeans>
havenwood: This is in config management parsing a json struct and the second field will always be there in this case, if it's not the config management will crash on its own, I'm not sure I gain anything by trying to avoid a raise with an unreliable value, I have no other alternative. If the value is not there for some reason, the only safe alternative would be null, but that would crash later. Might
<riceandbeans>
as well just crash trying to use the value and realize that the data source has to be fixed.
<havenwood>
riceandbeans: So that's two extra method calls, which has a cost — hence dig.
<rubydoc>
stderr: -e:4:in `<main>': undefined local variable or method `foo' for main:Object (NameError)... check link for more (https://carc.in/#/r/9xav)
<havenwood>
riceandbeans: "Content in the AppStream repository includes additional user space applications, runtime languages, and databases in support of the varied workloads and use cases."
<havenwood>
riceandbeans: "runtime languages"
<riceandbeans>
Well, if we can get that, then I should be ok...