<Regenaxer>
so did it crash in "(set (: 0 1) (++ West)) # west" ?
clacke_movim has joined #picolisp
<tankf33der>
i think its not finished.
<tankf33der>
i just copyed code, but this is real grid function
<tankf33der>
not smaller code.
<tankf33der>
if i comment south code, code dont crash
<tankf33der>
any ather dont help.
<Regenaxer>
ok
<Regenaxer>
If the 'for' was not finished, it crashed in (gc) after (set (: 0 1) (++ West)) # west
<Regenaxer>
which makes no sense to me, as 'set' cannot be Ca problem
<Regenaxer>
as far as I see
<Regenaxer>
The problem must be at some completely different place
<Regenaxer>
The crash is just a symptom
<Regenaxer>
hmm, what to do?
<Regenaxer>
I still suspect the compiler
<Regenaxer>
Can you try different optimization flags?
<Regenaxer>
-O1 or -Ot
<Regenaxer>
What is the default? -O2 ?
<freemint>
Good Morning
<Regenaxer>
Hi freemint
<Regenaxer>
tankf33der, perhaps try -O0 first?
<tankf33der>
maybe stack protector
<tankf33der>
as module in gcc
<Regenaxer>
?
<Regenaxer>
I see we use -O2 in src/Makefile
<freemint>
That has to be a nasty, nasty bug if you still have not fixed it.
<Regenaxer>
T
<Regenaxer>
I think it is not a bug in pil
<freemint>
What tools have you used so far to find it?
<Regenaxer>
I'm afraid there are no useful general tools
<tankf33der>
-O0 didnt help
<Regenaxer>
tankf33der used sanity checckers
<Regenaxer>
oh
<tankf33der>
afk.
<Regenaxer>
thanks tankf33der!
rob_w has joined #picolisp
<Regenaxer>
What is needed to find the cell which seems to have a bad pointer when it causes gc to crash, then go *back* and find the place in the code which *wrote* this pointer
<tankf33der>
-fno-stack-protector didnt help.
<Regenaxer>
Searching the needle in the haystack
<Regenaxer>
ok
<freemint>
Could you modify the GC to make a it log which cells it touches?
<Regenaxer>
yes, but you must know what ti search for
<Regenaxer>
atm I find no reasonable explanation
<Regenaxer>
Problem is it is very many cells
<Regenaxer>
and gc touches *all*
<Regenaxer>
I do it with gdb usually
<freemint>
Yeah but you can just look at which cell it was going to touch before it crashed
<Regenaxer>
correct
<Regenaxer>
or write a custom checker
<Regenaxer>
Debugging gc is really tough, as it modifies all pointers (sets mark bit)
<Regenaxer>
A custom checker could check the whole heap for bad pointers
<Regenaxer>
after finding out first what "bad" means here
<Regenaxer>
eg.points outside any legal heap, or such
<Regenaxer>
or is it the stack pointer as tankf33der suspecwed?
<Regenaxer>
'gc' in pil32 is recursive (in pil64 not)
<Regenaxer>
well, it is also recursive in pil64, but does not use the stack
<Regenaxer>
when it crashed, give bt command (backtrace)
<beneroth>
Good morning all
<Regenaxer>
Hi beneroth
orivej has quit [Ping timeout: 248 seconds]
<freemint>
where are the coredump stored?
<beneroth>
run dir usually
<beneroth>
I mean the dir the process was executed in. local relative path.
<beneroth>
I think...
<Regenaxer>
Linux creates no core by default
<beneroth>
ah.. didn't it once? or maybe debian?
<Regenaxer>
Not as long as I remember. Can re switched on easily, but I forgot how to do it
<beneroth>
I guess this is a usable tutorial:https://linux-audit.com/understand-and-configure-core-dumps-work-on-linux/
<beneroth>
ulimit
<Regenaxer>
ah
<Regenaxer>
T
<Regenaxer>
freemint, how big is the stack in your Fedora/Qemu setup?
ubLIX has joined #picolisp
<Regenaxer>
I more and more think it must be the stack
<Regenaxer>
Traversing 'grid' recursively takes a lot of stack, as the grid nodes are all connected
<Regenaxer>
I forgot what tankf33der said about his stack situation
<Regenaxer>
*and* we should test mini
<Regenaxer>
'grid' works in mini the same
<Regenaxer>
Traversing 'grid' recursively takes a lot of stack, as the grid nodes are all connectedse
<Regenaxer>
oops
<freemint>
i have not managed to create a stack trace
<Regenaxer>
I mean what is the limit
<Regenaxer>
ulimit -s
<Regenaxer>
and what if you set it unlimited?
<Regenaxer>
ulimit -s unlimited
<Regenaxer>
And then mini would not crash as it does not recurse in mark()
<Regenaxer>
yes, it must simply be a stack issue!!
<freemint>
with qemu-i386 i can load and execute a binary so i have to find a way to call ulimit inside the same qemu session
<tankf33der>
(gc 200)
<tankf33der>
(load "@lib/simul.l")
<tankf33der>
(gc)
<tankf33der>
(setq G (grid 2000 2000))
<tankf33der>
this code crash on linux, on pil32
<Regenaxer>
What stack size?
<Regenaxer>
and in mini?
<Regenaxer>
ah, sorry!
<Regenaxer>
mini uses recursive mark() too!
<Regenaxer>
No need to test
<tankf33der>
tried on ulimit -s 65000
<Regenaxer>
Only pil64 uses non-recursive mark
<Regenaxer>
ulimit -s unlimited
<tankf33der>
so 1000 1000 works on linux, 2000 2000 - crash
<Regenaxer>
mark() recurses on the CAR of each cell
<Regenaxer>
'grid' connects *all* cells
<Regenaxer>
So it recurses very deep!
<Regenaxer>
I think it is such simple
<freemint>
if understand it right the recursion should not go much deeper than (* 2000 2000
<freemint>
9
<tankf33der>
works everywhere if set unlimited stack depth.
<Regenaxer>
👍👍💪💪
<Regenaxer>
freemint, yes, or is it only 2000?
<freemint>
so the same should bug should happen with a build list in terms of cars of size 2000 2000
<freemint>
* 2000 * 2000
<Regenaxer>
yes, probably
<tankf33der>
i think issue closed.
<Regenaxer>
grid has (west . east) in the CAR
<Regenaxer>
tankf33der, yesss! :)
<Regenaxer>
so right freemint, it is 2000*2000
<Regenaxer>
every cell is reachable from every cell
<freemint>
It is 263*2000 which is the minimal situation where it crashes in qemu
<Regenaxer>
So in the beginning of gc it traverses completely down
<Regenaxer>
ok
<freemint>
(setq G (cons)) (for i 2000 (prinl i) (for j 2000 (setq G (cons G NIL))) (gc))
<freemint>
*262*2000
<Regenaxer>
ok, 1-dim here
<Regenaxer>
recursing on CAR
<Regenaxer>
consing
<freemint>
Yes
<Regenaxer>
good
<Regenaxer>
One more reason to abandon pil32 ;)
<Regenaxer>
well, unlimited stack is recommended anyway
<beneroth>
so no bug, but stack overflow crash?
<beneroth>
running into stack limit?
<Regenaxer>
correcbn
<Regenaxer>
correcy
<Regenaxer>
grrrr
<Regenaxer>
correct
<freemint>
i think this setup is simpler, the reason why it crashes double so fast (* 262 2000) vs (* 1000 1000) is that some cells get visited by cdr in grid
<Regenaxer>
not sure, iirc it recurses on CAR first
<beneroth>
no way to recognize the situation and issue a better error message/symptom, I guess? (not without complicating things needlessly...)
<Regenaxer>
so CDR of the CARs
<Regenaxer>
beneroth, it would need runtime checks
<Regenaxer>
but even so, C has no way to control the stack
<Regenaxer>
This was one of the main reasons for assembly
<beneroth>
could there be a little script which can detect the actual stack limit and tell if the current program seems to run into it? I mean as a kind of additional profiler/debug scripts, to be loaded on purpose, not for general runtime use.
<beneroth>
T
<beneroth>
hmm
<freemint>
(for i 2000 (prinl i ":2size G)) (for j 2000 (setq G (cons G NIL))) (gc))
<Regenaxer>
you can remember the address of a local var
<Regenaxer>
and take the diff
<beneroth>
but why the symptom difference on BSD and linux? different ulimit settings/mechanism, tankf33der ?
<freemint>
are there different ulimits on bsd?
<Regenaxer>
I think so, plus perhaps different stack usage
<beneroth>
more overhead?
<Regenaxer>
alignment perhaps
<beneroth>
alignment and lisp are orthogonal concepts as far as I understand...
<Regenaxer>
but it was all gcc I think
<freemint>
larger stack?
ubLIX has quit [Ping timeout: 258 seconds]
<Regenaxer>
larger stack means different default ulimit
ubLIX has joined #picolisp
<Regenaxer>
Physically the stack in Unix is limited only by virtual mem size
<tankf33der>
unlimit on openbsd is 32M
<freemint>
if ulimit says unlimited does that mean ulimit is configured rigth?
<Regenaxer>
hmm, quite big, but Linux is 64 iirc
<Regenaxer>
right for pil
<Regenaxer>
or other recursive programs
<freemint>
I had a crash with ulimit -> unlimited
<freemint>
(for i 20 (setq G (grid (** 2 i) (** 2 i)) (gc) (prinl i))) it crashed some time after printing 12.
<Regenaxer>
on qemu? This might have also small virtual mem size
<Regenaxer>
image size
<Regenaxer>
or no virtual at all
<Regenaxer>
ie no swapping
<Regenaxer>
like on Android too
<freemint>
ah ulimit -s is stacksize which is not unlimited
<Regenaxer>
ok
<Regenaxer>
So great at least that we could explain it!
<Regenaxer>
Thanks a lot to tankf33der!
<freemint>
beneroth, you know the stack pointer in the beginning, you know the current stack pointer and you know the stack size, i think you can write a programm that detects that
<beneroth>
T
<Regenaxer>
In C the stack pointer cannot be accessed, but you can do int dummy; Stack = &dummy;
<Regenaxer>
yes, same, take address of a local var
<freemint>
sorry that i joined so late, it might have saved you some time if i supported you earlier
<freemint>
there are other solution as well, like gcc function, or looking at process information under linux ... all are listed at there
<beneroth>
not really unusual for a picolisp debugging story, to be honest. such weird bugs are either extreme rare real bugs in pil - or, more likely, so stupid one would not have guessed *g*
<Regenaxer>
T
* beneroth
hides in shame about the one time he corrupted his execution stack...
<beneroth>
when the easy bugs are unlikely, only nasty bugs remain
<beneroth>
evolution at work :D
ubLIX has quit [Ping timeout: 248 seconds]
<Regenaxer>
I should have thought about the stack earlier, but I believed to have heard from tankf33der that he had already tested with unlimited
<beneroth>
yeah. well, happens.
<beneroth>
what was this now? 3 weeks brutto, maybe about 1-2 workday effort from you two?
<beneroth>
still not much. and as you use to say (and I agree), most time developing with picolisp is going into logic/conceptual work, not programming.
<Regenaxer>
Most time was invested by tankf33der
<beneroth>
others stack require much more and more periodically debugging efforts to get even simple things running :)
<Regenaxer>
T
<beneroth>
would be nice if we could make error reporting/diagnosis easier, but I guess the right way would be documentation (checklists), we don't want to pay the runtime overhead costs.
<Regenaxer>
right
<beneroth>
while this may get us from time to time, it's easily off-set by the gains. seriously.
<Regenaxer>
yep, runtime
<beneroth>
no way to solve this completely and in-depth on current environments (not without paying runtime overhead and high additional complexity costs we don't want to pay).
<beneroth>
looking forward to the picolisp machine and pilOS :)
<beneroth>
haha
<beneroth>
just got an idea
<Regenaxer>
pil64 does more checks, ie it checks the stack
<Regenaxer>
also because of coroutines
<beneroth>
also because the runtime costs for these checks are much lower in ASM, I guess?
<Regenaxer>
it is lower overhead in asm
<beneroth>
haha
<Regenaxer>
yep :)
<beneroth>
we have to train this more.
<beneroth>
accomplishing brain-sync.
<Regenaxer>
compare registers
<Regenaxer>
☺
<beneroth>
btw. I have an idea, we should get huawei to invest into pilOS :P they got money, and currently they also have a need :D
<beneroth>
(not happening, but yeah)
<Regenaxer>
good idea ;)
<beneroth>
;)
<clacke_movim>
narrator: "this was the pivotal comment, back in 2019, that would change mobile computing as we know it"
<freemint>
oh no then the chinese can use picolisp to spy on us ;)
ubLIX has joined #picolisp
<beneroth>
the chinese don't spy on other (or at least, no proof could be produced yet!). they spy on their own population, or at least want to ensure that they feel watched so they behave.
<beneroth>
not even a glimpse of a believable story of Chinese, or even more Huawei (which is NOT a state-owned enterprise, unlike many other) back-dooring equipment has yet surfaced.
<beneroth>
NSA hacked Huawei. GHCQ looked years for such a backdoor. nothing found.
<beneroth>
of course their subject to the usual stupid security lapses (e.g. uploaded passwords to github), but it's not comparable to e.g. Cisco (competitor for Huawei from USA) which "patches" obvious backdoors every 3 weeks or so, for 2+ years now.
<freemint>
First: They also spy on Taiwan. This is documented. Second ";)"
<clacke_movim>
PRC obviously spies on ROC and USA
<beneroth>
but not by backdooring
<beneroth>
no need with all the NSA security holes and exploits in the wild
<clacke_movim>
whether they do it via the active cooperation of Huawei is an open research question
<freemint>
";)" !
<beneroth>
well they didn't shut down UK medical facilities for days, which happened through stolen NSA "weapons" :)
<beneroth>
offensive IT "security" is not possible without weakening your own IT security defense at the same time.
<clacke_movim>
The story of the PLA infiltrating SuperMicro supply lines to insert nano wiretaps seemed pretty dubious and its the best evidence in this direction
<beneroth>
yes its the best story into this direction, and its superb dubios.
<beneroth>
probably not by intention, but blown-up rumours and projected fears.
<beneroth>
US Gov is a lot like a cheating partner in this regard, always suspecting the other to be as bad as themselves.
<beneroth>
(also to what-aboutism its own actions)
<beneroth>
just to be clear: I'm not taking sides. all sides suck. all sides have great accomplishment and true evil inside. but we must not be confused about reality, else we're unable to deal with it accordingly.
<beneroth>
kalisti.
<beneroth>
nice to have you more active here again, clacke_movim :)
<clacke_movim>
Ballistic, beneroth, Keeper of the Sacred Chao!
<beneroth>
thanks!
<clacke_movim>
Still not contributing to the actual ontopic talk, so thanks for the welcome :-)
<clacke_movim>
Haha, Kallisti -> Ballistic, my autocorrect is truly Erisian
<beneroth>
I fear I'm not capable of going the peaceful calm way of the Dealay Lama (unlike Regenaxer), so I guess I have I'll end up on the Celine way, which is bound to bad karma ;-)
<beneroth>
haha, I wondered what slang that is to comment my comment.
<clacke_movim>
The ballistic Apple of Discord
ubLIX has quit [Quit: ubLIX]
<beneroth>
btw. freemint, we don't need the Chinese to ruin us, we do it ourselves. german politicians are UPSET about the increased security planned/required in 5G (like TLSing all connections, and authenticate cell towers securely, so fake-cell towers cannot be used for MitM-attacks, = disabling IMSI-catching attack used by police)
<clacke_movim>
Message integrity is anti-democratic
<beneroth>
Ballistic Apple Chapter. good name. :D
<beneroth>
"Citizens of the world, in these times the future of mankind is at stake. [...] To sustain order and to increase our economic, social and mental efficiency, We've got to make sure, that we can rely on your cooperation in providing, Any requested data in a structured, detailed and predictable manner."
<beneroth>
"To eliminate any irregularities that may compromise flawless operation. We have declared war on error. so be warned:"
<beneroth>
"Random is resistance! - I repeat: random is resistance!"
<beneroth>
clacke_movim, can you give me an A? for Ballistic Apple Kult A? B.A.K.A. ?
<freemint>
beneroth, you seem very triggered
<beneroth>
freemint, I'm not. I'm happy procrastinating instead of doing my work :)
<freemint>
You defintly are
<freemint>
beneroth, offensive cybersecurity is possible without weakening the software you use. Just target software that the opponent uses but not you.
<freemint>
yeah but with specific kernel modules not seen in the wild
<beneroth>
yeah, but the exploits working against the kernel modules in the wild gets them too. and yourself. :)
<beneroth>
you cannot do offensive development without having ANY samples.
<beneroth>
so you take the one you know.
<beneroth>
thats the one you use yourself :)
<beneroth>
and you cannot patch it for your own, because the patches will be leaked, and reverse engineered.
<beneroth>
within hours.
<beneroth>
I see no way how " Just target software that the opponent uses but not you." can be done realistically.
<beneroth>
our nations just focus on pure defense and declare so publicly. that is also the only way you cannot be wrongly attributed to be the attacker in a "cyber war"
<beneroth>
*should
<freemint>
Yes ofc. But people can still claim germany attacked
aw- has quit [Quit: Leaving.]
<freemint>
Even the line between reasearch in to weaknesses and attacking is murky
<beneroth>
it's murky because you have in Germany this "Hacker paragraph" law.
<freemint>
yes but it is even mzrky in principle
<freemint>
Even if you hunt bugs and patch them you are still building a weapons aresenal
<beneroth>
and I suspect it's murky on purpose, not only (though this plays clearly a part) IT-illiterate politicians. murky laws are on the rise (same here in CH). murky laws are tool to transform democracy into tyranny, as murkiness creates space for arbitrariness.
<freemint>
it is fundamentally murly
<freemint>
*murky
<beneroth>
you have then law violations which are usually ignored, but if you want to get someone, you have all the legal measures you need.
<beneroth>
agreed, true.
<beneroth>
same as biological security research = e.g. creating more harmful diseases.
<beneroth>
<freemint> Even if you hunt bugs and patch them you are still building a weapons aresenal
<beneroth>
if you do it publicly, no then you are not. as you hand everyone the means to defend themselves at the same time, equalling the level.
<freemint>
Even if you publish 99 security bugs with patches who can prove you didn't kept the hunderth for future use. Can you prove that?
<beneroth>
no. the 100 is about trust.
<beneroth>
but we managed to solve this before, even when both parties mutually distrust each other absolutely.
<freemint>
And even if you are you still have a stock pile of weapons against old unpatched versions
<beneroth>
(nuclear disarmament)
<beneroth>
sure, but only because you don't research this security holes is no protection at all. somebody will do it eventually.
<freemint>
Destruction of bombs is easy to verify. Destructin of data is impossible to verify
<beneroth>
but having done the research but then NOT publishing it, that is weapon-building.
<beneroth>
no need to delete your exploit. a patch is often as good as an exploit. if you publish a patch, attackers will reverse engineer it.
<freemint>
even the research is weapon building
<beneroth>
yes, insofar as you cannot do defense building without weapon building. it's true both ways :)
<freemint>
the nsa took this arguement to it's other extreme
<freemint>
Everything is defensive
<beneroth>
yeah, attack is defense.
<beneroth>
war is peace.
* freemint
rolls his eyes over the distracting reference
<beneroth>
ideology and philosophical arguments do not matter, I say. only thing that matters is consequences, real-life results.
<beneroth>
most bad stuff is done following "good" ideologies.
<beneroth>
only the results matter.
<freemint>
except that is not how human sense og justive work or the legel system
<beneroth>
T - but technology works this way.
<freemint>
What do you mean by technology here?
<beneroth>
math. physics. computer science.
<beneroth>
it doesn't matter how much you believed to make a good program, if it leads to harm. no difference in technological sense.
<beneroth>
it does matter in legal if you intended to do harm or not.
<beneroth>
clacke_movim, you got a fan on the mailing list
<beneroth>
somebody interest in racket
<clacke_movim>
Cool, will check :-)
<beneroth>
:-)
<beneroth>
I'm also interested in which industries/use cases Racket is used. Haven't stumbled on it irl yet.
<freemint>
with this definition i agree
<beneroth>
freemint, thanks for asking to express myself more clearly!
<freemint>
here is another ugly effect: sometimes cyber defense requires active targeting and killing of enemy servers (in DoS attacks via missconfigured NTP servers for example)
<beneroth>
hm, I'm sceptical about this. the usual way is to "black-hole" such a source-IP, meaning banning it in firewall or route it to a /dev/null-destination on network level.
<beneroth>
hackback might sometimes be an option, but rarely so.
<beneroth>
if it's a clearly identifiable source, than you can blackhole it (if you got the infrastructure to handle the traffic - else you have to step closer to the attacker until you are).
<beneroth>
if its not a clearly identifiable source or too many of them (e.g. constantly switching IP-addresses, a bot-net), then hackback is also meaningless (unless you can find the C&C-servers and pwn them, but that is rather unlikely)
<freemint>
but hack back can help you find the C&C
<beneroth>
freemint, one issue I happen to have with operating internet servers is that one of my customers domain happened to be a C&C domain. todays botnet have a huge collection of C&C domains built-in, so that the C&C servers cannot be easily be found/shut off.
<freemint>
I think hack back is on the same level as Isreal bombing actual rocket launchs sites from the hamas.
<freemint>
yeah but you could look to which one he sends packages succesfull
<beneroth>
my customer was of course not an owner and no C&C server running, but countless infected accounts tried to ask this domain, effectively resulting in a DDoS against my server.
<freemint>
afk
<beneroth>
freemint, I don't think comparing "cyber" to real-life military is a working analogy. real-life military needs physical material, which is cost, and cannot be scaled easily - so harming that, is really impacting significantly the attacker. but that is not true for an internet attacker.
<beneroth>
freemint, I recommend https://ansuz.sooke.bc.ca/entry/23 "colour of bits" essay. it goes deeper into the general problems and difference between social systems (like legal) and technological systems.
freemint has quit [Read error: Connection reset by peer]
freemint has joined #picolisp
<freemint>
beneroth, but cyber needs material too the only thing that is different is the magnitude of the price
orivej has joined #picolisp
<freemint>
If you hack-back and run wireshark on the machine to see what attempted C&C connections work you can find the C&C better.
<freemint>
(simplified but true)
<freemint>
beneroth, remember the huge bot spam on FreeNode?
<beneroth>
aye. where they deployed port-scanning in the end?
<beneroth>
(which I would not see as hackback)
<beneroth>
(german legal sees it different though...)
<beneroth>
I've gotta go. will read when I'm back in 2-3 hours. thanks for the discussion :)
<beneroth>
bbl
<freemint>
to my understandng the reason why they could not bann all spammers is that there were many irc clients with a weakness out there which was exploitable via PM,
<beneroth>
ah, ok, I didn't look that deep into it. yeah, IRC is back from when everything was naively trusted.
<freemint>
now everytime you connect freenode makes an pre-emptive hack back by trying to exploit this weakness