13:23 <wwilly> krzk, I'm at it again after a nice week-end. when I do as suggested, the first revision I hand up is: 9f68e3655aae6d49d6ba05dd263f99f33c2567af, and this one for instance loop somewhere, printing indefinitely samsung-i2s 3830000.i2s-sec: DMA channels sourced from device 3830000.i2s, but if I CONFIG_SND_SOC_SAMSUNG=n, because this log seems to comes from the sound card stuff, it's working well

13:24 <wwilly> after some other jump, hand up with a9eeb0e61128ed6cdd0b910eab2df7b2729d7d15 which kernel panic

13:49 <wwilly> 7d6292ab11199ef596cbe6c87180e49510c8b7c7 fault with panfrost bug sleep in atomic

13:57 <wwilly> and it kernel panic when I CONFIG_DRM_PANFROST=n

14:00 <wwilly> https://pastebin.com/kVLSjntd

14:12 <krzk> wwilly: you have unusually high amount of trbubles with bisect :)

14:12 <krzk> Most of revisions are bisectable and only exceptionally things get broken... and in your case it's quite a lot

14:13 <krzk> around panfrost indeed, I remember, there were some issues

14:14 <wwilly> with bisect should be almost straight forward right?

14:14 <krzk> wwilly: the first point - 9f68e3655aae6d49d6ba05dd263f99f33c2567af - is a merge commit. Sometimes merges with conflicts break things, sometimes merges discover issues... but more likely is that you annotated wrong revisions somehow :)

14:14 <krzk> a9eeb0e61128ed6cdd0b910eab2df7b2729d7d15 - again merge... that's weird

14:15 <krzk> and third - again merge... are you doing it correctly?

14:15 <wwilly> from git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git

14:15 <wwilly> git bisect start

14:15 <wwilly> git bisect bad v5.6

14:15 <krzk> It is highly unlikely that three different merges from different branches and trees (drm, arm-soc dt, arm-soc mach/soc) make troubles

14:16 <wwilly> git bisect good v5.5

14:16 <wwilly> then make exynos_defconfig; make; then boot with zImage and dtd produced

14:17 <wwilly> should I configure something else that you forgot to suggest?

14:55 <krzk> wwilly: no, it looks okay. Just check whether really you boot with that zImage and DTB (at least zImage by kernel version string)

14:56 <krzk> wwilly: on HC1 on all kernels between v5.5 and v5.6 were booting fine from network

14:57 <wwilly> it's booting from network, it's after the boot when you try to connect to it

14:57 <wwilly> I use tftp to boot

14:57 <krzk> so I mean, booted and worked fine (including SSH to it)

14:58 <wwilly> ok

14:58 <krzk> although I tested only RCs (so v5.5, v5.6-rc1, rc2 and so on)

14:58 <wwilly> hc1 use the same network chip?

14:58 <krzk> no, different

14:59 <krzk> but I am referring to your problems - panfrost, i2s etc.

14:59 <wwilly> ah ok

14:59 <krzk> none of them appeared on HC1 (which is possible... for example HC1 does not have audio codec so maybe that i2s stays disabled)

14:59 <wwilly> xu3 doesn't have audio either right?

15:00 <krzk> XU3 has but I do not have a working one anymore

15:00 <wwilly> ok I remember

15:01 <wwilly> is it possible that "wrong" value in dts for usb related stuff on the xu3 could cause this kind of weirdness?

15:02 <krzk> around your network - maybe, although unlikely, other issues look different - i2s, panfrost/gpu/drm and finally null pointer exception on mount_block_root

15:02 <wwilly> usb related mean from all voltage on the regulator to new feature...

15:03 <wwilly> yes ok

15:03 <krzk> I

15:03 <krzk> I'll test the 7d6292ab1119

15:03 <wwilly> I guess, new mid patch of a patch serie is not acceptable to leave a board panicking

15:09 <krzk> wwilly: heh, 7d6292ab1119 indeed fails

15:09 <krzk> on panfrost... there were issues in it fixed in some rc

15:10 <wwilly> ok

15:10 <wwilly> cool

15:10 <wwilly> I'm reassured that is not only me... :)

15:10 <krzk> another approach is to narrow your case by testing all RC and then starting bisect on narrower region

15:10 <krzk> e.g. if 5.5 is good, and 5.6 is bad... test v5.6-rc1, rc2 etc

15:11 <wwilly> ok, and use git bisect bad v5.6-rc1 ?

15:11 <wwilly> because you've suggested to not do that...

15:11 <krzk> I suggested not to bisect stables, RC are fine to include

15:11 <krzk> if 5.6-rc1 turns out bad, then yes - use it as bad as it narrows the search

15:12 <krzk> I am not sure if this approach will help much because most likely your issue is actually between v5.5 and v5.6-rc1 :)

15:13 <wwilly> it's actually between v5.5.16 which runs fine, but v5.6-rc1 doesn't run good

15:13 <wwilly> so it make sense to git bisect between v5.5.16 and v5.6-rc1?

15:13 <wwilly> or that is not correct?

15:14 <wwilly> or v5.5.16 is a stable version?

15:47 <krzk> did you test v5.5? if v5.5 was good, then the bisect between v5.5.16 and v5.6-rc1 will be the same as v5.5 - v5.6-rc1

15:47 <krzk> and will be actually the same what you already did (this commits you mentioned are before v5.6-rc1)

15:48 <krzk> wwilly: so in your case you are kind of screwed... the merge window was so bad that it is no in bisectable state

15:49 <wwilly> arf ok... f***k s**t

15:49 <wwilly> so, basically would the same as testing every commit myself?

15:50 <krzk> no

15:51 <krzk> wwilly: I mean, that testing v5.5.16 does not make sense. Bisecting v5.5 and v5.6-rc1 is the same for your case as v5.5 and v5.6 :)

15:51 <wwilly> ah ok

15:52 <krzk> let me test these few commits... probably the solution is to either find kernel config which does not have panfrost/snd issues and still reproduces your problem

15:52 <krzk> or apply specific fixes on top of bisected commits so these weird errors do not appear

15:52 <wwilly> when I have issue with panfrost/snd i just simply remove the option on .config

15:53 <wwilly> and if it kernel panic, I simply git bisect skip

15:53 <wwilly> I'm at 2e04d1bd540c849495c6f50d3c8086be824bd4e5

15:54 <wwilly> started from v5.5 to v5.6, I haven't tried from v5.6-rc1

15:54 <wwilly> and this last kernel panic

15:55 <wwilly> ... Bisecting: 315 revisions left to test after this (roughly 9 steps)

15:55 <wwilly> spent my night on this...

18:09 <wwilly__> there is also something anoying about bisect is that it doesn't really take into account useless change, some are only towards dts that is not targeting my board, could be an improvement for the tool

18:10 <wwilly__> like giving the defconfig that I'm using, and compute the right commit that impact zImage and dtb

18:10 <LiquidAcid> wwilly__, you can specifiy paths to git bisect

18:12 <wwilly__> LiquidAcid, sure, but if my prob with the particular lan device is impacted by a dts change in the power rail, giving the particular network subfolder path would be effective?

18:13 <LiquidAcid> wwilly__, https://stackoverflow.com/questions/31112179/git-bisect-with-list-of-uninteresting-paths <- that looks like it could solve your probem

18:13 <LiquidAcid> *problem

18:15 <wwilly__> LiquidAcid, I saw you posting on hardkernel forum a few time, by any change, do you have an xu3 board and tested v5.6?

18:18 <LiquidAcid> no, sorry, i only have an x2, which i haven't used in months

18:19 <wwilly__> ouké

19:21 <krzk> wwilly__: git bisect is a git tool, not kernel... how can it now anything about some Soc specific stuff? It is purely git history oriented, so what is actually stored in git does not matter for bisect

19:22 <krzk> You can tune bisect for mentioned by LiquidAcid paths or by using automatic commands to test revision (although with boots I guess it will be very difficult)

19:23 <wwilly__> yep I get it

19:23 <krzk> some commits are indeed broken, e.g. 7d6292ab1119 even without panfrost doe snot work in my case - reboots consantly around udev and coldplug

19:24 <krzk> unfortunately it seems that v5.6-rc1 merge window was pretty bad and not stable

19:29 <wwilly__> does this happen often, bad and not stable version?

19:29 <wwilly__> for such a big thing, I'm surprised about commits that breaks "everything"...

19:29 <wwilly__> I understand that is not possible to test EVERY device on earth, but still

19:31 <krzk> I must say I rarely bisect merge window because most errors we spot in linux-next and the bisecting happens between some RC and current next. There were times when many things were broken. were times that everything was mostly ok

19:33 <krzk> there are around 13 000 patches in each merge window... and most of them were not tested on more than few devices/configurations... so there is huge chance that something will be broken

19:34 <krzk> wwilly__: just for the record, XU, HC1 and Arndale Octa all boot fine on v5.6

19:34 <krzk> XU has I think the same network: 0424:9730

19:34 <wwilly__> ok, and rpi3 with a smsc95xx boot as well

19:35 <krzk> then it could be one damn small thing in DTS :)

19:35 <wwilly__> yes

19:35 <wwilly__> $ git bisect good

19:35 <wwilly__> [1019fe2c728003f89ee11482cf8ec81dbd8f15ba] ARM: dts: exynos: Adjust bus related OPPs to the values correct for Exynos5422 Odroids

19:35 <wwilly__> Bisecting: 1 revision left to test after this (roughly 1 step)

19:35 <wwilly__> nearly there ... :)

19:36 <wwilly__> it's like a few bad/good that it jumps between dts for exynos

19:44 <wwilly__> krzk,

19:44 <wwilly__> $ git bisect good

19:44 <wwilly__> 1019fe2c728003f89ee11482cf8ec81dbd8f15ba is the first bad commit

19:44 <wwilly__> Author: Marek Szyprowski <m.szyprowski@samsung.com>

19:44 <wwilly__> commit 1019fe2c728003f89ee11482cf8ec81dbd8f15ba

19:44 <wwilly__> Date: Thu Dec 19 11:51:30 2019 +0100

19:46 <wwilly__> is it possible to reverse just this without breaking anything after that?

19:50 <willmore> Only XU4 era hardware here.

19:54 <wwilly__> uhm ok

20:04 <Putti> Hi, I'm getting graphics glitches on exynos 4412 based i9305 board (samsung s3) with Lima driver. I would like to try to get the G3D PMU out of low power mode to try if that fixes the issue. I already clocked the mali GPU core at 50Mhz but that didn't help. Reading the exynos4 pmu driver code it is not very clear to me which bytes I need to set to the register to get out of low power mode.

20:10 <wwilly__> will continue investigating that tomorrow, but 1019fe2c728003f89ee11482cf8ec81dbd8f15ba is not working, and c6d0192afa24e315df501b1bd3f339a13ccb0692 is working

20:12 <wwilly__> thx krzk LiquidAcid and willmore, c ya tom

21:34 <Putti> I enabled the BUCK4 / vdd_g3d to be "regulator-always-on" and that didn't help, so I'm thinking the issue is somewhere else, maybe some bus clock issue

21:35 <Putti> any guesses?

21:36 <Putti> The glitches look exactly same as reported in https://gitlab.freedesktop.org/drm-hwcomposer/drm-hwcomposer/issues/24 and there the issue was fixed by disabling CONFIG_PM_DEVFREQ