16:03 <pigiron> I'm having some fear and loathing with my ODroid-C2 ...

16:04 <pigiron> After upgrading the Arch Linux mainline kernel from 5.2 to 5.3, the ethernet (eth0) won't come up.

16:04 <pigiron> The key line in dmesg seems to be:

16:05 <pigiron> "meson8b-dwmac c9410000.ethernet: Failed to reset the dma"

16:06 <pigiron> That gets repeated again and again.

16:08 <pigiron> That line comes from the stmmac_init_dma_engine() function in drivers/net/ethernet/stmicro/stmmac/stmmac_main.c

16:10 <pigiron> But... after reverting commit f29cabf240ed6e67993f17594e5e6fffc5bc07e0 then life is good again.

16:14 <pigiron> Anyone else with a "C2" running 5.3 seeing this?

17:23 <xdarklight> pigiron: strange, that should not happen

17:24 <xdarklight> pigiron: Kevin's KernelCI lab doesn't seem to have an Odroid-C2 anymore but there's the GXBB P200 board (same SoC, also Gbit/s Ethernet, just a Micrel PHY instead of a Realtek one) and it seems to work fine: https://storage.kernelci.org/next/master/next-20191010/arm64/defconfig/clang-8/lab-baylibre/boot-meson-gxbb-p200.html

17:26 <xdarklight> can you compare your kernel .config with what the P200 board uses (https://storage.kernelci.org/next/master/next-20191010/arm64/defconfig/clang-8/kernel.config)?

17:34 <pigiron> I sure will... give me a minute... and thanks!

18:02 <pigiron> There's a massive difference. So I narrowed it to the dwmac stuff by doing this:

18:02 <pigiron> diff c2-kernel.config p200-kernel.config |grep -e STMICRO -e STMMAC -e DWMAC

18:03 <pigiron> And the result is:

18:03 <pigiron> < CONFIG_DWMAC_DWC_QOS_ETH=m

18:03 <pigiron> > # CONFIG_DWMAC_DWC_QOS_ETH is not set

18:03 <pigiron> > CONFIG_DWMAC_SOCFPGA=m

18:08 <pigiron> I *assume* that the failing dma reset was in the dwmac code/area. Perhaps I'm wrong.

18:16 <pigiron> And yea, and I was bummed when I checked kernelci about a week ago, and there weren't any Odroid-C2 results.

19:01 <pigiron> I compared the "meson8b-dwmac" output from that P200's dmesg output on KernelCI and that part is identical with mine...

19:02 <pigiron> Except, mine has later "meson8b-dwmac" dmesg output and then it starts failing.

19:03 <pigiron> I pasted the journalctl ouput from my C2 here: http://dpaste.com/38FX689

20:06 <xdarklight> pigiron: can you pastebin your .config as well please?

20:07 <pigiron> Certainly. Give me a minute...

20:11 <pigiron> Here it is (and thanks for the help!!): http://dpaste.com/35WH92X

20:12 <Xogium> pigiron: are you running on an eMMC ?

20:12 <Xogium> asking because I'm tempted to migrate to 5.3 kernel but still unsure about it since I run on eMMC

20:15 <pigiron> Yes, and I have the one that was failing with the 5.2 kernel. That's why I was waiting so much for 5.3, because that problem is fixed in that version. And then I immediately ran into this problem :-(

20:16 <Xogium> pigiron: hah

20:16 <Xogium> pigiron: did the speed improve on eMMC ? Or does it still gets limited at 80 mb/s or so ?

20:16 <Xogium> I can try the 5.3 kernel too :)

20:17 <pigiron> hmm... I've never tested the speed. But it sure "feels" a LOT faster than the SD card.

20:17 <Xogium> oh yeah it is faster

20:17 <Xogium> hdparm -Tt can give you the speed

20:17 <Xogium> both buffered and non-buffered

20:19 <pigiron> You can read the long and sad eMMC story here (but it has a happy ending):

20:20 <pigiron> https://archlinuxarm.org/forum/viewtopic.php?f=23&t=13495&sid=ef8fe7aa9ba12f59d8648012e882f92c

20:20 <Xogium> good

20:20 <Xogium> I was still on 4.19

20:23 <Tartarus> FWIW, eMMC works on my S905 board but the v5.2 kernel in meta-meson seems very unstable on it

20:23 <Xogium> yeah I'm not gonna try 5.2

20:23 <Tartarus> I'm about to crack open another one of these boxes and see if it's just the one I'm using or a real problem.

20:24 <Tartarus> Some general concerns or ?

20:25 <Tartarus> I'm looking at just backporting the driver to 4.20 for now as that was fine

20:27 <pigiron> All I can say is that eMMC Version 5.1 HS400 was failing for some of us on 5.2. Yet some (undefined) eMMC chips seemed to work.

20:27 <xdarklight> pigiron: can you do an experiment with your kernel config? please replace CONFIG_HZ_100=y with CONFIG_HZ_250=y and set CONFIG_HZ to 250 (your old value is 100), rebuild your kernel and test if that "fixes" (rather works around) the issue

20:27 <Tartarus> pigiron: unstable or just not showing up?

20:28 <Tartarus> actually scratch that, I'm crashing writing to the SD card anyhow

20:28 <pigiron> xdarklight: You bet. But that will take a long time. Can I give the results tomorrow?

20:29 <xdarklight> pigiron: sure - it's only an experiment and may lead to nothing but there was an Odroid-C1 user some time ago who reported that this fixes a similar problem

20:30 <pigiron> xdarklight: Good. That's encouraging ;)

20:31 <xdarklight> pigiron: I made a similar change that's problematic for you on Odroid-C1 and tested that on my own board and in my tests it worked fine regardless of CONFIG_HZ set to 100 or 250. I've also tested it on a Khadas VIM2 with similar Ethernet setup as your Odorid-C2 and it was working fine for me. however, I've learned that just because it works for me doesn't mean it works for everyone ;)

20:31 <Xogium> xdarklight: well I'm starting a build, do you want me to do this too ?

20:31 <Xogium> odroid c2 from 2016 -- I don't know what rev of the board that is

20:31 <pigiron> Tartarus: It fails to boot 100% of the time... if you have the "wrong" chip.

20:32 <Tartarus> pigiron: ah, so not this problem then, ok

20:32 <Tartarus> thanks

20:33 <Xogium> eMMC are awesome, at least the ones hk is using, still going strong and reproting no issue after almost 4 years of 24/7 usage

20:33 <Xogium> reporting

20:34 <pigiron> xdarklight: Chuckle. Now that you say that, I was looking at that P200 output on KernelCI and I don't think it captures *everything*...

20:34 <xdarklight> Xogium: I'm not sure - I'm more curious to see whether CONFIG_HZ=100 works for you, but you could end up with broken Ethernet (in theory, not sure how high the chances are). if it does then pigiron's issue may be a timing problem "somewhere"

20:34 <Xogium> nod

20:34 <Xogium> will left at default first and see how it goes

20:34 <Xogium> *leave

20:36 <Xogium> hmm. I should really reinstall my system, one of these days

20:36 <xdarklight> pigiron: indeed, but it has this: "Starting network: OK"

20:38 <pigiron> xdarklight: Ahh, OK. Its missing the "Link is Up" or "Link is Down" type messages I find on my box (failing or not).

20:39 <Xogium> so issue is that you get link is up, then it goes downafter that dma thing, all in a loop ? So up, down, up, down again and again ?

20:40 <pigiron> Then things are good, it's UP, DOWN, and UP. When it fails, it's UP, DOWN, and never comes UP again.

20:40 <pigiron> s/Then/When/

20:41 <Xogium> ah

20:41 <Xogium> alright

20:45 <Xogium> woah 24 days uptime on my c2 ? I didn't even notice

21:04 <pigiron> Xogium: Yes, you'll need the USB dongle. If/when it fails, revert commit f29cabf240ed6e67993f17594e5e6fffc5bc07e0. If it *doesn't* fail, please let us know. I'm not sure if my box is unique with this problem. But I really doubt it. It's failed on the Arch 5.3.1 and 5.3.5 kernels for me.

21:06 <Xogium> alrighty

21:18 <pigiron> xdarklight: FYI - I just remembered that I've got an appointment tomorrow morning. So I might not get back with results until Friday afternoon. Maybe around happy hour ;-) But I'm not in any rush. And once again, a HUGE thanks for looking into this!!!