[Zaurus-devel] Fwd: Memory corruption with 2.6.32.10, but not with 2.6.34-rc3

[Andrea Adami]

Forwarded conversation
Subject: Memory corruption with 2.6.32.10, but not with 2.6.34-rc3
------------------------

From: *Daniel Mack* <daniel at caiaq.de>
Date: Thu, Apr 1, 2010 at 3:21 PM
To: LKML <linux-kernel at vger.kernel.org>, AKML <
linux-arm-kernel at lists.infradead.org>, linux-wireless at vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh at suse.de>, Sven Neumann <
s.neumann at raumfeld.com>


Hi,

we observed repeated occurances of memory corruptions (Ooopes somewhere
deep down in the memory mangement code) on ARM PXA300 based boards.

The systems we see this on (arch/arm/mach-pxa/raumfeld.c) feature a
libertas chipset for WiFi, an ethernet controller (smsc9220), a USB
fullspeed host, and NAND flash which is used as UBIFS storage.

Currently, these boards run a 2.6.32.10 kernel. After collecting
evidences for a week or so about when and how and why the memory
corruptions happen, I tried a 2.6.34-rc3 today and the issue seems fixed
there. So - appearantly some important fix since 2.6.32 didn't get
enough care to be backported to the stable branch.

The bug is rather hard to trigger. What I currently do is: after the
system booted from NAND (UBIFS root partition), I wait for the WPA2
secured WiFi link to get active and then download a file (~8MB) over
WiFi to local storage. This download is done in an endless loop. Once in
a while this crashes the 2.6.32.10 kernel instantly, sometimes it takes
up to ~5hrs to happen.

Some findings I collected over the last weeks:

 - when calling wget with '-O /dev/null' to not write any file
  -> does NOT crash

 - downloading via Ethernet instead of WiFi
  -> does NOT crash

 - writing the file to either a tmpfs parition or a fatfs (on USB
  connected external media)
  -> DOES still crash (so it is most likely not an UBIFS issue)

 - passing --download-rate=50000 to wget (to limit the traffic
  thruput to 50kb/s) _in_creases the probability of the crash

 - running userspace applications which heavily allocate and
  deallocate memory doesn't seem to make the bug more likely or
  unlikely

So my current summary is that this is related to WiFi, but OTOH it still
only happens when file system traffic is issued.

We would like to have a fix for this annoying bug in the stable series
(especially 2.6.32.x) as well, but I don't have much ideas about where
to search for it. Hence, I would appreciate if maintainers could think
about any possible commits in the described time window which haven't
reached stable. Does the description ring anyone's bell?

I can cherry-pick things if anyone pin-points something and run
lont-time tests again. Any pointer appreciated.

Thanks,

----------
From: *Greg KH* <gregkh at suse.de>
Date: Thu, Apr 1, 2010 at 6:51 PM
To: Daniel Mack <daniel at caiaq.de>
Cc: Sven Neumann <s.neumann at raumfeld.com>, linux-wireless at vger.kernel.org,
LKML <linux-kernel at vger.kernel.org>, AKML <
linux-arm-kernel at lists.infradead.org>


Oh, how about running 'git bisect' to try to find the solution?  Just
remember to reverse 'good' and 'bad' for when you tell git bisect what
the results are.

thanks,

greg k-h

----------
From: *Greg KH* <gregkh at suse.de>
Date: Thu, Apr 1, 2010 at 6:50 PM
To: Daniel Mack <daniel at caiaq.de>
Cc: Sven Neumann <s.neumann at raumfeld.com>, linux-wireless at vger.kernel.org,
LKML <linux-kernel at vger.kernel.org>, AKML <
linux-arm-kernel at lists.infradead.org>


I can't think of any USB specific patches that would be related to this,
sorry.

good luck,

greg k-h

----------
From: *Daniel Mack* <daniel at caiaq.de>
Date: Thu, Apr 1, 2010 at 6:58 PM
To: Greg KH <gregkh at suse.de>
Cc: Sven Neumann <s.neumann at raumfeld.com>, linux-wireless at vger.kernel.org,
LKML <linux-kernel at vger.kernel.org>, AKML <
linux-arm-kernel at lists.infradead.org>


Jep, I thought about that of course. But unfortunately, the platform
got merged mainline in the middle of that time window which makes
bisecting tricky. And worse than that - every test run take around half
a day at least :(

Daniel

----------
From: *Daniel Mack* <daniel at caiaq.de>
Date: Thu, Apr 1, 2010 at 7:00 PM
To: Greg KH <gregkh at suse.de>
Cc: Sven Neumann <s.neumann at raumfeld.com>, linux-wireless at vger.kernel.org,
LKML <linux-kernel at vger.kernel.org>, AKML <
linux-arm-kernel at lists.infradead.org>


Yes, I'd rule out USB anyway. It crashes without any USB function as
well. I just copied you as the maintainer of the stable tree :)

Daniel

----------
From: *Anders Grafström* <grfstrm at users.sourceforge.net>
Date: Thu, Apr 1, 2010 at 7:29 PM
To: Daniel Mack <daniel at caiaq.de>
Cc: linux-wireless at vger.kernel.org, Greg Kroah-Hartman <gregkh at suse.de>,
Sven Neumann <s.neumann at raumfeld.com>, LKML <linux-kernel at vger.kernel.org>,
stable at kernel.org, AKML <linux-arm-kernel at lists.infradead.org>


You could try this one:

http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=commit;h=8e4971f2fb2380ce66196136e113d04196b80fcd

----------
From: *Daniel Mack* <daniel at caiaq.de>
Date: Thu, Apr 1, 2010 at 7:57 PM
To: Anders Grafström <grfstrm at users.sourceforge.net>
Cc: linux-wireless at vger.kernel.org, Greg Kroah-Hartman <gregkh at suse.de>,
Sven Neumann <s.neumann at raumfeld.com>, LKML <linux-kernel at vger.kernel.org>,
stable at kernel.org, AKML <linux-arm-kernel at lists.infradead.org>


Thanks. Unfortunately, that didn't fix it (this time, it crashed after a
few minutes only). But shouldn't we have that in -stable anyway?

Daniel
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxtogo.org/pipermail/zaurus-devel/attachments/20100404/81f5ae52/attachment.htm>