Project

General

Profile

Actions

Bug #5921

open

simtrace2 cardem vs. Linux kernel USB autosuspend

Added by laforge about 1 year ago. Updated 6 months ago.

Status:
In Progress
Priority:
High
Assignee:
Category:
-
Target version:
-
Start date:
02/23/2023
Due date:
% Done:

10%

Spec Reference:

Description

(at least) after the following patch was merged, simtrace2-cardem doesn't work with Linux kernels' USB autosuspend anymore:

commit a5d537973db9359804e82a506057f3dd6d53fab9
Author: Harald Welte <laforge@osmocom.org>
Date:   Mon Jul 25 19:59:08 2022 +0200

    cardem: reset the uC in case of USB disconnect

The problem is that the USB kernel notices the simtrace2 device is not in use (no application using it), which in turn means it will power-down the device after 5s. The device then recognizes this as USB disconnect, and we use that to reset the firmware:

=============================================================================
SIMtrace2 firmware 0.8.1.58-773d, BOARD=simtrace, APP=cardem
(C) 2010-2019 by Harald Welte, 2018-2019 by Kevin Redon
=============================================================================
-I- Chip ID: 0x299b0a60 (Ext 0x00000000)
-I- Serial Nr. 44203020-48574336-30303931-32323032
-I- Reset Cause: software reset (processor reset required by the software)
-I- USB init...
USBD_Init
SetAddr(22) -W- Sta 0x888A8 [0] -W- _ -W- Sta 0x888A8 [0] -W- _ -W- Sta 0x888A8 [0] -W- _ SetCfg(1) cfgChanged1 -I- calling configure of all configurations...
-I- calling init of config 1...
-I- Modem 0: physical SIM
-I- 0: Use local/physical SIM
-I- entering main loop...
-I- USB is now configured
-I- Resetting uC on USB disconnect

=============================================================================
SIMtrace2 firmware 0.8.1.58-773d, BOARD=simtrace, APP=cardem
(C) 2010-2019 by Harald Welte, 2018-2019 by Kevin Redon
=============================================================================
-I- Chip ID: 0x299b0a60 (Ext 0x00000000)
-I- Serial Nr. 44203020-48574336-30303931-32323032
-I- Reset Cause: software reset (processor reset required by the software)
-I- USB init...
USBD_Init
SetAddr(23) -W- Sta 0x888A8 [0] -W- _ -W- Sta 0x888A8 [0] -W- _ -W- Sta 0x888A8 [0] -W- _ SetCfg(1) cfgChanged1 -I- calling configure of all configurations...
-I- calling init of config 1...
-I- Modem 0: physical SIM
-I- 0: Use local/physical SIM
-I- entering main loop...
-I- USB is now configured
-I- Resetting uC on USB disconnect

This in turn will make the device enumerate and re-enumerate in 5s cycles:

[585591.174222] usb 1-1: new full-speed USB device number 84 using xhci_hcd
[585591.330180] usb 1-1: New USB device found, idVendor=1d50, idProduct=60e3, bcdDevice= 0.02
[585591.330216] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=11
[585591.330231] usb 1-1: Product: SIMtrace 2
[585591.330242] usb 1-1: Manufacturer: sysmocom - s.f.m.c. GmbH
[585591.330253] usb 1-1: SerialNumber: 44203020485743363030393132323032
[585594.759881] usb 1-1: USB disconnect, device number 84
[585595.530214] usb 1-1: new full-speed USB device number 85 using xhci_hcd
[585595.682690] usb 1-1: New USB device found, idVendor=1d50, idProduct=60e3, bcdDevice= 0.02
[585595.682697] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=11
[585595.682701] usb 1-1: Product: SIMtrace 2
[585595.682704] usb 1-1: Manufacturer: sysmocom - s.f.m.c. GmbH
[585595.682706] usb 1-1: SerialNumber: 44203020485743363030393132323032
[585598.802549] usb 1-1: USB disconnect, device number 85
[585602.158170] usb 1-1: new full-speed USB device number 86 using xhci_hcd
[585602.313720] usb 1-1: New USB device found, idVendor=1d50, idProduct=60e3, bcdDevice= 0.02
[585602.313757] usb 1-1: New USB device strings: Mfr=1, Product=2, SerialNumber=11
[585602.313772] usb 1-1: Product: SIMtrace 2
[585602.313784] usb 1-1: Manufacturer: sysmocom - s.f.m.c. GmbH
[585602.313795] usb 1-1: SerialNumber: 44203020485743363030393132323032
[585606.602416] usb 1-1: USB disconnect, device number 86

Related issues

Related to SIMtrace 2 - Bug #5578: osmo-remsim-client-st2 hangs after usb-reset without power loss on qmodStalledlaforge06/15/2022

Actions
Actions #1

Updated by laforge about 1 year ago

  • Related to Bug #5578: osmo-remsim-client-st2 hangs after usb-reset without power loss on qmod added
Actions #2

Updated by laforge about 1 year ago

  • Status changed from New to In Progress
  • % Done changed from 0 to 10

so the easy work-around is to disable autosuspend by something like

echo 'on' > /sys/bus/usb/devices/usb8/8-1/power/control

which can be made persistent via an udev rule like this
ACTION=="add", SUBSYSTEM=="usb", ATTR{idVendor}=="1d50", ATTR{idProduct}=="60e3", TEST=="power/control", ATTR{power/control}="on" 

The proper solution is of course to fix the firmware, but the workaround will buy us some time.

Actions #3

Updated by mschramm about 1 year ago

N.B.: besides QMOD, also ngff-cardem is affected. - Last version which does not contain this problem is 0.8.1.35-fdfb.

Actions #4

Updated by laforge about 1 year ago

On Fri, Feb 24, 2023 at 04:23:48PM +0000, mschramm wrote:

N.B.: besides QMOD, also ngff-cardem is affected. - Last version which does not contain this problem is 0.8.1.35-fdfb.

did you confirm the "re-enumerates every 5s" behavior on some other machine?

Actions #5

Updated by mschramm about 1 year ago

laforge wrote in #note-4:

did you confirm the "re-enumerates every 5s" behavior on some other machine?

Have to verify this again as it might have been a wrong observation during testing of a certain ngff-cardem PCBA which initially had another unrelated problem.

Actions #6

Updated by mschramm 7 months ago

laforge wrote in #note-4:

N.B.: besides QMOD, also ngff-cardem is affected. - Last version which does not contain this problem is 0.8.1.35-fdfb.

did you confirm the "re-enumerates every 5s" behavior on some other machine?

yes - confirmed on another host, different ngff-cardem PCBAs and several recent versions:
  • 0.8.1.35-fdfb is last version which does not contain this problem (as already stated above)
  • 0.8.1.36-a5d53, 0.8.1.58-773d and 0.8.1.65-1f77 (most recent version by now) reset in 5s cycles on an Linux host (5.15 kernel)
Actions #7

Updated by mschramm 7 months ago

mschramm wrote in #note-6:

yes - confirmed on another host, different ngff-cardem PCBAs and several recent versions:
  • 0.8.1.35-fdfb is last version which does not contain this problem (as already stated above)
  • 0.8.1.36-a5d53, 0.8.1.58-773d and 0.8.1.65-1f77 (most recent version by now) reset in 5s cycles on an Linux host (5.15 kernel)

strange thing ist - on elab2 (Debian 9, kernel 4.9), I can not reproduce it: a PCBA with fw 0.8.1.58-773d does not cycle. Same PCBA on a Mint 21.2 host (kernel 5.15) cycles...

Actions #8

Updated by mschramm 7 months ago

mschramm wrote in #note-7:

yes - confirmed on another host, different ngff-cardem PCBAs and several recent versions:
  • 0.8.1.35-fdfb is last version which does not contain this problem (as already stated above)
  • 0.8.1.36-a5d53, 0.8.1.58-773d and 0.8.1.65-1f77 (most recent version by now) reset in 5s cycles on an Linux host (5.15 kernel)

strange thing ist - on elab2 (Debian 9, kernel 4.9), I can not reproduce it: a PCBA with fw 0.8.1.58-773d does not cycle. Same PCBA on a Mint 21.2 host (kernel 5.15) cycles...

confirmed exact same thing now for QuadModem, for Debian 9 vs. Mint 21, QMOD fw 0.8.1.35 (stable on both hosts) and 0.8.1.36 (target resets, enum cycles on Mint).

Actions #9

Updated by Hoernchen 7 months ago

Is this a kernel feature, or just a misconfigured mint system with TLP or some other laptop power save package, see for example https://wiki.archlinux.org/title/TLP ?

Actions #10

Updated by mschramm 7 months ago

Hoernchen wrote in #note-9:

Is this a kernel feature, or just a misconfigured mint system with TLP or some other laptop power save package, see for example https://wiki.archlinux.org/title/TLP ?

tlp with USB_AUTOSUSPEND=1 does not mean a misconfigured system. But yes: toggling it to off state avoids the mentioned behaviour.

What interests me more, is how this came to our knowledge. AFAIR it was reported by a customer (maybe rk?) but could be reproduced by laforge (?).
If it was rk, then tlp itself can't be the root cause.

Actions #11

Updated by Hoernchen 6 months ago

https://www.kernel.org/doc/html/v6.5/driver-api/usb/power-management.html

For this reason, by default the kernel disables autosuspend (the power/control attribute is initialized to on) for all devices other than hubs. Hubs, at least, appear to be reasonably well-behaved in this regard.

(In 2.6.21 and 2.6.22 this wasn't the case. Autosuspend was enabled by default for almost all USB devices. A number of people experienced problems as a result.)

Also true for usb3/l1 lpm.

Tlp just knows better and turns it on.

might be possible to fix this in the rules by adding

, TEST=="power/control", ATTR{power/control}:="on" 

or
, ATTR{power/autosuspend}="-1" 

to the end of all /etc/udev.d/99-simtrace.rules lines - unless tlp just ignores all that and needs a seperate blacklist file.

Actions

Also available in: Atom PDF

Add picture from clipboard (Maximum size: 48.8 MB)