Skip to content

[BUG] If ChainDMA is enabled FW crashes during multiple-pause-resume test #7084

Closed

Description

@jsarha

Describe the bug
If ChainDMA is enabled in topology the multiple-pause-resume test fails in CI:
https://sof-ci.01.org/sofpr/PR7073/build3851/devicetest/index.html

To Reproduce
Enable ChainDMA for TGL by reverting c3807ae and run on TGL upx-i11:
TPLG=/lib/firmware/intel/avs-tplg/sof-hda-generic.tplg MODEL=TGLU_UP_HDA_IPC4ZPH ~/work/sof-test/test-case/multiple-pause-resume.sh -r 50

Reproduction Rate
Does not always fail immediately, but sometimes takes over ten seconds. However, the FW always crashes well before the test completes.

Expected behavior
multiple-pause-resume test would pass.

Impact
Can not enable ChainDMA when its broken and no multichannel audio without ChainDMA.

Environment

  1. Branch name and commit hash of the 2 repositories: sof (firmware/topology) and linux (kernel driver).
    • Kernel: {aa9bd450439db90f28c438641d159a3dd8c41297}
    • SOF: {7b8da69} with c3807ae reverted.
  2. Name of the topology file
    • Topology: {sof-hda-generic.tplg}
  3. Name of the platform(s) on which the bug is observed.
    • Platform: {TGL upx-i11}

Screenshots or console output
Please refer to: https://sof-ci.01.org/sofpr/PR7073/build3851/devicetest/index.html

I have executed the test several times and there is no clear pattern to be seen on what happened on FW logs before the crash. However, from the Linux side logs its visible that the crash always happens while handling ChainDMA messages:
Feb 13 15:35:58 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe040000|0x0
...
Feb 13 16:19:46 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe050101|0x0
...
Feb 13 16:26:02 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe070101|0x0
...
Feb 13 22:13:53 jhary kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc timed out for 0xe050000|0x0

Activity

  1. added
    bugSomething isn't working as expected
    on Feb 13, 2023
  2. jsarha commented on Feb 13, 2023

    @jsarha
    ContributorAuthor

    This thing is annoyingly getting out of hand. This latest problem was not there couple of weeks back when I was tracking some earlier issue with ChainDMA, and its not a result of a change in the chain dma code, since the only change there is 6009c21 and reverting it does not help with the issue. We can not have ChainDMA enabled because its broken, and because its not enabled, it gets even more broken.

  3. jsarha commented on Feb 14, 2023

    @jsarha
    ContributorAuthor

    This blocks #7073

  4. self-assigned this
    on Feb 14, 2023
  5. added
    P1Blocker bugs or important features
    on Feb 14, 2023
  6. added this to the v2.5 milestone on Feb 14, 2023
  7. makarukp commented on Feb 15, 2023

    @makarukp
    Contributor

    @jsarha Please provide full IPC logs with payloads (for example for init instance) for local reproduction.

  8. jsarha commented on Feb 15, 2023

    @jsarha
    ContributorAuthor

    I can of course provide a full Linux side log, or even FW log, but I doubt that will help you to track down the problem. The multiple-pause-resume test is a stress test in nature, and there hundreds of IPC message before the crash happens.

  9. makarukp commented on Feb 15, 2023

    @makarukp
    Contributor

    We are interrested in full linux IPC log because we would like to reproduce the exact flow with specific ipc timings and all payloads necessary for module initializations. We would like to identify porential dependencies and detailed FW state before chain dma running. It is also worth to check how many chain DMA IPCs in average is needed to reproduce the problem .... (would be helpful )

    So far we haven't encountered reproduction applying clear stress test on logs you provided (only chain dma part below)...
    I think in case not send to much data you can filterout all background information except IPCs.

    [ 592.571174] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x180: GLB_CHAIN_DMA
    [ 592.571893] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 592.672310] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x180: GLB_CHAIN_DMA
    [ 592.672814] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 592.773227] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 592.773831] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 592.938600] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 592.939796] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 593.040179] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 593.040561] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 593.041020] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 593.042290] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 593.210475] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 593.211732] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 593.312100] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 593.312571] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 593.412984] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 593.413593] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 593.583423] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 593.584529] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 593.684979] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 593.685762] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 593.686113] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 593.687523] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 593.787914] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe040101|0x0: GLB_CHAIN_DMA
    [ 593.788350] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe040000|0x0: GLB_CHAIN_DMA
    [ 594.604371] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x180: GLB_CHAIN_DMA
    [ 594.605109] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 594.705582] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x180: GLB_CHAIN_DMA
    [ 594.706016] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 594.706450] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 594.707745] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 594.786958] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 594.788402] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 594.870971] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 594.872267] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 594.874045] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 594.875247] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 594.957035] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 594.958277] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 594.959983] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 594.962124] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 595.043999] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 595.047072] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 595.048011] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 595.049415] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 595.131977] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050000|0x0: GLB_CHAIN_DMA
    [ 595.133288] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070000|0x0: GLB_CHAIN_DMA
    [ 595.134987] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA
    [ 595.140212] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe070101|0x0: GLB_CHAIN_DMA
    [ 595.140632] kernel: snd_sof:sof_ipc4_log_header: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx : 0xe050101|0x0: GLB_CHAIN_DMA

  10. jsarha commented on Feb 15, 2023

    @jsarha
    ContributorAuthor

    The full logs are attached:
    linux-log.txt
    fw-log.txt

  11. jsarha commented on Feb 15, 2023

    @jsarha
    ContributorAuthor

    I spent the whole day desperately trying to find the setup that I recall recently passing multiple-pause-resume test, but I could not find it. However, if I go back far enough I can still find a working version. Last version that works is d92f07a, but the next commit 19df10f breaks the pause-resume functionality completely. The pause-resume starts working again after fc501b1 and 7383cbe, but that version crashes in multiple-pause-resume test at the same place as the current main branch. Same happens also, if I cherry-pick relevant commits on top off 19df10f.

  12. jsarha commented on Feb 15, 2023

    @jsarha
    ContributorAuthor

    Made this PR out of curiosity, just to see if the old implementation would still pass the whole CI: #7104

  13. jsarha commented on Feb 21, 2023

    @jsarha
    ContributorAuthor

    It does not look like #7138 fixes the issue. Still trying to root cause what is actually wrong with zephyr_ll.c, but so far no luck. The situation is such that there is two or three HDMI devices banging the zephyr_ll work queue by adding and removing tasks there in tight loop, and eventually something goes wrong, but its pretty hard to say where and how.

    It seems that my #7073 somehow gets past the issue by some luck, but maybe a better approach would be not to init() and free the tasks all the time, but instead init the task only once. Then just schedule the task in chain_task_start() and cancel it in chain_task_pause(). The should ease up the stress caused by the multiple-suspend-resume & chain DMA on the zepyhyr_ll queue.

    Still it would be so nice to find the actual root cause.

  14. jsarha commented on Feb 22, 2023

    @jsarha
    ContributorAuthor

    I could not get the schedule() cancel() change to work out of the box... something gets out of sync, not sure what. But I have to drop this now, since there is other work piling up. #7073 is still the only "fix" that solves the chain DMA multiple-pause-resume issue, too bad nobody understands why that works, and more importantly why the other versions do not.

  15. makarukp commented on Mar 3, 2023

    @makarukp
    Contributor

    Correct fix in #7138 and #7189. Issue is resolved.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

P1Blocker bugs or important featuresbugSomething isn't working as expected

Type

No type

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions