Skip to content

[BUG] TGL_RVP_NOCODEC_ZEPHYR ipc timed out when TRIG_STOP with multiple-pipeline test #4680

Closed

Description

@kv2019i

Describe the bug
TGLH_RVP_NOCODEC_ZEPHYR ipc timed out when issuing GLB_STREAM_MSG: TRIG_STOP.

To Reproduce
TPLG=/lib/firmware/intel/sof-tplg/sof-tgl-h-nocodec.tplg ~/sof-test/test-case/multiple-pipeline.sh -f p -l 50

Reproduction Rate
100%

Environment
Kernel Branch: topic/sof-dev
Kernel Commit: eb55816743ecf7a68a94e0306d069c5e50783722
SOF Branch: main
SOF Commit: 7df3674
Topology:/lib/firmware/intel/sof-tplg/sof-tgl-h-nocodec.tplg
Platform: TGL_RVP_NOCODEC_ZEPHYR

Console output

[1374480.474916] sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[1374480.475125] sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: trigger stream 3 dir 0 cmd 0
[1374480.982904] sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x60050000 size 12
[1374480.982912] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump start ]------------
[1374480.982921] sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda irq intsts 0x00000000 intlctl 0xc0000781 rirb 00
[1374480.982923] sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspis 0x00000000
[1374480.982929] sof-audio-pci-intel-tgl 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[1374480.982931] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump end ]------------
[1374480.982933] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[1374480.982936] sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
[1374480.983031] sof-audio-pci-intel-tgl 0000:00:1f.3: error: unexpected fault 0x00000000 trace 0x00004000
[1374480.983033] sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[1374480.983053] sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0x160]=0x20140000 successful

Activity

  1. added
    bugSomething isn't working as expected
    ZephyrIssues only observed with Zephyr integrated
    TGLApplies to Tiger Lake
    TGL-HTGL-H platform
    on Aug 25, 2021
  2. self-assigned this
    on Aug 25, 2021
  3. changed the title [-][BUG][/-] [+][BUG] TGL_RVP_NOCODEC_ZEPHYR ipc timed out when TRIG_STOP with multiple-pipeline test[/+] on Aug 25, 2021
  4. XiaoyunWu6666 commented on Aug 26, 2021

    @XiaoyunWu6666
    Contributor

    in inner daily 6090, on TGLH_RVP_NOCODEC_ZEPHYR
    multiple-pipeline-all-15: ipc timed out when issuing GLB_DAI_MSG: CONFIG(msg/reply size: 216/12)
    check-runtime-pm-status-15: ipc timed out when issuing GLB_TPLG_MSG: PIPE_FREE(msg/reply size: 12/12)

  5. lgirdwood commented on Aug 26, 2021

    @lgirdwood
    Member

    @kv2019i do you suspect this is related to allocator or something else ?

  6. kv2019i commented on Aug 30, 2021

    @kv2019i
    CollaboratorAuthor

    @lgirdwood This is something else. When DSP is overloaded (>100% workload), processing of TRIG_STOP sometimes fails. The DSP doesn't crash and subsequent actions succeed. The overload situation does occur because we moved the audio buffers back to uncached now.

  7. lgirdwood commented on Aug 30, 2021

    @lgirdwood
    Member

    @kv2019i ok, lets wait for @lyakh triggering PR to merge. The 100% utilisation means we get in a state - do we get any data from the Zephyr LL scheduler ? i.e. is tick late or does work overrrun in logs ?

  8. ujfalusi commented on Aug 31, 2021

    @ujfalusi
    Contributor

    @lgirdwood This is something else. When DSP is overloaded (>100% workload), processing of TRIG_STOP sometimes fails. The DSP doesn't crash and subsequent actions succeed. The overload situation does occur because we moved the audio buffers back to uncached now.

    @kv2019i, the TRIG_STOP ipc message times out, Linux would think that the DSP is crashed (we get timeout on IPC message). I have pending new fw_state to be taken by Linux in this case and this would block further IPC communications until the DSP is rebooted.
    But in this case the DSP is just too busy to respond. Should the IPC timeout be treated as a feature from now on and only the panic message from DSP should be treated as crash? I would think that the firmware can crash in a not clean way to be able to tell Linux that it crashed..

  9. lgirdwood commented on Aug 31, 2021

    @lgirdwood
    Member

    @ujfalusi @kv2019i if FW is too busy to respond I would expect to see something in the logs to say this. e.g. LL should count number of times (per 1024 LL perf logging cycle) that there is little overhead for other work. This overhead could just be a subtraction of current time to next LL tick time and if it's less than limit we increment the count.

  10. kv2019i commented on Oct 18, 2021

    @kv2019i
    CollaboratorAuthor

    With #4884 , this issue should be finally fixed.

  11. lgirdwood commented on Oct 18, 2021

    @lgirdwood
    Member

    #4884 merged.

  12. keqiaozhang commented on Oct 19, 2021

    @keqiaozhang
    Collaborator

    @1994lwz , please help to verify this issue w/ latest firmware.

  13. 8 remaining items

  14. kv2019i commented on Oct 20, 2021

    @kv2019i
    CollaboratorAuthor

    @lyakh working on this. With TGL systems and HDA generic tplg, I can no longer locally reproduce it. Will need to retry with nocodec tplg, but due to some system config issues, this will take a bit of time.

  15. lgirdwood commented on Oct 26, 2021

    @lgirdwood
    Member

    @kv2019i can we close ?

  16. kv2019i commented on Oct 28, 2021

    @kv2019i
    CollaboratorAuthor

    No @lgirdwood . This is still happening. The test cases causes a 95% DSP load (on a single core). XTOS build can manage it, but with Zephyr we hit an xrun systematically and the test fails:

    First a lot of message from DMA that we are running out of data to process:

    [    11045636.748586] (        2663.802002) c0 dw-dma                 src/drivers/dw/dma.c:1081 INFO dw_dma_free_data_size() size is 0!
    [    11045659.873585] (          23.125000) c0 dai          3.14           src/audio/dai.c:950  WARN dai_copy(): nothing to copy
    [    11062863.518735] (       17203.644531) c0 dw-dma                 src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
    [    11062886.747900] (          23.229166) c0 dai          21.70          src/audio/dai.c:950  WARN dai_copy(): nothing to copy
    [    11082851.070024] (       19964.322266) c0 dw-dma                 src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
    [    11082874.299189] (          23.229166) c0 dai          21.70          src/audio/dai.c:950  WARN dai_copy(): nothing to copy
    [    11088642.996877] (        5768.697754) c0 dw-dma                 src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
    [    11088665.965626] (          22.968750) c0 dai          2.10           src/audio/dai.c:950  WARN dai_copy(): nothing to copy
    [    11089733.569750] (        1067.604126) c0 dw-dma                 src/drivers/dw/dma.c:1081 INFO dw_dma_free_data_size() size is 0!
    [    11089756.382249] (          22.812500) c0 dai          5.26           src/audio/dai.c:950  WARN dai_copy(): nothing to copy
    [    11089840.600996] (          84.218750) c0 dw-dma                 src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
    [    11089862.475995] (          21.875000) c0 dai          20.62          src/audio/dai.c:950  WARN dai_copy(): nothing to copy
    

    Then xrun:

    [    11125584.557909] (          37.656250) c0 host         21.64         src/audio/host.c:373  INFO no bytes to copy, 0 avail in buffer, 256 free in DMA
    [    11128500.599460] (        2916.041504) c0 dw-dma                 src/drivers/dw/dma.c:1114 ERROR dw_dma_get_data_size(): xrun detected
    [    11128625.130705] (         124.531242) c0 dai          21.70          src/audio/dai.c:882  ERROR dai_report_xrun(): overrun due to no space available
    [    11128771.328616] (         146.197906) c0 dai          21.70          src/audio/dai.c:744  ERROR comp_overrun(): sink->free = 512, copy_bytes = 0
    [    11128921.849443] (         150.520828) c0 pipe         21.71 ....../pipeline-stream.c:310  INFO pipe trigger cmd 8
    [    11128956.693192] (          34.843750) c0 pipe         21.71 ....../pipeline-stream.c:205  ERROR pipeline_copy(): ret = -61, start->comp.id = 70, dir = 0
    [    11129172.943183] (         216.249985) c0 pipe         21.71 ..../pipeline-schedule.c:127  ERROR pipeline_task(): xrun recovery failed! pipeline is stopped.
    

    It seems we need to wait for the IIR optimizatons @singalsu is doing.

  17. lgirdwood commented on Nov 12, 2021

    @lgirdwood
    Member

    @lyakh @kv2019i still an issue with the sync STOP PR now merged ?

  18. lyakh commented on Nov 15, 2021

    @lyakh
    Collaborator

    @lgirdwood multiple-pipeline-playback.sh passes now in daily tests, @kv2019i good to close or were you reproducing this manually?

  19. kv2019i commented on Nov 15, 2021

    @kv2019i
    CollaboratorAuthor

    @lyakh @lgirdwood Good to close -- I think it's Seppo's mixer optimization PR that makes the tet pass. It was also now merged.

  20. miRoox commented on Nov 22, 2021

    @miRoox

    Reproduced in recently daily tests.

  21. reopened this on Nov 22, 2021
  22. XiaoyunWu6666 commented on Nov 22, 2021

    @XiaoyunWu6666
    Contributor

    happened again in inner daily test result 8364
    on model TGLH_RVP_NOCODEC_ZEPHYR
    in testcase=multiple-pipeline-all-50

    [dmesg]

    [ 1308.063842] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx timed out for 0x60050000 (msg/reply size: 12/12)
    [ 1308.063855] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: preventing DSP entering D3 state to preserve context
    [ 1308.063859] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump start ]------------
    [ 1308.063899] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: hda irq intsts 0x00000000 intlctl 0xc00007b7 rirb 00
    [ 1308.063906] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: dsp irq ppsts 0x00000000 adspis 0x00000000
    [ 1308.063943] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
    [ 1308.063949] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump end ]------------
    [ 1308.063953] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
    [ 1308.063957] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: Firmware exception
    [ 1308.063962] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: fw_state: SOF_FW_BOOT_COMPLETE (6)
    [ 1308.063998] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
    [ 1308.064118] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: unexpected fault 0x00000000 trace 0x00004000
    [ 1308.064124] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
    [ 1308.064173] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0xa0]=0x20240000 successful
    [ 1308.064183] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
    [ 1308.064213] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at soc_component_trigger on 0000:00:1f.3: -110
    [ 1308.064223] kernel:  DMIC: ASoC: trigger FE cmd: 0 failed: -110
    [ 1308.064281] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: free stream 10 dir 1
    [ 1308.064382] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc rx: 0x90020000: GLB_TRACE_MSG: DMA_POSITION
    

    EDIT: and again in daily 9715, Jan 25th same test.

  23. added this to the v2.1 milestone on Dec 21, 2021
  24. kv2019i commented on Jan 4, 2022

    @kv2019i
    CollaboratorAuthor

    This should be fixed with Zephyr PR zephyrproject-rtos/zephyr#41547

  25. lgirdwood commented on Jan 27, 2022

    @lgirdwood
    Member

    @miRoox @kv2019i good to close ?

  26. miRoox commented on Jan 28, 2022

    @miRoox

    It seems there is only the sof-logger issue (#5120) in this platform. I think it's good to close.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

IPC timeoutIPC timeout observedTGLApplies to Tiger LakeTGL-HTGL-H platformZephyrIssues only observed with Zephyr integratedbugSomething isn't working as expected

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions