You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[BUG] TGL_RVP_NOCODEC_ZEPHYR ipc timed out when TRIG_STOP with multiple-pipeline test #4680
in inner daily 6090, on TGLH_RVP_NOCODEC_ZEPHYR
multiple-pipeline-all-15: ipc timed out when issuing GLB_DAI_MSG: CONFIG(msg/reply size: 216/12)
check-runtime-pm-status-15: ipc timed out when issuing GLB_TPLG_MSG: PIPE_FREE(msg/reply size: 12/12)
@lgirdwood This is something else. When DSP is overloaded (>100% workload), processing of TRIG_STOP sometimes fails. The DSP doesn't crash and subsequent actions succeed. The overload situation does occur because we moved the audio buffers back to uncached now.
@kv2019i ok, lets wait for @lyakh triggering PR to merge. The 100% utilisation means we get in a state - do we get any data from the Zephyr LL scheduler ? i.e. is tick late or does work overrrun in logs ?
@lgirdwood This is something else. When DSP is overloaded (>100% workload), processing of TRIG_STOP sometimes fails. The DSP doesn't crash and subsequent actions succeed. The overload situation does occur because we moved the audio buffers back to uncached now.
@kv2019i, the TRIG_STOP ipc message times out, Linux would think that the DSP is crashed (we get timeout on IPC message). I have pending new fw_state to be taken by Linux in this case and this would block further IPC communications until the DSP is rebooted.
But in this case the DSP is just too busy to respond. Should the IPC timeout be treated as a feature from now on and only the panic message from DSP should be treated as crash? I would think that the firmware can crash in a not clean way to be able to tell Linux that it crashed..
@ujfalusi@kv2019i if FW is too busy to respond I would expect to see something in the logs to say this. e.g. LL should count number of times (per 1024 LL perf logging cycle) that there is little overhead for other work. This overhead could just be a subtraction of current time to next LL tick time and if it's less than limit we increment the count.
@lyakh working on this. With TGL systems and HDA generic tplg, I can no longer locally reproduce it. Will need to retry with nocodec tplg, but due to some system config issues, this will take a bit of time.
No @lgirdwood . This is still happening. The test cases causes a 95% DSP load (on a single core). XTOS build can manage it, but with Zephyr we hit an xrun systematically and the test fails:
First a lot of message from DMA that we are running out of data to process:
[ 11045636.748586] ( 2663.802002) c0 dw-dma src/drivers/dw/dma.c:1081 INFO dw_dma_free_data_size() size is 0!
[ 11045659.873585] ( 23.125000) c0 dai 3.14 src/audio/dai.c:950 WARN dai_copy(): nothing to copy
[ 11062863.518735] ( 17203.644531) c0 dw-dma src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
[ 11062886.747900] ( 23.229166) c0 dai 21.70 src/audio/dai.c:950 WARN dai_copy(): nothing to copy
[ 11082851.070024] ( 19964.322266) c0 dw-dma src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
[ 11082874.299189] ( 23.229166) c0 dai 21.70 src/audio/dai.c:950 WARN dai_copy(): nothing to copy
[ 11088642.996877] ( 5768.697754) c0 dw-dma src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
[ 11088665.965626] ( 22.968750) c0 dai 2.10 src/audio/dai.c:950 WARN dai_copy(): nothing to copy
[ 11089733.569750] ( 1067.604126) c0 dw-dma src/drivers/dw/dma.c:1081 INFO dw_dma_free_data_size() size is 0!
[ 11089756.382249] ( 22.812500) c0 dai 5.26 src/audio/dai.c:950 WARN dai_copy(): nothing to copy
[ 11089840.600996] ( 84.218750) c0 dw-dma src/drivers/dw/dma.c:1060 INFO dw_dma_avail_data_size() size is 0!
[ 11089862.475995] ( 21.875000) c0 dai 20.62 src/audio/dai.c:950 WARN dai_copy(): nothing to copy
Then xrun:
[ 11125584.557909] ( 37.656250) c0 host 21.64 src/audio/host.c:373 INFO no bytes to copy, 0 avail in buffer, 256 free in DMA
[ 11128500.599460] ( 2916.041504) c0 dw-dma src/drivers/dw/dma.c:1114 ERROR dw_dma_get_data_size(): xrun detected
[ 11128625.130705] ( 124.531242) c0 dai 21.70 src/audio/dai.c:882 ERROR dai_report_xrun(): overrun due to no space available
[ 11128771.328616] ( 146.197906) c0 dai 21.70 src/audio/dai.c:744 ERROR comp_overrun(): sink->free = 512, copy_bytes = 0
[ 11128921.849443] ( 150.520828) c0 pipe 21.71 ....../pipeline-stream.c:310 INFO pipe trigger cmd 8
[ 11128956.693192] ( 34.843750) c0 pipe 21.71 ....../pipeline-stream.c:205 ERROR pipeline_copy(): ret = -61, start->comp.id = 70, dir = 0
[ 11129172.943183] ( 216.249985) c0 pipe 21.71 ..../pipeline-schedule.c:127 ERROR pipeline_task(): xrun recovery failed! pipeline is stopped.
It seems we need to wait for the IIR optimizatons @singalsu is doing.
Activity
[-][BUG][/-][+][BUG] TGL_RVP_NOCODEC_ZEPHYR ipc timed out when TRIG_STOP with multiple-pipeline test[/+]XiaoyunWu6666 commentedon Aug 26, 2021
in inner daily 6090, on TGLH_RVP_NOCODEC_ZEPHYR
multiple-pipeline-all-15: ipc timed out when issuing GLB_DAI_MSG: CONFIG(msg/reply size: 216/12)
check-runtime-pm-status-15: ipc timed out when issuing GLB_TPLG_MSG: PIPE_FREE(msg/reply size: 12/12)
lgirdwood commentedon Aug 26, 2021
@kv2019i do you suspect this is related to allocator or something else ?
kv2019i commentedon Aug 30, 2021
@lgirdwood This is something else. When DSP is overloaded (>100% workload), processing of TRIG_STOP sometimes fails. The DSP doesn't crash and subsequent actions succeed. The overload situation does occur because we moved the audio buffers back to uncached now.
lgirdwood commentedon Aug 30, 2021
@kv2019i ok, lets wait for @lyakh triggering PR to merge. The 100% utilisation means we get in a state - do we get any data from the Zephyr LL scheduler ? i.e. is tick late or does work overrrun in logs ?
ujfalusi commentedon Aug 31, 2021
@kv2019i, the TRIG_STOP ipc message times out, Linux would think that the DSP is crashed (we get timeout on IPC message). I have pending new fw_state to be taken by Linux in this case and this would block further IPC communications until the DSP is rebooted.
But in this case the DSP is just too busy to respond. Should the IPC timeout be treated as a feature from now on and only the panic message from DSP should be treated as crash? I would think that the firmware can crash in a not clean way to be able to tell Linux that it crashed..
lgirdwood commentedon Aug 31, 2021
@ujfalusi @kv2019i if FW is too busy to respond I would expect to see something in the logs to say this. e.g. LL should count number of times (per 1024 LL perf logging cycle) that there is little overhead for other work. This overhead could just be a subtraction of current time to next LL tick time and if it's less than limit we increment the count.
kv2019i commentedon Oct 18, 2021
With #4884 , this issue should be finally fixed.
lgirdwood commentedon Oct 18, 2021
#4884 merged.
keqiaozhang commentedon Oct 19, 2021
@1994lwz , please help to verify this issue w/ latest firmware.
8 remaining items
kv2019i commentedon Oct 20, 2021
@lyakh working on this. With TGL systems and HDA generic tplg, I can no longer locally reproduce it. Will need to retry with nocodec tplg, but due to some system config issues, this will take a bit of time.
lgirdwood commentedon Oct 26, 2021
@kv2019i can we close ?
kv2019i commentedon Oct 28, 2021
No @lgirdwood . This is still happening. The test cases causes a 95% DSP load (on a single core). XTOS build can manage it, but with Zephyr we hit an xrun systematically and the test fails:
First a lot of message from DMA that we are running out of data to process:
Then xrun:
It seems we need to wait for the IIR optimizatons @singalsu is doing.
lgirdwood commentedon Nov 12, 2021
@lyakh @kv2019i still an issue with the sync STOP PR now merged ?
lyakh commentedon Nov 15, 2021
@lgirdwood multiple-pipeline-playback.sh passes now in daily tests, @kv2019i good to close or were you reproducing this manually?
kv2019i commentedon Nov 15, 2021
@lyakh @lgirdwood Good to close -- I think it's Seppo's mixer optimization PR that makes the tet pass. It was also now merged.
miRoox commentedon Nov 22, 2021
Reproduced in recently daily tests.
XiaoyunWu6666 commentedon Nov 22, 2021
happened again in inner daily test result 8364
on model TGLH_RVP_NOCODEC_ZEPHYR
in testcase=multiple-pipeline-all-50
[dmesg]
EDIT: and again in daily 9715, Jan 25th same test.
kv2019i commentedon Jan 4, 2022
This should be fixed with Zephyr PR zephyrproject-rtos/zephyr#41547
lgirdwood commentedon Jan 27, 2022
@miRoox @kv2019i good to close ?
miRoox commentedon Jan 28, 2022
It seems there is only the sof-logger issue (#5120) in this platform. I think it's good to close.