Repository navigation
Description
Describe the bug
TGLH_RVP_NOCODEC_ZEPHYR ipc timed out when COMP_FREE , when capturing with -Dhw:0,0 for 100 seconds,
after capturing with -Dhw:0,10 and -Dhw:0,11 (DMIC and DMIC16) for 100 seconds all
found in inner daily 5971
According to dmesg , it is related to failure of freeing widget EQIIR21.0 (on PCM DMIC [hw:0,10]) , BUF21.2(on PCM DMIC16kHz [hw:0,11])….
To Reproduce
TPLG=/lib/firmware/intel/sof-tplg/sof-tgl-h-nocodec.tplg ~/sof-test/test-case/check-capture.sh -d 100 -l 1 -r 1
Reproduction Rate
not 100%
Environment
Kernel Branch: topic/sof-dev
Kernel Commit: 34fd0612
SOF Branch: main
SOF Commit: db8ee55
Topology:/lib/firmware/intel/sof-tplg/sof-tgl-nocodec.tplg
Platform: TGLH_RVP_NOCODEC_ZEPHYR
Screenshots or console output
[console]
2021-08-19 21:37:11 UTC [REMOTE_INFO] ===== Testing: (Round: 1/1) (PCM: Port0 [hw:0,0]) (Loop: 1/1) =====
2021-08-19 21:37:11 UTC [REMOTE_INFO] no file prefix, use /dev/null as dummy capture output
2021-08-19 21:37:11 UTC [REMOTE_COMMAND] arecord -Dhw:0,0 -r 48000 -c 2 -f S16_LE -d 100 /dev/null -v -q
arecord: set_params:1407: Unable to install hw params:
ACCESS: RW_INTERLEAVED
FORMAT: S16_LE
SUBFORMAT: STD
SAMPLE_BITS: 16
FRAME_BITS: 32
CHANNELS: 2
RATE: 48000
PERIOD_TIME: (85333 85334)
PERIOD_SIZE: 4096
PERIOD_BYTES: 16384
PERIODS: 4
BUFFER_TIME: (341333 341334)
BUFFER_SIZE: 16384
BUFFER_BYTES: 65536
TICK_TIME: 0
【dmesg】
[ 743.261668] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: trigger stream 11 dir 1 cmd 1
[ 743.262081] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0xa0]=0x2024001e successful
[ 743.262090] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x60040000: GLB_STREAM_MSG: TRIG_START
[ 743.262990] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x60040000: GLB_STREAM_MSG: TRIG_START
[ 843.264039] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: trigger stream 11 dir 1 cmd 0
[ 843.264057] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[ 843.264560] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x60050000: GLB_STREAM_MSG: TRIG_STOP
[ 843.264595] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: FW Poll Status: reg[0xa0]=0x240000 successful
[ 843.264635] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x60030000: GLB_STREAM_MSG: PCM_FREE
[ 843.265178] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x60030000: GLB_STREAM_MSG: PCM_FREE
[ 843.265242] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: free stream 11 dir 1
[ 843.265253] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30020000: GLB_TPLG_MSG: COMP_FREE
[ 843.265704] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x30020000: GLB_TPLG_MSG: COMP_FREE
[ 843.265713] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget PCM11C freed
[ 843.265742] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30210000: GLB_TPLG_MSG: BUFFER_FREE
[ 843.266030] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x30210000: GLB_TPLG_MSG: BUFFER_FREE
[ 843.266036] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget BUF21.0 freed
[ 843.266069] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30020000: GLB_TPLG_MSG: COMP_FREE
[ 843.266334] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x30020000: GLB_TPLG_MSG: COMP_FREE
[ 843.266345] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget Dmic1 freed
[ 843.266354] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30210000: GLB_TPLG_MSG: BUFFER_FREE
[ 843.266665] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx succeeded: 0x30210000: GLB_TPLG_MSG: BUFFER_FREE
[ 843.266670] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget BUF21.1 freed
[ 843.266706] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30020000: GLB_TPLG_MSG: COMP_FREE
[ 843.769043] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x30020000 size 12
[ 843.769060] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: preventing DSP entering D3 state to preserve context
[ 843.769064] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump start ]------------
[ 843.769091] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: hda irq intsts 0x00000000 intlctl 0xc0000001 rirb 00
[ 843.769097] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: dsp irq ppsts 0x00000000 adspis 0x00000000
[ 843.769125] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: host status 0x00000000 dsp status 0x00000000 mask 0x00000003
[ 843.769131] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ IPC dump end ]------------
[ 843.769135] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump start ]------------
[ 843.769175] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: status: fw entered - code 00000005
[ 843.769269] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: unexpected fault 0x00000000 trace 0x00004000
[ 843.769274] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ------------[ DSP dump end ]------------
[ 843.769281] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to free widget EQIIR21.0
[ 843.769290] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget EQIIR21.0 freed
[ 843.769302] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30210000: GLB_TPLG_MSG: BUFFER_FREE
[ 844.273016] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x30210000 size 12
[ 844.273035] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to free widget BUF21.2
[ 844.273045] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget BUF21.2 freed
[ 844.273057] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30110000: GLB_TPLG_MSG: PIPE_FREE
[ 844.777016] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x30110000 size 12
[ 844.777035] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to free widget PIPELINE.21.DMIC1.IN
[ 844.777045] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget PIPELINE.21.DMIC1.IN freed
[ 844.777082] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_pcm_component_hw_free on 0000:00:1f.3: -110
[ 844.777139] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x80010000: GLB_DAI_MSG: CONFIG
[ 845.280968] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x80010000 size 216
[ 845.280987] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed resetting DAI config for DMIC1.IN
[ 845.281001] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30020000: GLB_TPLG_MSG: COMP_FREE
[ 845.784955] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x30020000 size 12
[ 845.784975] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to free widget DMIC1.IN
[ 845.784984] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: widget DMIC1.IN freed
[ 845.785098] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: close stream 11 dir 1
[ 845.811467] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: open stream 0 dir 1
[ 845.811474] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: period min 192 max 16384 bytes
[ 845.811478] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: period count 2 max 16
[ 845.811481] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: buffer max 65536 bytes
[ 845.811660] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: DAI config 0 matches pcm hw params
[ 845.811668] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: DAI config 0 matches pcm hw params
[ 845.811671] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: rate_min: 48000 rate_max: 48000
[ 845.811674] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: channels_min: 2 channels_max: 2
[ 845.811679] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: DAI config 0 matches pcm hw params
[ 845.811681] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: DAI config 0 matches pcm hw params
[ 845.811683] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: rate_min: 48000 rate_max: 48000
[ 845.811686] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: channels_min: 2 channels_max: 2
[ 845.811702] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ipc tx: 0x30010000: GLB_TPLG_MSG: COMP_NEW
[ 846.312833] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: ipc timed out for 0x30010000 size 96
[ 846.312853] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed to load widget SSP0.IN
[ 846.312859] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: error: failed setting up DAI widget SSP0.IN
[ 846.312866] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: ASoC: error at snd_soc_dai_hw_params on SSP0 Pin: -110
[ 846.312896] kernel: NoCodec-0: ASoC: soc_pcm_hw_params() failed (-110)
[ 846.312905] kernel: Port0: ASoC: dpcm_fe_dai_hw_params failed (-110)
[ 846.312919] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: free stream 0 dir 1
[ 846.313440] kernel: sof-audio-pci-intel-tgl 0000:00:1f.3: pcm: close stream 0 dir 1
Activity
kv2019i commentedon Aug 20, 2021
Under debug.
kv2019i commentedon Aug 23, 2021
I've not had success reproducing this bug. I've tried on two separate TGL systems, but the test keeps passing.
I'm now running tests with large number of pipeline create/frees without going to runtime-suspend in between. If we are leaking some resources in pipeline-free on Zephyr, this should pick it up. If this fails, then the bug is something more subtle, related to timing or similar.
kv2019i commentedon Aug 23, 2021
Hmm, this is really puzzling. The large pipe-new/free iteration test passed without errors.
I also tried to take the FW binary from the failing CI build machine and run that locally, but test still passes for me.
aiChaoSONG commentedon Aug 24, 2021
@XiaoyunWu6666 Could you please manually reproduce the issue with the FW CI built? Kai said he could not reproduce this issue.
@kv2019i Please note that CI still stays with GCC. XCC support for zephyr on TGL in CI may come next week.
XiaoyunWu6666 commentedon Aug 24, 2021
@kv2019i @aiChaoSONG last week I could manually reproduce the problem with CI built fw , let me retry again and attach some details here if any
XiaoyunWu6666 commentedon Aug 24, 2021
in inner daily 6013 , on case check-pause-resume-capture-100
After arecording with -Dhw:0,10 and pause release for 100 times firstly , arecording with -Dhw:0,11 and pause release for 100 times secondly , and trying to arecord with -Dhw:0,0 ,"arecord: set_params:1407: Unable to install hw params:" error happened in console.
Checked the dmesg and found ipc timed out when COMP_FREE
XiaoyunWu6666 commentedon Aug 24, 2021
hi ,@kv2019i , I managed to reproduce it on ubuntu@sh-tglh-rvp-nocodec-zephyr-01 manually with command 'TPLG=/lib/firmware/intel/sof-tplg/sof-tgl-h-nocodec.tplg ~/sof-test/test-case/check-capture.sh -d 100 -l 1 -r 1' and recipe below
Kernel Branch: topic/sof-dev
Kernel Commit: 34fd0612
SOF Branch: main
SOF Commit: db8ee55
But today it took me more than 5 times to reproduce it

kv2019i commentedon Aug 24, 2021
@XiaoyunWu6666 Thanks! I could today get the ipc time out with "multiple-pause-resume" test, but not yet with the check-capture check, but it's good input that you had to run it 5+ times, so maybe I just need to run more iterations.
kv2019i commentedon Aug 24, 2021
Potential bugfix: #4673
XiaoyunWu6666 commentedon Aug 25, 2021
This issue did not happen in today's daily 6047 .
I also run test case manually , captured for 100 seconds and pause-release when capturing for 100 times , ten times . Did not reproduce then.
kv2019i commentedon Aug 25, 2021
@XiaoyunWu6666 This should be fixed now indeed. We do have a new failure that shows up in other tests. You can identify this from IPC timeout for TRIG_STOP. I filed #4680 for this.
XiaoyunWu6666 commentedon Aug 26, 2021
inner daily 6090 shows good result
Close it via #4673