Skip to content

ASoC: SOF: ipc: Rate-limit IPC transmission error messages - #6020

Open
anantxgarg wants to merge 2 commits into
thesofproject:topic/sof-devfrom
anantxgarg:topic/sof-ipc-ratelimit
Open

anantxgarg wants to merge 2 commits into
thesofproject:topic/sof-devfrom
anantxgarg:topic/sof-ipc-ratelimit

Conversation

@anantxgarg

Copy link
Copy Markdown

When an audio DSP fails to properly reinitialise or desynchronises after a system suspend/resume cycle, subsequent stream parameter setup requests fail with -EINVAL. In response, userspace audio servers (such as PipeWire) attempt to restart the PCM stream immediately in a rapid retry loop. This unthrottled loop can flood system logs with millions of lines, consuming tens of gigabytes and exhausting root filesystems.

Mitigate this logging storm by converting dev_err() to dev_err_ratelimited() in ipc3_wait_tx_done() and ipc4_wait_tx_done(), and adding spcm_err_ratelimited() in sof-audio.h for stream preparation error paths.

Closes: thesofproject/sof#10442

@sofci

sofci commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Can one of the admins verify this patch?

reply test this please to run this test once

Comment thread sound/soc/sof/pcm.c
spcm_err(spcm, substream->stream,
"failed to set hw_params after resume\n");
spcm_err_ratelimited(spcm, substream->stream,
"failed to set hw_params after resume\n");

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ASoC still going to fill up the kernel log even if we rate limit.

You can still reproduce the issue with recent kernel and user space?

checkpatch reports several issues, the commit message is misaligned and if possible also align the parameters as well.

@anantxgarg anantxgarg Nov 4, 2026 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wrapping fixed.
The issue occurred on kernel 6.8.0-147-generic, PipeWire 1.0.5 where in from what i could determine, after waking up from suspend the the DSP failed the resume handshake which led to STREAM_PCM_PARAMS returning -EINVAL. Pipewire then repeatedly retried snd_pcm_prepare() which produced around 26 GB of log files within a few hours. I havent yet tried to recreate it on newer kernels.

ASoC is still going to fill it up even if we ratelimit in sof, however it would lead to a 75% reduction in error messages. It seems to me a worthwhile change to make.
Further changes upstream to ASoC core will be required to address that issue in particular.

@anantxgarg
anantxgarg force-pushed the topic/sof-ipc-ratelimit branch from 23c4355 to 0823a06 Compare November 4, 2026 14:30
@anantxgarg
anantxgarg requested a review from ujfalusi November 5, 2026 19:45
When an audio DSP fails to properly reinitialize or desynchronizes after
a system suspend/resume cycle, subsequent stream parameter setup
requests fail with -EINVAL.

In response, userspace audio servers (such as PipeWire) attempt to
restart the PCM stream immediately in a rapid retry loop. This unthrottled
loop can flood system logs with millions of lines, consuming tens of
gigabytes and exhausting root filesystems.

Mitigate this logging storm by converting dev_err() and spcm_err() to
dev_err_ratelimited() and spcm_err_ratelimited() across IPC transmission
paths (ipc3_wait_tx_done, ipc4_wait_tx_done) and PCM stream setup paths
(sof_pcm_prepare, sof_pcm_hw_params, and sof_pcm_stream_free).

Closes: thesofproject/sof#10442
Signed-off-by: Anant Garg <anantxgarg@gmail.com>
When lower-level drivers fail during stream operations, snd_soc_ret()
emits an unthrottled dev_err() for all unhandled negative error codes.

During repeated userspace retry loops (such as snd_pcm_prepare() retries
on failing hardware), this unthrottled logging produces thousands of
duplicate error lines, filling system logs.

Convert dev_err() in snd_soc_ret() to dev_err_ratelimited() to mitigate
log flooding across all ASoC component and DAI operations.

Signed-off-by: Anant Garg <anantxgarg@gmail.com>
@anantxgarg
anantxgarg force-pushed the topic/sof-ipc-ratelimit branch from 0823a06 to 1240a67 Compare November 21, 2026 13:05
@anantxgarg

Copy link
Copy Markdown
Author

Hi @ujfalusi I re-tested this running a 7.0 kernel with PipeWire 1.0.5, and the issue still reproduces reliably. During stream prepare retries, it spat out around 700 error lines into dmesg within a few seconds. Roughly a third of those logs were coming straight from snd_soc_ret() in soc-utils.c, so rate-limiting SOF alone wasn't enough to stop the flood.

The branch is reworked into two clean commits:

  1. Rate-limits the remaining unthrottled paths in SOF (pcm.c widget setup, hw_free, platform hw_params) alongside the IPC calls.
  2. Switches dev_err() to dev_err_ratelimited() in snd_soc_ret() to catch the ASoC core logging.

Commit wrapping, parameter alignment, and checkpatch warnings should all be sorted now. Please run CI when you get a chance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] [CML] Infinite ipc tx error loop after suspend/resume floods journal

3 participants