Skip to content

Add support for IPC4 loadable modules and smart_amp_test module - #4096

Closed
aiChaoSONG wants to merge 5 commits into
thesofproject:topic/sof-devfrom
aiChaoSONG:smt_upstream
Closed

aiChaoSONG wants to merge 5 commits into
thesofproject:topic/sof-devfrom
aiChaoSONG:smt_upstream

Conversation

@aiChaoSONG

Copy link
Copy Markdown

All IPC4 loadable module will require base module config + base module config extension style blob to be initialized. the IPC4 smart_amp_test module is implemented with the same style init blob. Add the support in the kernel for them.

Patches have been verified with IADK dummy smart amp loadable module, and SOF smart_amp_test module here: thesofproject/sof#6751

Chao Song added 5 commits December 8, 2022 17:20
There are two styles IPC payload blob for SOF module
instance initialization:

  - base module config + module specific extension
  - base module config + base module config extension

The first one is commonly used by basefw modules, and
the second one is commonly used by SOF loadable modules.

The patch adds structures for base module config extension.

Signed-off-by: Chao Song <chao.song@linux.intel.com>
This patch adds a new token to tell which one of the
below module instance initialization blob style a widget
is using:

  - base module config + module specific extension
  - base module config + base module config extension

So that the kernel knows how to prepare IPC4 payload
for the widget.

Signed-off-by: Chao Song <chao.song@linux.intel.com>
The new base module config + base module config extension
module initialization blob is commonly used by SOF loadable
process modules, this patch adds support for those modules.

Signed-off-by: Chao Song <chao.song@linux.intel.com>
For module uses base module config + base module config
extension style init blob, the pin format may be different
from base module config.

This patch will set copier sink from the connected pin
format.

Signed-off-by: Chao Song <chao.song@linux.intel.com>
Some process modules may change the pipeline params,
update the pipeline params from output format in the
prepare ops for process modules.

Currently, only channels are required to be updated.

Signed-off-by: Chao Song <chao.song@linux.intel.com>
u8 reserved[8];
u32 priv_param_len;
struct sof_ipc4_pin_format sink_pin_fmt[0];
struct sof_ipc4_pin_format source_pin_fmt[0];

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

use DECLARE_FLEX_ARRAY() ?

ghost Dec 14, 2022 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The init blob comes from topology bytes kcontrol, we will use this structure to decode the binary blob, we don't need to alloc anything, I don't think FLEX_ARRAY is good to be used for this case.

ghost Dec 14, 2022 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aiChaoSONG I dont think it makes sense to have 2 flexible arrays in a struct. Please combine them into one, pin_fmts[]

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed this is heavily frowned upon. I don't understand what this is about quite frankly.

ghost Dec 15, 2022 •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aiChaoSONG I dont think it makes sense to have 2 flexible arrays in a struct. Please combine them into one, pin_fmts[]

Actually there are three: struct sof_ipc4_pin_format sink_pin_fmt[0]; struct sof_ipc4_pin_format source_pin_fmt[0]; u8 priv[0]; Maybe the easiest way is just to use void *priv_data[0]; or u8 priv[0] directly.

ghost Dec 15, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not compute for me either.

	struct sof_ipc4_pin_format sink_pin_fmt[0];
	struct sof_ipc4_pin_format source_pin_fmt[0];
	u8 priv[0];

does not make any sense at all. Where priv[0] is pointing to? They all point to the same offset as all of them have size of 0.
at best you have data[] at the end with a comment that from there you have num_sink_pin_fmts x sof_ipc4_pin_format followed by num_source_pin_fmts x sof_ipc4_pin_format then priv_param_len number of bytes of some data.

Comment thread sound/soc/sof/sof-audio.h
*
* This flag is used to tell which initialization blob style a widget is using.
*/
int init_blob_style;

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is IPC4 specific isnt it? Does it need to go in struct snd_sof_widget? Can we find a better place?

ghost Dec 14, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

snd_sof_widget is shared between IPC3 and IPC4, I don't think we have a IPC4 specific struct which is a good fit for this.

we can later add snd_sof_ipc3_widget/snd_sof_ipc4_widget if necessary.

ghost Dec 14, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about struct sof_ipc4_process? We know that the base_fw modules already use module specific extension.

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

payload_with_output_fmt is already IPC4-specific @ranj063, looks like another poorly defined partitioning.

ghost Dec 15, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

payload_with_output_fmt is already IPC4-specific @ranj063, looks like another poorly defined partitioning.

Maybe fix both of them?

ghost Dec 15, 2022

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

payload_with_output_fmt is already IPC4-specific @ranj063, looks like another poorly defined partitioning.

@ujfalusi raised this concern about payload_with_output_fmt before. But we don't have ipc4 related structure at that time, so we put them here temporarily. I think we do need a IPC4 specific place to put ipc4 stuff. If this is only used by process module, struct sof_ipc4_process is good.


memcpy(cfg, data, control_data->data->size);

/* Update pipeline params from output format */

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

does this conflict with @libinyang 's PR?

ghost Dec 14, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we do the same thing to update pipeline params, we have synced, and I will merge some of his code in this PR, then we can close #4097

struct sof_ipc4_msg msg;
};

/**

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

are we already using this type "base module config + module specific extension"?

ghost Dec 14, 2022 •

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, all bese fw module use this type, for example:
copier:

struct ipc4_copier_module_cfg {
	struct ipc4_base_module_cfg base;

	struct ipc4_audio_format out_fmt;
	uint32_t copier_feature_mask;
	struct ipc4_copier_gateway_cfg  gtw_cfg;
} __attribute__((packed, aligned(4)));

peakvol

struct ipc4_peak_volume_module_cfg {
	struct ipc4_base_module_cfg base_cfg;
	struct ipc4_peak_volume_config config[0];
} __packed __aligned(8);

module specific extension is optional, mixin/mixout don't have module specific extension.
https://github.com/thesofproject/sof/blob/73f6171bb3c8c5b1d0963883014508022445b997/src/audio/mixin_mixout/mixin_mixout.c#L157

ghost Dec 14, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aiChaoSONG can youplease add this to the commit message saying that support for the first type is already implemented

/*
* Output format is optional for process modules.
* Process modules setup the output format based on audio format tokens in topology.
* Output format is only valid and optional for modules with base module config +

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'd switch around the optional and only valid bits. "Output format is optional and only valid ...." and the check below as well

sizeof(struct sof_ipc4_pin_format);
if (data_size != size) {
dev_err(sdev->dev, "Insufficient data for widget %s initialization\n",
swidget->widget->name);

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should this check be done on line 1660? What if there are no kcontrols for the widget?

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what if there are more than one kcontrols for the widget?

}
}

static const struct sof_ipc4_audio_format *

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo in commit message " For modules using..."

if (swidget->init_blob_style == SOF_INIT_BLOB_BASE_WITH_MOD_EXT) {
struct sof_ipc4_base_module_cfg *base_config = swidget->private;

if (pin_type == SOF_PIN_TYPE_SOURCE && swidget->payload_with_output_fmt) {

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when will this be true? If I see the call below, pin_type is always SOF_PIN_TYPE_SINK?

ghost Dec 14, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

in updating pipeline params (process module prepare ops), we need to get the output pin format, and set the output format to pipeline param, the pin_type will be SOF_PIN_TYPE_SOURCE there.

Comment on lines +227 to +228
} else {
pin_start = base_ext->num_sink_pin_fmts;

ghost Dec 13, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need the else? same as the comment above pin_type will not be different at all

ghost Dec 14, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the function supports to get any input/output pin format, in setting copier sink, we need to get the input format of the widget that connected to copier, in updating pipeline params, we need to get the output format of the widget, and update the format to pipeline params.

ghost left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we're quite far from a mergeable PR @aiChaoSONG, please see comments below.

* struct sof_ipc4_pin_format - widget's pin format
* @pin_index: pin index for the widget
* @ibs: input buffer size for input pin
* @obs: output buffer size for output pin

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

how do we know if the pin is input or output?


/**
* struct sof_ipc4_pin_format - widget's pin format
* @pin_index: pin index for the widget

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

is the pin_index relative to a direction, i.e. you can have both pin_index=1 for input and pin_index=1 for output

u8 reserved[8];
u32 priv_param_len;
struct sof_ipc4_pin_format sink_pin_fmt[0];
struct sof_ipc4_pin_format source_pin_fmt[0];

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

indeed this is heavily frowned upon. I don't understand what this is about quite frankly.

#define SOF_TKN_COMP_SINK_PIN_BINDING_WNAME 413
#define SOF_TKN_COMP_SRC_PIN_BINDING_WNAME 414
#define SOF_TKN_COMP_PAYLOAD_WITH_OUTPUT_FMT 415
#define SOF_TKN_COMP_INIT_BLOB_STYLE 416

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are the values for this token?

Comment thread sound/soc/sof/sof-audio.h

/* Widget initialization blob style */
#define SOF_INIT_BLOB_BASE_WITH_MOD_EXT 0
#define SOF_INIT_BLOB_BASE_WITH_BASE_EXT 1

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well the values should be in the tokens.h definition, no? This defines the UAPI, it should be clear.
I am afraid we have similar issues with the pin type. Gah.

}
}

static const struct sof_ipc4_audio_format *

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

commit message: I have idea what this means

"This patch will set copier sink from the connected pin format."

The connected pin format can only be used to set a sink pin format, no?

ghost Dec 15, 2022

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The already merged PR name the function as sof_ipc4_set_copier_sink_format to align with firmware naming, it is actually setting the copier source(output) pin format from the connected sink(input) pin format of the downstream widget.

I will update the commit message to: set copier source pin from connected sink pin format


static const struct sof_ipc4_audio_format *
sof_ipc4_get_widget_pin_format(struct snd_sof_widget *swidget, bool pin_type,
int pin_id)

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

one line


base_ext = (struct sof_ipc4_base_module_cfg_ext *)control_data->data->data;
break;
}

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is spaghetti code to me. Just can't figure out how safe/reliable this is.

}

for (i = pin_start; i < pin_end; i++) {
if (pin_fmt[i].pin_index == pin_id)

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you've got to explain how the pins are ordered or numbered. This is super hard to review.

hw_param_interval(pipeline_params, SNDRV_PCM_HW_PARAM_CHANNELS)->min =
SOF_IPC4_AUDIO_FORMAT_CFG_CHANNELS_COUNT(out_fmt->fmt_cfg);
hw_param_interval(pipeline_params, SNDRV_PCM_HW_PARAM_CHANNELS)->max =
SOF_IPC4_AUDIO_FORMAT_CFG_CHANNELS_COUNT(out_fmt->fmt_cfg);

ghost Dec 14, 2022

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't understand what this is about.

a) does a pipeline have format parameters? If not, then the commit message is just wrong.
b) can avoid the repetition of hw_param_interval(pipeline_params, SNDRV_PCM_HW_PARAM_CHANNELS)-> and use a local variable to make the code more readable?

if (control_data->data->blob_type != SOF_IPC4_MOD_INIT_INSTANCE)
continue;

data_size = control_data->data->size;

ghost Dec 15, 2022

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if data_size = 0? Or there is no SOF_IPC4_MOD_INIT_INSTANCE blob type in the kcontrol?

@aiChaoSONG

ghost commented Jan 10, 2023 •

Copy link
Copy Markdown
Author

a better alternative is #4121, close this one.

@aiChaoSONG aiChaoSONG closed this Jan 10, 2023
@aiChaoSONG
aiChaoSONG deleted the smt_upstream branch November 21, 2026 02:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants