Skip to content

component: add key phrase buffer - #994

Merged
mmaka1 merged 7 commits into
thesofproject:masterfrom
mrajwa:ghw_dev
Mar 1, 2019
Merged

mmaka1 merged 7 commits into
thesofproject:masterfrom
mrajwa:ghw_dev

Conversation

@mrajwa

@mrajwa mrajwa commented Feb 12, 2019 •

Copy link
Copy Markdown
Contributor

This is the initial skeleton of key phrase buffer manager which aims to copy data from it's input pin to the output pin in real time and buffer this stream in its own internal buffer. There is also a possibility to register additional "client" which is an output buffer interested in data buffered in KPB's internal "history" buffer.
On client's request, data from internal buffer are copied into clients sink buffer.

PS: Component barely tested, more tests are ongoing. Macro "PLATFORM_LPSRAM_EBB_COUNT" needs to be added for more platforms.

Signed-off-by: Marcin Rajwa marcin.rajwa@linux.intel.com

Comment thread src/include/sof/audio/kpb.h Resolved
Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrajwa should r_ptr be initialized here?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ranj063, r_ptr of source or internal buffer? The former one is the responsibility of the source itself I think. The later is my job and will be initialized at buffering time.

Comment thread src/audio/kpb.c Outdated

@ranj063 ranj063 Feb 12, 2019 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrajwa is the expectation that these commands will be semt from the driver? Generally these cmds are top level comp commands such as set_data/get_data etc.
And what is the rationale for using sof_ipc_ctrl_data here? ctrl cmds are of type VOLUME/SWITCH/ENUM/BINARY typically

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all this is the place where I have most of my doubts. Somewhere in KPB module we need to handle requests specific for this module: add output sink (kpb_register_client) and start of draining (kpb_begin_draining) for example. These requests, needs to be triggered from outside of the KPB module and needs to be accompanied by DATA appropriate for this request. So for example kpb_register_client event needs client's/sink related data like draining history, state, id. So yes I though "int cmd" would contain request, and void *data its related DATA.
Now, can we simplify thing by agree "void *data" would be off type kpb_client instead of struct sof_ipc_ctrl_data?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@keyonjie, @xiulipan, can you comment on this issue?

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrajwa I feel that this step might unnecessary. I am thinking of something like a MUX which is connected to 5 sink components. This information can be obtained from topology. And can be retriedved by looking at the sink buffer list for the component.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not sure, KPB HLD documents specifies this client registration procedure. Keep in mind that every sink other than the real time one, needs to register in KPB prior to its draining request. This is obvious since at draining time KPB must know how much data shall be drained. I can see the use of MUX which may specify sink as one ofthe buffer on bsink_list and KPB cold fetch history depth as sink->free - is that your proposition about?
However I think this client oriented approach is much cleaner and allows us to easy extend KPB functionality later on if we decide to add more pre checks before draining.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrajwa is HLD = high level design? Can you point me to these documents?

@mrajwa mrajwa Feb 18, 2019 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dbaluta, I don't know its rather for internal use only. I will ask the owner of it whether it can be shared.
Anyway, I need to agree with driver side as for how are we going to communicate for events like "draining". I can imagine handle this scenario without clients/sinks registrations etc (although I do like this idea) but we need to make some decisions:

  1. What .ops will be used to start draining for specific sink/client.
  2. How draining parameters like "history_depth" will be passed into KPB and how they can be modify, if at all.
  3. How to communicate failure/success of this procedure, if at all.

@plbossart, @lgirdwood, @xiulipan, @keyonjie could you please answer above questions?

Comment thread src/audio/kpb.c Outdated

@ranj063 ranj063 Feb 12, 2019 •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrajwa what you might really need here is a cmd callback that can determine which sink to activate. So for the simplest case of 2 sinks where one is the keyword_detect comp and the other is the host capture PCM comp, you just need a command which tells you which sink buffer to send the data to. By default you'd send the data to the keyword detect comp buffer. When the keyword_detect detects the keyphrase, it can send a command to KPB indicating that. This can be handled with a set_value command with a 1 indicating a successful detection.

Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/include/uapi/ipc/topology.h Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/include/sof/audio/kpb.h Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checking source after getting source->avail doesn't make any sense.
Also r_ptr and w_ptr are initialized on buffer creation, so I think checking them is not necessary.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the first sentence I agree, for the second, I would leave this check of r_ptr & w_ptr as it is - at least for cmocka tests.

Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this be check for null?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes or rather just:
(!cd->his_buf_hp.sta_addr || !cd->his_buf_lp.sta_addr)

I applied checkpatch style corrections and cut these '!' unintentionally.

Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How do you handle the case that's after all the past audio data from buffer has been drained into ASR client (fast streaming mode completed) and client starts to receive the real-time stream until the voice interaction is complete. I was wondering if it would happen here in copy() into second sink.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But this part is about real time copying. Draining on demand is handled in kpb_begin_draining();
As for how it will be triggered from driver I need to agree with driver side.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yep, need to agree and I have no clear opinion here.

The background of this question was that switching to real-time streaming happens after key phrase has been passed and the remaining streaming can last quite long so I was wondering if this this could be more like generic capture with just a super large 1st copy() to host when streaming starts and the rest of copy() operations like normal audio capture to reuse the scheduler host/pcm infrastructure.

@lgirdwood lgirdwood left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, can you squash this down into less patches and please use a commit message for each patch otherwise it's difficult to follow.

Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you add some detail to the commit message about how this module works. It may be generic enough to be used as a generic detector i.e. also for voice and tones.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, I will update commit message. As for many comments making it difficult to follow - when you review some commit you can unselect show comments checkbox. This option is on the same bar as file name (left side). Also on the right side, right above this bar there is an option to show all commits so you don't have to move next/prev.

Comment thread src/include/sof/audio/kpb.h Resolved

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like this should be a separate patch

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you mean separate commit? I can reorganize commits a bit and apply all platcfg.h changes before I do anything KPB related. Although it requires a bit of rework.

Comment thread src/include/uapi/ipc/topology.h Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really just a buffer. This pipeline only needs to use two buffers

 PCM 0 <------------------------------+- Mux 0 <-- B0 <-- DMIC6 (DMIC01)
                                      |
 Keyword <-----B1 <---- Selector <----+

So this component should be of type sof_ipc_comp_process (please align with @ranj063 for details) and it shoud be next to a buffer in the topology.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood, as for now I don't see such type sof_ipc_comp_process. This part needs to be clarified since I have no idea what parameters are going to be provided when KPB is created. Are things like sampling_freq, sampling_width, history_depth etc going to be included in that sof_ipc_comp_process structure which you mentioned?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood I am not sure if Mux should be capable of all the logic that is being implemented inside the kpb right now (pls consider all the future scalability for multiple detectors being able to control their individual history depth etc.).
I believe that any future extensions of Mux should include current simple MicSel functionality etc. (in general being able to copy from multiple inputs to multiple outputs with the vector of channel coefficients provided for each input(s)-output set).

Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

best to use an unsigned here otherwise the compiler generates more code for byte handling

Comment thread src/include/sof/audio/component.h Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not used anywhere.

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

params method is missing

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Return 0 instead of error.

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sink_list

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

source_list

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it make any difference? I do know other components provide the offset member as you suggested but it doesn't make any difference to me.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You won't get the right pointer without the right offset.

Comment thread src/audio/kpb.c Resolved
Comment thread src/audio/kpb.c Resolved

@keyonjie keyonjie left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, can you rename kpb to kpbm? It is a component here, not a buffer.

Comment thread src/include/sof/audio/kpb.h Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not good to add something in a commit and remove it next in the same series, can you clean up it?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@keyonjie, I understand but how do you expect me to clean this? When I'll make a follow up commit, applying propositions from this PR there will be plenty of such removes anyway.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrajwa Good practice is to do rebase.

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So you are doing draining for the last client only, is this the intention?

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So the sink buffer size need to be up to 2 seconds also?

Comment thread src/include/sof/audio/component.h Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use #ifdef CONFIG_... in a similar way to other recently changed init declarations.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jajanusz, looks like you need to support CONFIG_COMP_KPB from kconfig.

@jajanusz jajanusz Feb 19, 2019 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mrajwa component author should add it in the same PR ;p
Please check this for reference: https://github.com/thesofproject/sof/pull/955/commits

Comment thread src/include/sof/audio/kpb.h Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use may want to use 'sink' instead of 'pin' for consistent naming with other pieces of the code.

@mrajwa mrajwa Feb 19, 2019 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In general, can you rename kpb to kpbm? It is a component here, not a buffer.

@keyonjie, well in fact it is also a buffer - it has its own internal (history_buffer). In cAVS it is also called KPB so for consistency I would keep it as KPB... @lgirdwood, @mmaka1 what are yout thought on this?

Comment thread src/include/sof/audio/kpb.h Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use dox format for comments, even if not included in the generated documentation yet.

Comment thread src/include/uapi/ipc/topology.h Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@lgirdwood I am not sure if Mux should be capable of all the logic that is being implemented inside the kpb right now (pls consider all the future scalability for multiple detectors being able to control their individual history depth etc.).
I believe that any future extensions of Mux should include current simple MicSel functionality etc. (in general being able to copy from multiple inputs to multiple outputs with the vector of channel coefficients provided for each input(s)-output set).

Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

With the latest comp_set_state implementation, it should be changed to if (ret).
We want to avoid multiple prepares.

@mrajwa
mrajwa force-pushed the ghw_dev branch 3 times, most recently from bf2fe0e to 5f1cbcc Compare February 24, 2019 11:07
@mrajwa
mrajwa force-pushed the ghw_dev branch 3 times, most recently from 6548f7e to e739698 Compare February 28, 2019 13:25
Comment thread src/audio/kpb.c Outdated

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
cli->id > KPB_MAX_NO_OF_CLIENTS) {
cli->id >= KPB_MAX_NO_OF_CLIENTS) {

Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
The key phrase buffer component skeleton. This component enables
to copy real time data stream from source to sink and in the same
time buffers this data into its own internal "history buffer".
Data from internal buffer can be drained later on specific event.

Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
KPB uses notifier module to implement events.

Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
@mmaka1
mmaka1 merged commit 53a63ea into thesofproject:master Mar 1, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

10 participants