Repository navigation
component: add key phrase buffer - #994
Conversation
There was a problem hiding this comment.
@ranj063, r_ptr of source or internal buffer? The former one is the responsibility of the source itself I think. The later is my job and will be initialized at buffering time.
There was a problem hiding this comment.
@mrajwa is the expectation that these commands will be semt from the driver? Generally these cmds are top level comp commands such as set_data/get_data etc.
And what is the rationale for using sof_ipc_ctrl_data here? ctrl cmds are of type VOLUME/SWITCH/ENUM/BINARY typically
There was a problem hiding this comment.
First of all this is the place where I have most of my doubts. Somewhere in KPB module we need to handle requests specific for this module: add output sink (kpb_register_client) and start of draining (kpb_begin_draining) for example. These requests, needs to be triggered from outside of the KPB module and needs to be accompanied by DATA appropriate for this request. So for example kpb_register_client event needs client's/sink related data like draining history, state, id. So yes I though "int cmd" would contain request, and void *data its related DATA.
Now, can we simplify thing by agree "void *data" would be off type kpb_client instead of struct sof_ipc_ctrl_data?
There was a problem hiding this comment.
@mrajwa I feel that this step might unnecessary. I am thinking of something like a MUX which is connected to 5 sink components. This information can be obtained from topology. And can be retriedved by looking at the sink buffer list for the component.
There was a problem hiding this comment.
I am not sure, KPB HLD documents specifies this client registration procedure. Keep in mind that every sink other than the real time one, needs to register in KPB prior to its draining request. This is obvious since at draining time KPB must know how much data shall be drained. I can see the use of MUX which may specify sink as one ofthe buffer on bsink_list and KPB cold fetch history depth as sink->free - is that your proposition about?
However I think this client oriented approach is much cleaner and allows us to easy extend KPB functionality later on if we decide to add more pre checks before draining.
There was a problem hiding this comment.
@mrajwa is HLD = high level design? Can you point me to these documents?
There was a problem hiding this comment.
@dbaluta, I don't know its rather for internal use only. I will ask the owner of it whether it can be shared.
Anyway, I need to agree with driver side as for how are we going to communicate for events like "draining". I can imagine handle this scenario without clients/sinks registrations etc (although I do like this idea) but we need to make some decisions:
- What .ops will be used to start draining for specific sink/client.
- How draining parameters like "history_depth" will be passed into KPB and how they can be modify, if at all.
- How to communicate failure/success of this procedure, if at all.
@plbossart, @lgirdwood, @xiulipan, @keyonjie could you please answer above questions?
There was a problem hiding this comment.
@mrajwa what you might really need here is a cmd callback that can determine which sink to activate. So for the simplest case of 2 sinks where one is the keyword_detect comp and the other is the host capture PCM comp, you just need a command which tells you which sink buffer to send the data to. By default you'd send the data to the keyword detect comp buffer. When the keyword_detect detects the keyphrase, it can send a command to KPB indicating that. This can be handled with a set_value command with a 1 indicating a successful detection.
There was a problem hiding this comment.
Checking source after getting source->avail doesn't make any sense.
Also r_ptr and w_ptr are initialized on buffer creation, so I think checking them is not necessary.
There was a problem hiding this comment.
For the first sentence I agree, for the second, I would leave this check of r_ptr & w_ptr as it is - at least for cmocka tests.
There was a problem hiding this comment.
Should this be check for null?
There was a problem hiding this comment.
Yes or rather just:
(!cd->his_buf_hp.sta_addr || !cd->his_buf_lp.sta_addr)
I applied checkpatch style corrections and cut these '!' unintentionally.
There was a problem hiding this comment.
How do you handle the case that's after all the past audio data from buffer has been drained into ASR client (fast streaming mode completed) and client starts to receive the real-time stream until the voice interaction is complete. I was wondering if it would happen here in copy() into second sink.
There was a problem hiding this comment.
But this part is about real time copying. Draining on demand is handled in kpb_begin_draining();
As for how it will be triggered from driver I need to agree with driver side.
There was a problem hiding this comment.
Yep, need to agree and I have no clear opinion here.
The background of this question was that switching to real-time streaming happens after key phrase has been passed and the remaining streaming can last quite long so I was wondering if this this could be more like generic capture with just a super large 1st copy() to host when streaming starts and the rest of copy() operations like normal audio capture to reuse the scheduler host/pcm infrastructure.
lgirdwood
left a comment
There was a problem hiding this comment.
Sorry, can you squash this down into less patches and please use a commit message for each patch otherwise it's difficult to follow.
There was a problem hiding this comment.
Can you add some detail to the commit message about how this module works. It may be generic enough to be used as a generic detector i.e. also for voice and tones.
There was a problem hiding this comment.
Sure, I will update commit message. As for many comments making it difficult to follow - when you review some commit you can unselect show comments checkbox. This option is on the same bar as file name (left side). Also on the right side, right above this bar there is an option to show all commits so you don't have to move next/prev.
There was a problem hiding this comment.
Looks like this should be a separate patch
There was a problem hiding this comment.
Do you mean separate commit? I can reorganize commits a bit and apply all platcfg.h changes before I do anything KPB related. Although it requires a bit of rework.
There was a problem hiding this comment.
This is really just a buffer. This pipeline only needs to use two buffers
PCM 0 <------------------------------+- Mux 0 <-- B0 <-- DMIC6 (DMIC01)
|
Keyword <-----B1 <---- Selector <----+
So this component should be of type sof_ipc_comp_process (please align with @ranj063 for details) and it shoud be next to a buffer in the topology.
There was a problem hiding this comment.
@lgirdwood, as for now I don't see such type sof_ipc_comp_process. This part needs to be clarified since I have no idea what parameters are going to be provided when KPB is created. Are things like sampling_freq, sampling_width, history_depth etc going to be included in that sof_ipc_comp_process structure which you mentioned?
There was a problem hiding this comment.
@lgirdwood I am not sure if Mux should be capable of all the logic that is being implemented inside the kpb right now (pls consider all the future scalability for multiple detectors being able to control their individual history depth etc.).
I believe that any future extensions of Mux should include current simple MicSel functionality etc. (in general being able to copy from multiple inputs to multiple outputs with the vector of channel coefficients provided for each input(s)-output set).
There was a problem hiding this comment.
best to use an unsigned here otherwise the compiler generates more code for byte handling
There was a problem hiding this comment.
Does it make any difference? I do know other components provide the offset member as you suggested but it doesn't make any difference to me.
There was a problem hiding this comment.
You won't get the right pointer without the right offset.
keyonjie
left a comment
There was a problem hiding this comment.
In general, can you rename kpb to kpbm? It is a component here, not a buffer.
There was a problem hiding this comment.
Not good to add something in a commit and remove it next in the same series, can you clean up it?
There was a problem hiding this comment.
@keyonjie, I understand but how do you expect me to clean this? When I'll make a follow up commit, applying propositions from this PR there will be plenty of such removes anyway.
There was a problem hiding this comment.
So you are doing draining for the last client only, is this the intention?
There was a problem hiding this comment.
So the sink buffer size need to be up to 2 seconds also?
There was a problem hiding this comment.
Use #ifdef CONFIG_... in a similar way to other recently changed init declarations.
There was a problem hiding this comment.
@jajanusz, looks like you need to support CONFIG_COMP_KPB from kconfig.
There was a problem hiding this comment.
@mrajwa component author should add it in the same PR ;p
Please check this for reference: https://github.com/thesofproject/sof/pull/955/commits
There was a problem hiding this comment.
Use may want to use 'sink' instead of 'pin' for consistent naming with other pieces of the code.
There was a problem hiding this comment.
In general, can you rename kpb to kpbm? It is a component here, not a buffer.
@keyonjie, well in fact it is also a buffer - it has its own internal (history_buffer). In cAVS it is also called KPB so for consistency I would keep it as KPB... @lgirdwood, @mmaka1 what are yout thought on this?
There was a problem hiding this comment.
Use dox format for comments, even if not included in the generated documentation yet.
There was a problem hiding this comment.
@lgirdwood I am not sure if Mux should be capable of all the logic that is being implemented inside the kpb right now (pls consider all the future scalability for multiple detectors being able to control their individual history depth etc.).
I believe that any future extensions of Mux should include current simple MicSel functionality etc. (in general being able to copy from multiple inputs to multiple outputs with the vector of channel coefficients provided for each input(s)-output set).
There was a problem hiding this comment.
With the latest comp_set_state implementation, it should be changed to if (ret).
We want to avoid multiple prepares.
bf2fe0e to
5f1cbcc
Compare
6548f7e to
e739698
Compare
There was a problem hiding this comment.
| cli->id > KPB_MAX_NO_OF_CLIENTS) { | |
| cli->id >= KPB_MAX_NO_OF_CLIENTS) { |
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
The key phrase buffer component skeleton. This component enables to copy real time data stream from source to sink and in the same time buffers this data into its own internal "history buffer". Data from internal buffer can be drained later on specific event. Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
KPB uses notifier module to implement events. Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
Signed-off-by: Marcin Rajwa <marcin.rajwa@linux.intel.com>
This is the initial skeleton of key phrase buffer manager which aims to copy data from it's input pin to the output pin in real time and buffer this stream in its own internal buffer. There is also a possibility to register additional "client" which is an output buffer interested in data buffered in KPB's internal "history" buffer.
On client's request, data from internal buffer are copied into clients sink buffer.
PS: Component barely tested, more tests are ongoing. Macro "PLATFORM_LPSRAM_EBB_COUNT" needs to be added for more platforms.
Signed-off-by: Marcin Rajwa marcin.rajwa@linux.intel.com