whisper : implement batched decoding

[ggerganov]

When using beam search, we currently run the decoders sequentially:

whisper.cpp/whisper.cpp

Lines 4416 to 4444 in f1c9df5

	// obtain logits for the next token
	for (int j = 0; j < n_decoders_cur; ++j) {
	auto & decoder = state->decoders[j];
	if (decoder.failed || decoder.completed) {
	continue;
	}
	decoder.tokens_tmp.resize(1);
	decoder.tokens_tmp[0] = decoder.sequence.tokens.back().id;
	//WHISPER_PRINT_DEBUG("%s: decoder %d: token %d, kv_self.n %d, seek_delta %d\n", __func__, j, decoder.tokens_tmp[0], decoder.kv_self.n, decoder.seek_delta);
	if (!whisper_decode_internal(*ctx, *state, decoder, decoder.tokens_tmp.data(), decoder.tokens_tmp.size(), decoder.kv_self.n, params.n_threads)) {
	fprintf(stderr, "%s: failed to decode\n", __func__);
	return -8;
	}
	{
	const int64_t t_start_sample_us = ggml_time_us();
	whisper_process_logits(*ctx, *state, params, decoder, t_cur);
	++decoder.kv_self.n;
	state->t_sample_us += ggml_time_us() - t_start_sample_us;
	}
	}

This is multiple times slower compared to a batched evaluation. This inefficiency is the major factor preventing efficient usage of beam search in whisper.cpp and thus often resulting in bad transcription quality.

Batched inference has been demonstrated in llama.cpp:

https://github.com/ggerganov/llama.cpp/blob/bd34cdde38f8fd661890ddd5f57ca30bf279877b/examples/baby-llama/baby-llama.cpp#L768-L777

This can be a starting point for doing the same in whisper.cpp and achieving efficient beam search implementation

[fire]

What is a good way to check this is working? For testing purposes to implement this.

[bobqianic]

I think we need some documentation on how to use ggml, as ggml's API is quite hard to understand. This way, more people can get started quickly, just like with PyTorch. @ggerganov

[ggerganov]

I agree. Actually simple example programs would be even better as they are easier to maintain long term.

I just need to find the time..