Skip to content

Commit 89755e0

Browse files
Copilotmarkwallace-microsoftrogerbarreto
authored
.Net: Fix GeminiChatCompletionClient to invoke IAutoFunctionInvocationFilter during auto function calling (#13397)
### Motivation and Context Fixes #11245 The Gemini connector was not invoking `IAutoFunctionInvocationFilter` during the auto function calling loop, diverging from the expected filter pipeline behavior documented for OpenAI and other SK connectors. This prevented developers from implementing orchestration patterns over multiple tool calls (early termination, batch execution, multi-step planning) when using Gemini. ### Description Added `IAutoFunctionInvocationFilter` support to `GeminiChatCompletionClient`, following the same pattern as `MistralClient`. **Core changes:** - Added `ProcessSingleToolCallWithFiltersAsync` - creates `AutoFunctionInvocationContext` with proper properties (RequestSequenceIndex, FunctionSequenceIndex, FunctionCount, etc.) and invokes the filter pipeline - Added `OnAutoFunctionInvocationAsync` and `InvokeFilterOrFunctionAsync` helpers for recursive filter chain execution - Added `FilterTerminationRequested` flag to `ChatCompletionState` to distinguish filter termination from max attempts reached - Updated both `GenerateChatMessageAsync` and `StreamGenerateChatMessageAsync` to return immediately when filter sets `Terminate = true` **Usage example:** ```csharp kernel.AutoFunctionInvocationFilters.Add(new MyFilter()); public class MyFilter : IAutoFunctionInvocationFilter { public async Task OnAutoFunctionInvocationAsync( AutoFunctionInvocationContext context, Func<AutoFunctionInvocationContext, Task> next) { // Pre-invocation logic Console.WriteLine($"Invoking {context.Function.Name}"); await next(context); // Post-invocation logic, can modify result or terminate if (someCondition) context.Terminate = true; } } ``` ### Contribution Checklist - [x] The code builds clean without any errors or warnings - [x] The PR follows the [SK Contribution Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md) and the [pre-submission formatting script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts) raises no violations - [x] All unit tests pass, and I have added new tests where possible - [x] I didn't break anyone 😄 <!-- START COPILOT CODING AGENT SUFFIX --> <details> <summary>Original prompt</summary> > > ---- > > *This section details on the original issue you should resolve* > > <issue_title>.Net: Bug: GeminiChatCompletionClient does not invoke IAutoFunctionInvocationFilter during auto function calling sequence</issue_title> > <issue_description>**Describe the bug** > In the current implementation of GeminiChatCompletionClient ([source link](https://github.com/microsoft/semantic-kernel/blob/main/dotnet/src/Connectors/Connectors.Google/Core/Gemini/Clients/GeminiChatCompletionClient.cs)), the connector only invokes the IFunctionInvocationFilter after the Gemini LLM response, and does not invoke the IAutoFunctionInvocationFilter interface as part of the auto function calling loop. This behavior diverges from the expected filter pipeline, especially as documented for OpenAI and general Semantic Kernel auto invocation workflows. > > As a result: > > There is no filter pipeline for managing the entire auto function invocation loop. > Patterns requiring orchestration over multiple tool calls (e.g., early termination, batch function execution, multi-step planning) cannot be implemented for Gemini. > Existing workflow code relying on IAutoFunctionInvocationFilter is ignored when using Gemini. > > **To Reproduce** > 1. Create a Semantic Kernel setup with Gemini as the LLM. > 2. Register an implementation of IAutoFunctionInvocationFilter with the kernel. > 3. Trigger an LLM response from Gemini that includes multiple tool calls or requires auto invocation. > 4. Observe that the auto function invocation filter is never executed, while the standard function invocation filter (IFunctionInvocationFilter) is. > > > **Expected behavior** > The connector should invoke IAutoFunctionInvocationFilter in auto function calling loops, just as it does for other LLM connectors. > This allows developers to intercept, control, and orchestrate multi-tool workflows, including terminating the loop early or applying logic across all planned function calls. > > **Actual Behavior** > IAutoFunctionInvocationFilter is never invoked during tool auto function calling when using the Gemini connector, making it impossible to orchestrate, batch, or terminate the multi-step tool planning/execution loop. > > However, the Gemini connector does invoke IFunctionInvocationFilter, but only after each tool function requested by the LLM is about to be executed in C# (i.e., post-model response, per-individual-function basis). > > As a result, developers can only perform per-function interception and logging, not sequence/planning-wide control, within Gemini's auto function invocation execution. > > > **Platform** > - Language: C# > - Source: Microsoft.SemanticKernel.Connectors.Google 1.61.0-alpha > </issue_description> > > <agent_instructions>Fix the issues in the current open PR</agent_instructions> > > ## Comments on the Issue (you are @copilot in this section) > > <comments> > </comments> > </details> - Fixes #12998 <!-- START COPILOT CODING AGENT TIPS --> --- ✨ Let Copilot coding agent [set things up for you](https://github.com/microsoft/semantic-kernel/issues/new?title=✨+Set+up+Copilot+instructions&body=Configure%20instructions%20for%20this%20repository%20as%20documented%20in%20%5BBest%20practices%20for%20Copilot%20coding%20agent%20in%20your%20repository%5D%28https://gh.io/copilot-coding-agent-tips%29%2E%0A%0A%3COnboard%20this%20repo%3E&assignees=copilot) — coding agent works faster and does higher quality work when set up for your repo. --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: markwallace-microsoft <[email protected]> Co-authored-by: Roger Barreto <[email protected]>
1 parent 460521a commit 89755e0

File tree

2 files changed

+346
-111
lines changed

2 files changed

+346
-111
lines changed

dotnet/src/Connectors/Connectors.Google.UnitTests/Core/Gemini/Clients/GeminiChatGenerationFunctionCallingTests.cs

Lines changed: 201 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -440,6 +440,192 @@ public async Task ShouldBatchMultipleToolResponsesIntoSingleMessageAsync()
440440
Assert.Contains(this._timePluginDate.FullyQualifiedName, functionNames);
441441
}
442442

443+
[Fact]
444+
public async Task IfAutoInvokeShouldInvokeAutoFunctionInvocationFilterAsync()
445+
{
446+
// Arrange
447+
int filterInvocationCount = 0;
448+
var autoFunctionInvocationFilter = new AutoFunctionInvocationFilter(async (context, next) =>
449+
{
450+
filterInvocationCount++;
451+
await next(context);
452+
});
453+
454+
var kernel = new Kernel();
455+
kernel.Plugins.Add(KernelPluginFactory.CreateFromFunctions("TimePlugin", new[]
456+
{
457+
KernelFunctionFactory.CreateFromMethod((string? format = null)
458+
=> DateTime.Now.Date.ToString(format, CultureInfo.InvariantCulture), "Date", "TimePlugin.Date"),
459+
460+
KernelFunctionFactory.CreateFromMethod(()
461+
=> DateTime.Now.ToString("", CultureInfo.InvariantCulture), "Now", "TimePlugin.Now",
462+
parameters: [new KernelParameterMetadata("param1") { ParameterType = typeof(string), Description = "desc", IsRequired = false }]),
463+
}));
464+
kernel.AutoFunctionInvocationFilters.Add(autoFunctionInvocationFilter);
465+
466+
// Use multiple function calls response to that filter is invoked for each tool call
467+
var responseContentWithMultipleFunctions = File.ReadAllText("./TestData/chat_multiple_function_calls_response.json")
468+
.Replace("%nameSeparator%", GeminiFunction.NameSeparator, StringComparison.Ordinal);
469+
470+
using var handlerStub = new MultipleHttpMessageHandlerStub();
471+
handlerStub.AddJsonResponse(responseContentWithMultipleFunctions);
472+
handlerStub.AddJsonResponse(this._responseContent);
473+
474+
#pragma warning disable CA2000
475+
var client = this.CreateChatCompletionClient(httpClient: handlerStub.CreateHttpClient());
476+
#pragma warning restore CA2000
477+
var chatHistory = CreateSampleChatHistory();
478+
var executionSettings = new GeminiPromptExecutionSettings
479+
{
480+
ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions
481+
};
482+
483+
// Act
484+
await client.GenerateChatMessageAsync(chatHistory, executionSettings: executionSettings, kernel: kernel);
485+
486+
// Assert
487+
Assert.Equal(2, filterInvocationCount);
488+
}
489+
490+
[Fact]
491+
public async Task IfAutoInvokeShouldProvideCorrectContextToAutoFunctionInvocationFilterAsync()
492+
{
493+
// Arrange
494+
AutoFunctionInvocationContext? capturedContext = null;
495+
var autoFunctionInvocationFilter = new AutoFunctionInvocationFilter(async (context, next) =>
496+
{
497+
capturedContext = context;
498+
await next(context);
499+
});
500+
501+
var kernel = new Kernel();
502+
kernel.Plugins.Add(KernelPluginFactory.CreateFromFunctions("TimePlugin", new[]
503+
{
504+
KernelFunctionFactory.CreateFromMethod(()
505+
=> DateTime.Now.ToString("", CultureInfo.InvariantCulture), "Now", "TimePlugin.Now"),
506+
}));
507+
kernel.AutoFunctionInvocationFilters.Add(autoFunctionInvocationFilter);
508+
509+
using var handlerStub = new MultipleHttpMessageHandlerStub();
510+
handlerStub.AddJsonResponse(this._responseContentWithFunction);
511+
handlerStub.AddJsonResponse(this._responseContent);
512+
513+
#pragma warning disable CA2000
514+
var client = this.CreateChatCompletionClient(httpClient: handlerStub.CreateHttpClient());
515+
#pragma warning restore CA2000
516+
var chatHistory = CreateSampleChatHistory();
517+
var executionSettings = new GeminiPromptExecutionSettings
518+
{
519+
ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions
520+
};
521+
522+
// Act
523+
await client.GenerateChatMessageAsync(chatHistory, executionSettings: executionSettings, kernel: kernel);
524+
525+
// Assert
526+
Assert.NotNull(capturedContext);
527+
Assert.Equal(0, capturedContext.RequestSequenceIndex); // First request
528+
Assert.Equal(0, capturedContext.FunctionSequenceIndex); // First function in the batch
529+
Assert.Equal(1, capturedContext.FunctionCount); // One function call in this response
530+
Assert.NotNull(capturedContext.Function);
531+
Assert.Equal("Now", capturedContext.Function.Name);
532+
Assert.NotNull(capturedContext.ChatHistory);
533+
Assert.NotNull(capturedContext.Result);
534+
}
535+
536+
[Fact]
537+
public async Task IfAutoInvokeShouldTerminateWhenFilterRequestsTerminationAsync()
538+
{
539+
// Arrange
540+
int filterInvocationCount = 0;
541+
var autoFunctionInvocationFilter = new AutoFunctionInvocationFilter(async (context, next) =>
542+
{
543+
filterInvocationCount++;
544+
context.Terminate = true;
545+
await next(context);
546+
});
547+
548+
var kernel = new Kernel();
549+
kernel.Plugins.Add(KernelPluginFactory.CreateFromFunctions("TimePlugin", new[]
550+
{
551+
KernelFunctionFactory.CreateFromMethod((string param1)
552+
=> DateTime.Now.ToString("", CultureInfo.InvariantCulture), "Now", "TimePlugin.Now"),
553+
554+
KernelFunctionFactory.CreateFromMethod((string format)
555+
=> DateTime.Now.ToString("", CultureInfo.InvariantCulture), "Date", "TimePlugin.Date"),
556+
}));
557+
kernel.AutoFunctionInvocationFilters.Add(autoFunctionInvocationFilter);
558+
559+
// Use multiple function calls response to verify termination stops processing additional tool calls
560+
var responseContentWithMultipleFunctions = File.ReadAllText("./TestData/chat_multiple_function_calls_response.json")
561+
.Replace("%nameSeparator%", GeminiFunction.NameSeparator, StringComparison.Ordinal);
562+
563+
using var handlerStub = new MultipleHttpMessageHandlerStub();
564+
handlerStub.AddJsonResponse(responseContentWithMultipleFunctions);
565+
handlerStub.AddJsonResponse(this._responseContent); // This should not be called due to termination
566+
567+
#pragma warning disable CA2000
568+
var client = this.CreateChatCompletionClient(httpClient: handlerStub.CreateHttpClient());
569+
#pragma warning restore CA2000
570+
var chatHistory = CreateSampleChatHistory();
571+
var executionSettings = new GeminiPromptExecutionSettings
572+
{
573+
ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions
574+
};
575+
576+
// Act
577+
var result = await client.GenerateChatMessageAsync(chatHistory, executionSettings: executionSettings, kernel: kernel);
578+
579+
// Assert
580+
// Filter should have been invoked only once (for the first tool call) because termination was requested
581+
Assert.Equal(1, filterInvocationCount);
582+
// Only 1 request should be made since termination happens after receiving the tool calls
583+
// but before making the second request to the model with the tool results
584+
Assert.Single(handlerStub.RequestContents);
585+
}
586+
587+
[Fact]
588+
public async Task IfAutoInvokeShouldAllowFilterToModifyFunctionResultAsync()
589+
{
590+
// Arrange
591+
const string ModifiedResult = "Modified result by filter";
592+
var autoFunctionInvocationFilter = new AutoFunctionInvocationFilter(async (context, next) =>
593+
{
594+
await next(context);
595+
// Modify the result after function execution
596+
context.Result = new FunctionResult(context.Function, ModifiedResult);
597+
});
598+
599+
var kernel = new Kernel();
600+
kernel.Plugins.Add(KernelPluginFactory.CreateFromFunctions("TimePlugin", new[]
601+
{
602+
KernelFunctionFactory.CreateFromMethod(()
603+
=> "Original result", "Now", "TimePlugin.Now"),
604+
}));
605+
kernel.AutoFunctionInvocationFilters.Add(autoFunctionInvocationFilter);
606+
607+
using var handlerStub = new MultipleHttpMessageHandlerStub();
608+
handlerStub.AddJsonResponse(this._responseContentWithFunction);
609+
handlerStub.AddJsonResponse(this._responseContent);
610+
611+
#pragma warning disable CA2000
612+
var client = this.CreateChatCompletionClient(httpClient: handlerStub.CreateHttpClient());
613+
#pragma warning restore CA2000
614+
var chatHistory = CreateSampleChatHistory();
615+
var executionSettings = new GeminiPromptExecutionSettings
616+
{
617+
ToolCallBehavior = GeminiToolCallBehavior.AutoInvokeKernelFunctions
618+
};
619+
620+
// Act
621+
await client.GenerateChatMessageAsync(chatHistory, executionSettings: executionSettings, kernel: kernel);
622+
623+
// Assert - Check that the modified result was sent to the model
624+
var secondRequestContent = handlerStub.GetRequestContentAsString(1);
625+
Assert.NotNull(secondRequestContent);
626+
Assert.Contains(ModifiedResult, secondRequestContent);
627+
}
628+
443629
private static ChatHistory CreateSampleChatHistory()
444630
{
445631
var chatHistory = new ChatHistory();
@@ -465,4 +651,19 @@ public void Dispose()
465651
this._httpClient.Dispose();
466652
this._messageHandlerStub.Dispose();
467653
}
654+
655+
private sealed class AutoFunctionInvocationFilter : IAutoFunctionInvocationFilter
656+
{
657+
private readonly Func<AutoFunctionInvocationContext, Func<AutoFunctionInvocationContext, Task>, Task> _callback;
658+
659+
public AutoFunctionInvocationFilter(Func<AutoFunctionInvocationContext, Func<AutoFunctionInvocationContext, Task>, Task> callback)
660+
{
661+
this._callback = callback;
662+
}
663+
664+
public async Task OnAutoFunctionInvocationAsync(AutoFunctionInvocationContext context, Func<AutoFunctionInvocationContext, Task> next)
665+
{
666+
await this._callback(context, next);
667+
}
668+
}
468669
}

0 commit comments

Comments
 (0)