LLM Function Calling

In the ever-expanding world of LLMs, "function calling" allows you to define custom functions that a model can use to extend its functionality and knowledge by giving it access to external APIs.

These functions help supply data and abilities that Large Language Models typically don't know about, such as (but certainly not limited to):

• Real-time data or current events
• Personal information, such as calendar appointments, emails, todo list items
• Business informatics, such as sales data, customer information, or support queries
• The ability to act upon data and make changes to or update information stored elsewhere

These functions, or "tools" as they are sometimes referred to, are provided to the model as a function description. This description tells the model everything it needs to know about what the function can achieve and what it needs to know to achieve it.

How function calling works

Step-by-step, almost all models follow this approach:

1. User submits a prompt
2. Prompt is submitted to an application responsible for handling the communication between the user and the model, along with one or more tool/function definitions. These are both passed to the model.
3. The model chooses a function that suits the execution of the user prompt
4. Model returns the chosen function name and values to the application
5. Application executes the function and returns the API response to the model
6. The model uses this response to determine the prompt response
7. Response is returned to the user via the application

LLMs don't execute functions at all (although this might change in the future). For now, an application must bridge the user and the model. This application will have the logic to execute any functions the model chooses.

Function calling overview

Let's take a look at how these tools are implemented in some of the major models, because altough conceptually the same, there are some small, and frustrating differences. To help with the comparison we'll use the ever popular "Get the current weather in..." example.

OpenAI

The first of the most recent crop of LLMs to implement function calling in their API for their gpt-3.5-turbo-0125 and gpt-4-turbo-preview models. They also allow users to build tools into their custom GPTs in the form of GPT Actions using Open API Specifications, which we won't cover here.

OpenAI expects any tools you want the model to be aware of to be defined using a JSON schema aligned with the Open API JSON schema:

[
  {
    "type": "function",
    "function": {
      "name": "get_current_weather",
      "description": "Get the current weather in a given location",
      "parameters": {
        "type": "object",
        "properties": {
          "location": {
            "type": "string",
            "description": "The city and state, e.g. San Francisco, CA"
          },
          "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] }
        },
        "required": ["location"]
      }
    }
  }
]

MistralAI

Great news! Mistral's JSON schema for tools is the same as OpenAI's. This means any function descriptions you have defined for one will work with the other without any changes.

Additionally, Mistral differentiates tool choice responses using a special purpose role of tool to keep them separate from the user and assistant roles. OpenAI also uses this approach.

Anthropic Claude3

Claude is the latest LLM to support function calling via API (in Beta as of April 2024). Again, the approach is the same, but unlike OpenAI and MistralAI, there are some differences to be aware of.

The first is with the schema. On the surface, it looks almost the same, but it isn't. Here, there is no requirement to declare a function object first, and the parameter definition lives in an object called input_schema rather than parameters.

Outside of that, the definition schema is the same. Some tweaking will be required to use tools that have previously been defined for OpenAI or MistralAI.

{
  "name": "get_current_weather",
  "description": "Get the current weather in a given location",
  "input_schema": {
    "type": "object",
    "properties": {
      "location": {
        "type": "string",
        "description": "The city and state, e.g. San Francisco, CA"
      },
      "unit": {
        "type": "string",
        "enum": ["celsius", "fahrenheit"],
        "description": "The unit of temperature, either 'celsius' or 'fahrenheit'"
      }
    },
    "required": ["location"]
  }
}

The second difference is in how the tool choice responses are differentiated. Anthropic's APIs use a pattern of alternating user and assistant messages, each containing an array of content blocks like text, image or tool_result.

In this format, any responses your application gets from an API when executing a function can be passed back to the model using the user role. The data is contained in a tool_result content block to aid with differentiation.

Cohere Command-R

The Cohere team's Command-R model also supports function calling. Again, the approach is similar to all of the above models, but the function definition schema is slightly different.

[
  {
    "name": "get_current_weather",
    "description": "Get the current weather in a given location",
    "parameter_definitions": {
      "location": {
        "description": "The city and state, e.g. San Francisco, CA",
        "type": "str",
        "required": True
      },
      "unit": {
        "description": "The unit of temperature, either 'celsius' or 'fahrenheit'",
        "type": "str",
        "enum": ["celcius", "farenheit"]
      }
    }
  }
]

Cohere's API expects an even leaner function definition that removes some of the extra explicit typing used by Anthropic, Mistral, and OpenAI and gets right into the parameter_defintions.

Unlike the others above, where required parameters are listed in a specific required array, Cohere expects that any necessary parameters are identified explicitly. This approach to the schema likely means that Cohere does not follow the Open API JSON schema approach used by the other models.

Google Gemini 1.0 Pro

Gemini's function calling approach follows the Open API JSON schema for function definitions, which means that using the same weather example as OpenAI and MistralAI would also work out of the box without any changes.

Others

These are just some of the models that now support function calling, but they are by no means a complete list.

You'll also find the functionality available in other models, such as Llama-34b, which can be used via Together.ai, Fireworks AI's Firefunction v1, and there are no doubt many more to come.

Observations

Most LLMs are sticking with using the Open API JSON schema as the basis for their definitions, which, given the early nature of this functionality, is encouraging as it allows for a relatively high level of cross-compatibility.

This cross-compatibility presents an opportunity for developers, allowing them to switch models based on many factors, including use case, cost, speed, and training.

With ever-expanding, community-powered development frameworks such as LangChain adding binding for function calling in more and more models every week, the time to implementation is getting shorter all the time.

Superface & Function Calling

If you like the idea of working with function calling in your application or agent, but want to save time on building function descriptors for tools and more functions to work with their APIs, Superface's Hub API is exactly what you need.

Using the Hub API you automatically get function descriptions for any tools that have been added to your Superface account. These function descriptions can be used by OpenAI, MistralAI, Anthropic, and LangChain.

The Hub API also allows you to securely authenticate users so they can provide their own credentials for any of the tools that Superface offers, inside your agent or application.

For more information on how to implement this in your application, check out our documentation, or drop us an email on support@superface.ai and we'll gladly tell you more.