Openai.cr 0.3.0 Release

June 23, 2023

Functions! We love when our jobs are made easier. Recently OpenAI announced the support for functions on their blog. Functions allow you to specify the structure of your output in a JSON compatible format. In the past, you’d have to wrangle the output by asking the GPT API to return a computer readable format like YAML or JSON, and specify the fields in natural language. The output would be unreliable. Most of the time I would resort to asking for YAML and I’d have to be very specific about the kinds of fields I would get back so I would get consistent results. I’m excited about this change and naturally had to add it to my OpenAI library!

As of today, we support functions using both the normal and streaming options on the chat method. Here are some example usages:

class CatNameResponse
  include JSON::Serializable

  @[JSON::Field(description: "A name of a cat")]
  getter cats : Array(String)
end

list_cats = OpenAI.def_function("list_cats", "A list of cat names", CatNameResponse)

client = OpenAI::Client.new

response = client.chat("gpt-3.5-turbo-0613", [
  {role: "user", content: "Give me a list of names for my cat."},
], {"functions" => [list_cats]})

pp response
pp CatNameResponse.from_json(response.result(0))

Notice that this is quite a bit different from the way you would use functions in an interpretted language. Normally you would send the entire json schema payload. I built this internally into the client so you can write crystal structs or classes and it will generate the JSON for you. I think this is a huge advantage for statically typed languages! (This feature was made possible with the json-schema shard!)

Here is an example using streaming:

class CatNameResponse
  include JSON::Serializable

  @[JSON::Field(description: "A name of a cat")]
  getter cats : Array(String)
end

list_cats = OpenAI.def_function("list_cats", "A list of cat names", CatNameResponse)

client = OpenAI::Client.new

output = ""
client.chat("gpt-3.5-turbo-0613", [
  {role: "user", content: "Give me a list of 30 names for a cat."},
], {"stream" => true, "functions" => [list_cats]}) do |chunk|
  if function_call = chunk.choices.first.delta.function_call
    output += function_call.arguments

    pp function_call.arguments
  end
end

pp CatNameResponse.from_json(output)

With streaming, you can deliver output to the user as the data flows in, parsing the final output at the end. Unfortunately I wasn’t able to find any good SAX JSON parsers so I’m not sure how to parse streaming JSON in Crystal yet. I will add this functionality if someone makes a shard for it!

This is a huge step for autonomous systems like guppi. It’s like lang chain built into OpenAI directly.

Some things to remember:

1. You must specify the 0613 models when using functions. OpenAI created special versions of their model specifically fine tuned to handle functions and JSON output.
2. With all LLM’s, you’re going to have some inconsistencies so always make sure to have robust JSON parse error handling.
3. Functions do use up more tokens in the request, but I think this is worth it to create more robust systems and may even reduce prompt sizes that had overspecified rules to deal with parsing outputs.

Check out openai.cr here: https://github.com/lancecarlson/openai.cr