Support for pdf content

[abh1shek-sh]

As of now, the Message can be either of type :text, image_url or :image, .

Would it be possible to also add the types: :pdf and :pdf_url.
To me, it looks like the process of sending the pdf to the llm is very similar to that of image. Atleast, in the case of Gemini APIs.

Here is code of snippet from the Gemini SDK:

from google import genai
from google.genai import types
import pathlib
import httpx

client = genai.Client()

doc_url = "https://discovery.ucl.ac.uk/id/eprint/10089234/1/343019_3_art_0_py4t4l_convrt.pdf"  # Replace with the actual URL of your PDF

# Retrieve and encode the PDF byte
filepath = pathlib.Path('file.pdf')
filepath.write_bytes(httpx.get(doc_url).content)

prompt = "Summarize this document"
response = client.models.generate_content(
  model="gemini-1.5-flash",
  contents=[
      types.Part.from_bytes(
        data=filepath.read_bytes(),
        mime_type='application/pdf',
      ),
      prompt])
print(response.text)

Since, the pdf parsing library in Elixir aren't that great, the general advice is to either use NIFs or rely on third party services. I think this would be very helpful if it could be implemented.

Thanks!

[mindok]

There's probably a more general case to consider - OpenAI supports PDFs via general file attachments - see https://platform.openai.com/docs/guides/pdf-files (the relevant bit is below). The Message.ContentPart should probably support attaching arbitrary attributes. It looks like the options are constrained in the models (e.g. for_api functions in LangChain.ChatModels.ChatOpenAI). Some way of overriding / forcing this would be great (with appropriate warnings!)

curl "https://api.openai.com/v1/responses" \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $OPENAI_API_KEY" \
    -d '{
        "model": "gpt-4o",
        "input": [
            {
                "role": "user",
                "content": [
                    {
                        "type": "input_file",
                        "filename": "draconomicon.pdf",
                        "file_data": "...base64 encoded PDF bytes here..."
                    },
                    {
                        "type": "input_text",
                        "text": "What is the first dragon in the book?"
                    }
                ]
            }
        ]
    }'

[brainlid]

I can see a more general mime-type like support for attachment types. The constraint would be that the model must support it.

[abh1shek-sh]

Sounds good to me, Almost all of the major providers support attachment of various types.
A list of attachment types could be maintained for each model/provider, I guess.

[nileshtrivedi]

I think atleast OpenAI and Gemini APIs now expect a "message" to be made of multiple "parts", each with its own type and object structure.

Here are OpenAI's input types:

Here are OpenAI's message part types:

message
file_search_call
function_call
web_search_call
computer_call
reasoning
image_generation_call
code_interpreter_call
local_shell_call
mcp_call
mcp_list_tools
mcp_approval_request

Basically, these vendors APIs are more and more for their "agents" and not just "LLMs". See #289

[brainlid]

Please check out v0.4.0-rc.1. It changes more things to use ContentParts which have types. PDF is a type as well.

I've used PDFs with ChatAnthropic, but I haven't used it with Gemini personally.

[conpat]

#478 Implements inline files (PDFs and CSVs) for Gemini the same way it is already implemented for others (at least Anthropic and Orq)

[brainlid]

Thanks @conpat!

@abh1shek-sh, can you create a PR to add the :pdf file types you want and the support for it in the model you want? That way you can validate the behavior for it too.

[abh1shek-sh]

@brainlid, I convert pdf to markdown using a Docling as an alternative to uploading the PDFs directly to models now, I think this is more reliable compared to uploading large files to LLM.
I usually use Vertex AI now, so I can make the same changes in #478 for Vertex AI and test it as well. What do you think ?

[brainlid]

Hi @abh1shek-sh! Thanks for the update. Docling looks really nice. I haven't had the need for working with such rich and complex PDFs.

Can you be a bit more clear about the change you're proposing for Vertex AI? Are you suggesting including Docling in the flow somehow? I ask because it sounds like if you're using Docling first, then you just have markdown when sending it to the LLM. So I'm not clear in how a PDF would come into it?

[abh1shek-sh]

@brainlid I meant the changes in the PR #478 are only for ChatGoogleAI, should I do it for ChatVertexAI as well ?