Harnessing LLMs

Workflow Demo - May 28, 2025

Intended audience

R and Python users
You’ve used LLMs via ChatGPT, Copilot, or similar
But you haven’t used LLMs from code

Getting Started with LLMs

Most LLMs are accessible through HTTP APIs
Posit has packages designed for calling LLMs
- R: ellmer
- Python: chatlas

What to expect

👍 Very easy to use
👍 Highly capable, and improving
👍 Extensible
👎 Often makes mistakes
👎 Nondeterministic, unpredictable
Summary: LLMs have a jagged frontier that is worth exploring through code.

How not to start

Do NOT start with local models
- Start with the best models (Claude 3.7 Sonnet and OpenAI’s latest, possibly Google Gemini 2.5)
Do NOT start with non-public information or data
- Get a good grasp of how these APIs work first, and only then think about security and data privacy

How to start

Sign up for a developer account with Anthropic or OpenAI and grab an API key (requires credit card)
- Just do it already 🙄
Add API key to your environment variables (R, Python)
Install an LLM client package
- For R: install.packages("ellmer")
- For Python: pip install chatlas

Hello, AI (in R)

Start a conversation using a chat_* function (like chat_anthropic, chat_openai, chat_ollama, etc.)

library(ellmer)

client <- chat_anthropic(model = "claude-3-7-sonnet-latest")
client$chat("Summarize the plot of Romeo and Juliet in 20 words or less.")

Two teens from feuding families fall in love, marry secretly, and die 
tragically due to miscommunication.

Continue the conversation by calling chat() again.

client$chat("Now Hamlet.")

Danish prince feigns madness to avenge his father's murder, leading to tragic 
deaths including his own.

Hello, AI (in Python)

Start a conversation using a Chat object (like ChatAnthropic, ChatOpenAI, ChatOllama, etc.)

from chatlas import ChatAnthropic

client = ChatAnthropic(model = "claude-3-7-sonnet-latest")
client.chat("Summarize the plot of Romeo and Juliet in 20 words or less.")

Two young lovers from feuding families meet, secretly marry, and tragically
die in a misunderstanding.

Continue the conversation by calling chat() again.

client.chat("Now Hamlet.")

Danish prince feigns madness to avenge his father's murder while contemplating 
existence, ultimately causing multiple deaths including his own.

ellmer/chatlas basics

Support many different LLM providers
Print the object to view conversation history
Built-in chat UI
- R: live_browser(client) or live_console(client)
- Python: client.app() or client.console()

Adding knowledge

Do NOT start with RAG
Do NOT start with fine tuning
Start with system prompt customization

Example 1

library(ellmer)

system_prompt <- paste(
  "You're a bot that answers questions based on this expense policy doc:",
  "<expense_policy>",
  readLines("expense-policy.md"),
  "</expense_policy>",
  "* Cite policy section numbers.",
  "* Use examples when helpful.",
  collapse = "\n"
)

client <- chat_anthropic(system_prompt, model = "claude-3-7-sonnet-latest")
client$chat("Can I expense a programming book?")

Based on the expense policy, programming books may be expensible under Training and Development (Section 3.4) if they're job-related and pre-approved. They would likely fall under "Courses and Certifications" which "must be job-related and approved by both the employee's manager and HR/L&D" (Section 3.4). Ensure you get proper pre-approval before purchasing.

Example 2

library(ellmer)

system_prompt <- paste(
  "You're a bot that helps data analysis for College Scorecard data.",
  "Do not attempt to analyze data; however, you may suggest R code instead.",
  "Use readr to read CSVs. You can assume Most-Recent-Cohorts-Institution.csv ",
  "and Most-Recent-Cohorts-Field-of-Study.csv are in the working directory.",
  "",
  readLines("college_scorecard_data_summary.md"),
  collapse = "\n"
)

client <- chat_anthropic(system_prompt, model = "claude-sonnet-4-20250514")
live_browser(client)

Customizing behavior

from chatlas import ChatAnthropic

with open("expense-policy.md", "r") as f:
    expense_policy = f.read()

system_prompt = f"""
You're a dungeon master for a D&D style game whose theme is this
expense policy. Start with character creation.

<expense_policy>
{expense_policy}
</expense_policy>
"""

client = ChatAnthropic(
  system_prompt=system_prompt,
  model="claude-3-7-sonnet-latest",
)
client.app()

A warning

On their own, LLMs are terrible at analyzing data
They cannot do simple arithmetic reliably
They cannot even count rows reliably
Having LLMs write code that analyzes data (and sometimes, even execute it) is the way

Creating custom apps

You can use Shiny as a frontend
- R: {shinychat}
- Python: ui.Chat
- These are designed to work with ellmer and chatlas
Deploy your app with one click
- On premises: Posit Connect
- Cloud: Posit Connect Cloud

More ellmer/chatlas features

Image input
Structured output
Stream responses as they are received
Optional async

Vision (in R)

library(ellmer)

client <- chat_anthropic(model = "claude-3-5-haiku-latest")
client$chat(
  "Describe the mood of this image in a single phrase.",
  content_image_file("photo.jpg", resize="low")
)

Blissful anticipation of a sweet treat.

Vision (in Python)

from chatlas import ChatAnthropic, content_image_file

client = ChatAnthropic(model = "claude-3-5-haiku-latest")
client.chat(
  "Describe the mood of this image in a single phrase.",
  content_image_file("photo.jpg", resize="low")
);

Delightful anticipation.

Structured output (in R)

library(ellmer)

image_info = type_object(
  subject = type_string("The main subject of the image"),
  object = type_string("The main object being acted upon"),
  colors = type_array("Major colors present in the image",
    type_string()
  )
)

client <- chat_anthropic(model = "claude-3-5-haiku-latest")
client$chat_structured(
  "Tell me about this image in the specified format.",
  content_image_file("photo.jpg", resize="low"),
  type = image_info
)

$subject
[1] "Child"

$object
[1] "S'more"

$colors
[1] "gray"  "white" "brown"

Structured output (in Python)

from pydantic import BaseModel, Field
from chatlas import ChatAnthropic, content_image_file

class ImageInfo(BaseModel):
    subject: str = Field("Main subject of the image")
    object: str = Field("Main object being acted upon")
    colors: list[str] = Field("Major colors present in the image")

client = ChatAnthropic(model = "claude-3-5-haiku-latest")
client.extract_data(
  "Tell me about this image in the specified format.",
  content_image_file("photo.jpg", resize="low"),
  data_model = ImageInfo
)

{'subject': 'Child eating a snack',
 'object': "S'more",
 'colors': ['gray', 'white', 'brown']}

Tool calling

Allows LLMs to interact with other systems
Sounds complicated? Not as much as you think!

How It Works

Not like this - with the assistant executing stuff
Yes like this
- User asks assistant a question; includes metadata for available tools
- Assistant asks the user to invoke a tool, passing its desired arguments
- User invokes the tool, and returns the output to the assistant
- Assistant incorporates the tool’s output as additional context for formulating a response

How It Works

Another way to think of it:

The client can perform tasks that the assistant can’t do
Tools put control into the hands of the assistant—it decides when to use them, and what arguments to pass in, and what to do with the results
Having an “intelligent-ish” coordinator of tools is a surprisingly general, powerful capability!