This vignette gives you some advice about how to use elmer to write prompts. We’ll work through two hopefully relevant examples: a prompt that generates code and another that extracts structured data. If you’ve never written a prompt, I’d highly recommend reading Ethan Mollick’s Getting started with AI: Good enough prompting. I think understanding his analogy about how AI works will really help you get started:
Treat AI like an infinitely patient new coworker who forgets everything you tell them each new conversation, one that comes highly recommended but whose actual abilities are not that clear. … Two parts of this are analogous to working with humans (being new on the job and being a coworker) and two of them are very alien (forgetting everything and being infinitely patient). We should start with where AIs are closest to humans, because that is the key to good-enough prompting
As well as learning general prompt design skills, it’s also a good idea to read any specific advice for the model that you’re using. Here are some pointers to the prompt design guides of some of the most popular models:
If you have a claude account, you can use its prompt-generator. It’s specifically tailored for Claude, but I suspect it will help you with many other LLMs, or at least give you some ideas as to what else to include in your prompt.
Best practices
It’s highly likely that you’ll end up writing long, possibly multi-page prompts. To ensure your success with this task, we have two recommendations. First, put each prompt its own, separate file. Second, write the prompts using markdown. The reason to use markdown is that it’s quite readable to LLMs (and humans), and it allows you to do things like use headers to divide up a prompt into sections and itemised lists to enumerate multiple options. You can see some examples of this style of prompt here:
- https://github.com/posit-dev/shiny-assistant/blob/main/shinyapp/app_prompt_python.md
- https://github.com/jcheng5/py-sidebot/blob/main/prompt.md
- https://github.com/simonpcouch/pal/tree/main/inst/prompts
- https://github.com/cpsievert/aidea/blob/main/inst/app/prompt.md
In terms of file names, if you only have one prompt in your project,
call it prompt.md. If you have multiple prompts, give them
informative names like prompt-extract-metadata.md or
prompt-summarize-text.md. If you’re writing a package, put
your prompt(s) in inst/prompts, otherwise it’s fine to put
them in the project’s root directory.
Your prompts are going to change over time, so we’d highly recommend commiting them to a git repo. That will ensure that you can easily see what has changed, and that if you accidentally make a mistake you can easily roll back to a known good verison.
If your prompt includes dynamic data, use
elmer::interpolate_file() to intergrate it into your
prompt. interpolate_file() works like glue but uses {{ }}
instead of { } to make it easier to work with JSON.
As you iterate the prompt, it’s a good idea to build up a small set of challenging examples that you can regularly re-check with your latest version of the prompt. Currently you’ll need to do this by hand, but we hope to eventually provide tools that’ll help you do this a little more formally.
Unfortunately, you won’t see these best practices in action in this vignette since we’re keeping the prompts short and inline to make it easier for you to grok what’s going on.
Code generation
Let’s explore prompt design for a simple code generation task:
question <- "
How can I compute the mean and median of variables a, b, c, and so on,
all the way up to z, grouped by age and sex.
"I’ll use chat_claude() for this problem because in our
experience it does the best job of generating code.
Basic flavour
When I don’t provide a system prompt, I sometimes get answers in a different languages or different styles of R code:
chat <- chat_claude()
chat$chat(question)I can ensure that I always get R code in a given style by providing a system prompt:
chat <- chat_claude(system_prompt = "
You are an expert R programmer who prefers the tidyverse.
")
chat$chat(question)Note that I’m using both a system prompt (which defines the general behaviour) and a user prompt (which askes the specific question). You could put all of the content in the user prompt and get similar results, but I think it’s helpful to use both to cleanly divide the general framing of the response from the specific questions that you want to ask.
Since I’m mostly interested in the code, I ask it to drop the explanation and the sample data:
chat <- chat_claude(system_prompt = "
You are an expert R programmer who prefers the tidyverse.
Just give me the code. I don't want any explanation or sample data.
")
chat$chat(question)And of course, if you want a different style of R code, you can of course ask for it:
chat <- chat_claude(system_prompt = "
You are an expert R programmer who prefers data.table.
Just give me the code. I don't want any explanation or sample data.
")
chat$chat(question)
chat <- chat_claude(system_prompt = "
You are an expert R programmer who prefers base R.
Just give me the code. I don't want any explanation or sample data.
")
chat$chat(question)Be explicit
If there’s something about the output that you don’t like, you can try being more explicit about it. For example, the code isn’t styled quite how I like, so I provide more details about what I do want:
chat <- chat_claude(system_prompt = "
You are an expert R programmer who prefers the tidyverse.
Just give me the code. I don't want any explanation or sample data.
Follow the tidyverse style guide:
* Spread long function calls across multiple lines.
* Where needed, always indent function calls with two spaces.
* Only name arguments that are less commonly used.
* Always use double quotes for strings.
* Use the base pipe, `|>`, not the magrittr pipe `%>%`.
")
chat$chat(question)This still doesn’t yield exactly the code that I’d write, but it’s prety close.
You could provide a different prompt if you were looking for more explanation of the code:
chat <- chat_claude(system_prompt = "
You are an expert R teacher.
I am a new R user who wants to improve my programming skills.
Help me understand the code you produce by explaining each function call with
a brief comment. For more complicated calls, add documentation to each
argument. Just give me the code. I don't want any explanation or sample data.
")
chat$chat(question)Teach it about new features
You can imagine LLMs as being a sort of an average of the internet at a given point in time. That means they will provide popular answers, which will tend to reflect older coding styles (either because the new features aren’t in their index, or the older features are so much more popular). So if you want your code to use specific features that are relatively recent, you might need to provide the examples yourself:
chat <- chat_claude(system_prompt = "
You are an expert R programmer.
Just give me the code; no explanation in text.
Use the `.by` argument rather than `group_by()`.
dplyr 1.1.0 introduced per-operation grouping with the `.by` argument.
e.g., instead of:
transactions |>
group_by(company, year) |>
mutate(total = sum(revenue))
write this:
transactions |>
mutate(
total = sum(revenue),
.by = c(company, year)
)
")
chat$chat(question)Structured data
Providing a rich set of examples is a great way to encourage the output to produce exactly what you want. This is also known as multi-shot prompting. Here we’ll work through a prompt that I designed to extract structured data from recipes, but the same ideas apply in many other situations.
Getting started
My overall goal is to turn a list of ingredients, like the following, into a nicely structured JSON that I can then analyse in R (e.g. to compute the total weight, scale the recipe up or down, or to convert the units from volumes to weights).
ingredients <- "
¾ cup (150g) dark brown sugar
2 large eggs
¾ cup (165g) sour cream
½ cup (113g) unsalted butter, melted
1 teaspoon vanilla extract
¾ teaspoon kosher salt
⅓ cup (80ml) neutral oil
1½ cups (190g) all-purpose flour
150g plus 1½ teaspoons sugar
"(This isn’t the ingredient list for a real recipe but it includes a sampling of styles that I encountered in my project.)
If you don’t have strong feelings about what the data structure should look like, you can start with a very loose prompt and see what you get back. I find this a useful pattern for underspecified problems where a big part of the problem is just defining precisely what problem you want to solve. Seeing the LLM’s attempt at a data structure gives me something to immediately react to, rather than having to start from a blank page.
instruct_json <- "
You're an expert baker who also loves JSON. I am going to give you a list of
ingredients and your job is to return nicely structured JSON. Just return the
JSON and no other commentary.
"
chat <- chat_openai(instruct_json)
chat$chat(ingredients)(I don’t know if the colour text, “You’re an expert baker who also loves JSON”, does anything, but I like to think this helps the LLM get into the right mindset of a very nerdy baker.)
Provide examples
This isn’t a bad start, but I prefer to cook with weight and I only want to see volumes if weight isn’t available so I provide a couple of examples of what I’m looking for. I was pleasantly suprised that I can provide the input and output examples in such a loose format.
instruct_weight <- r"(
Here are some examples of the sort of output I'm looking for:
¾ cup (150g) dark brown sugar
{"name": "dark brown sugar", "quantity": 150, "unit": "g"}
⅓ cup (80ml) neutral oil
{"name": "neutral oil", "quantity": 80, "unit": "ml"}
2 t ground cinnamon
{"name": "ground cinnamon", "quantity": 2, "unit": "teaspoon"}
)"
chat <- chat_openai(c(instruct_json, instruct_weight))
chat$chat(ingredients)Just providing the examples seems to work remarkably well. But I found it useful to also include description of what the examples are trying to accomplish. I’m not sure if this helps the LLM or not, but it certainly makes it easier for me to understand the organisation of the whole prompt and check that I’ve covered the key pieces that I’m interested in.
instruct_weight <- r"(
* If an ingredient has both weight and volume, extract only the weight:
¾ cup (150g) dark brown sugar
[
{"name": "dark brown sugar", "quantity": 150, "unit": "g"}
]
* If an ingredient only lists a volume, extract that.
2 t ground cinnamon
⅓ cup (80ml) neutral oil
[
{"name": "ground cinnamon", "quantity": 2, "unit": "teaspoon"},
{"name": "neutral oil", "quantity": 80, "unit": "ml"}
]
)"This structure also allows me to give the LLMs a hint about how I want multiple ingredients to be stored, i.e. as an JSON array.
I then iterated on the prompt, looking at the results from different recipes to get a sense of what the LLM was getting wrong. Much of this felt like I waws iterating on my understanding of the problem as I didn’t start by knowing exactly how I wanted the data. For example, when I started out I didn’t really think about all the various ways that ingredients are specified. For later analysis, I always want quantities to be number, even if they were originally fractions, or the if the units aren’t precise (like a pinch). It made me realise that some ingredients are unitless.
instruct_unit <- r"(
* If the unit uses a fraction, convert it to a decimal.
⅓ cup sugar
½ teaspoon salt
[
{"name": "dark brown sugar", "quantity": 0.33, "unit": "cup"},
{"name": "salt", "quantity": 0.5, "unit": "teaspoon"}
]
* Quantities are always numbers
pinch of kosher salt
[
{"name": "kosher salt", "quantity": 1, "unit": "pinch"}
]
* Some ingredients don't have a unit.
2 eggs
1 lime
1 apple
[
{"name": "egg", "quantity": 2},
{"name": "lime", "quantity": 1},
{"name", "apple", "quantity": 1}
]
)"You might want to take a look at the full prompt to see what I ended up with.
Structured data
Now that I’ve iterated to get a data structure that I like, it seems useful to formalise it and tell the LLM exactly what I’m looking for using structured data. This guarantees that the LLM will only return JSON, the JSON will have the fields that you expect, and that elmer will convert it into an R data structure for you.
type_ingredient <- type_object(
name = type_string("Ingredient name"),
quantity = type_number(),
unit = type_string("Unit of measurement")
)
type_ingredients <- type_array(items = type_ingredient)
chat <- chat_openai(c(instruct_json, instruct_weight))
data <- chat$extract_data(ingredients, type = type_object(ingredients = type_ingredients))
do.call(rbind, lapply(data$ingredients, as.data.frame))Capturing raw input
One thing that I’d do next time would also be to include the raw ingredient name in the output. This doesn’t make much difference here, in this simple example, but it makes it much easier to align the input and the output and start to develop automated measures of how well my prompt is doing.
instruct_weight_input <- r"(
* If an ingredient has both weight and volume, extract only the weight:
¾ cup (150g) dark brown sugar
[
{"name": "dark brown sugar", "quantity": 150, "unit": "g", "input": "¾ cup (150g) dark brown sugar"}
]
* If an ingredient only lists a volume, extract that.
2 t ground cinnamon
⅓ cup (80ml) neutral oil
[
{"name": "ground cinnamon", "quantity": 2, "unit": "teaspoon", "input": "2 t ground cinnamon"},
{"name": "neutral oil", "quantity": 80, "unit": "ml", "input": "⅓ cup (80ml) neutral oil"}
]
)"I think this is particularly important if you’re working with even less structured text. For example, imagine you had this text:
recipe <- r"(
In a large bowl, cream together one cup of softened unsalted butter and a
quarter cup of white sugar until smooth. Beat in an egg and 1 teaspoon of
vanilla extract. Gradually stir in 2 cups of all-purpose flour until the
dough forms. Finally, fold in 1 cup of semisweet chocolate chips. Drop
spoonfuls of dough onto an ungreased baking sheet and bake at 350°F (175°C)
for 10-12 minutes, or until the edges are lightly browned. Let the cookies
cool on the baking sheet for a few minutes before transferring to a wire
rack to cool completely. Enjoy!
)"Including the input text in the output makes it easier to see if it’s doing a good job:
chat <- chat_openai(c(instruct_json, instruct_weight_input))
chat$chat(recipe)When I ran it while writing this vignette, it seems to be working out the weight of the ingredients specified in volume, even though the prompt specifically asks it not to do that. This may suggest I need to broaden my examples.