Imagine getting this cold email:

Why should you be the one to answer it? What do you get from answering it?
Needless to say: LLMs are horrible at writing cold emails.
I’ve written 500+ cold emails, and at Stanford GSB we spent time on this in Winning Writing taught by Glenn Kramon, a New York Times editor. The crux of the class is: how do you make your writing one in a million rather than one of a million?
While learning this, I observed that more companies were training their own specialized LLMs. Vercel trained a model to fix errors in the code v0 generates. Harvey post-trained a model for legal work and published how it did on its own benchmark. I had a much smaller job in mind: Could I use the same approach to teach a model what I’d learned from writing cold emails?

Harvey and Applied Compute report rubric pass rates of 85.3% for base GLM-5.1 and 91.3% after post-training on their Legal Agent Benchmark. Source
What I tried
One of the best cold emails I've sent targeted someone working on ChatGPT pricing. I leveraged my experience working on pricing at Shippo and research on AI memory systems to pitch that OpenAI can charge for AI memory. I used Tinker to run Qwen3.5-4B on this email and its context. The results were promising, and I decided to expand the experiment.
I took the LoRA adapter from Tinker and started testing it on new emails. I gave it a brief for someone at Vercel, aiming to talk about her career in AI and connect over my experience using Vercel. The model carried over parts of the first pitch, but hallucinated and said I worked at Vercel. Despite the original promise, the tuned version was horrible.
Thinking that data was the problem, I revisited 30 of the best cold emails I’ve sent. For each one, I built a brief around what I knew at the time, what interested me about the recipient, and why I was reaching out. I tried more complete briefs, examples in the prompt, and excerpts from my own writing. I again used Tinker to generate those drafts and train another LoRA adapter on an eight-example pilot.
On four development cases I’d already reviewed, I compared drafts from the untuned and tuned models without seeing the model labels. I preferred the tuned drafts every time. On four test cases (e.g., emails and recipients the tuned model has never seen), I preferred the base model twice, tied once, and rejected both drafts once. Those results sent me back to the briefs to understand what was going wrong.
Labeling whose work was whose
The richer briefs gave the model more material to work with, but I started seeing another problem: it would take something I’d written about someone else’s research and describe it as my own work.
I went back to untuned Qwen on Tinker and changed how I organized the evidence. I labeled my own experience, the recipient’s work, and my summaries of other people’s research. I also removed claims I couldn’t verify. I wanted to see whether clearer attribution would help the model use the context accurately.
In a draft to a FlashAttention coauthor, untuned Qwen had said inference economics was critical to my work at Shippo. I worked on pricing and packaging there, but knew nothing about inference economics. With the revised brief, the same untuned model and instructions produced a draft that connected the paper’s point about actual runtime to this set of LoRA experiments. In this case, the untuned model had a much better result.
I compared four pairs of drafts from untuned Qwen with the original and revised briefs. Looking at them together, I only preferred emails that used the revised briefs twice. There was clearly more opportunity to make the model better.
I again went to Tinker to train a second adapter on the same eight emails, with updated instructions and clearer source labels in four training briefs. The results of this model were much better, but every untuned draft needed edits or was a pass. So despite being a slightly better starting point, the emails still required significant input from me before sending.
Turning it into a tool
I merged the second adapter into Qwen and built a Codex skill to draft and revise emails on my Mac. I could give it a recipient brief, my background, and a question, then review what came back–running inference locally with Codex as the driver!
I then compared the eight-example fine-tune with a general-purpose LLM on ten researchers, giving both the same evidence and question. LLM judges reviewed the shuffled drafts and preferred the general-purpose model in every pair. The tuned model preserved the exact same call-to-action for the recipient in 8/10 drafts. Again, there was still room for improvement.
What Astra changed
Until now, I’d picked the examples myself and judged what came back after each run. Then I saw Rikiya Takehi’s post on X about giving Astra access to Qwen’s weights. Astra edited the weights directly to improve the model’s performance on math problems. I wanted to see what would happen if I handed Astra the fine-tuning problem itself. I told it to improve the model and show me the results.
Astra’s first training run on Tinker added seven historical emails in the old format. On the same ten development briefs, the model kept my exact question 5 times, down from 8. I’d been looking for more good emails, and now I had to look at what each example was actually teaching the model.
For the next run on Tinker, Astra kept the same 15 cases and rewrote the examples to preserve my exact question and sign off with just my name. With those examples and an updated prompt, the new tune went from keeping the question 5 times out of 10 to all 10. Would it hold up on new recipients? I ran the new weights locally on 30 fresh briefs and compared the drafts with the eight-example tune and untuned Qwen.
I was surprised by the improvements solely from asking Astra to "make this better". I’d been deciding what the training data and test sets should look like, then checking each run myself. Astra could work through those decisions and keep iterating without me babysitting every step. I could ask for an improvement, then come back to an actually better result.
Would I send it?
One of the fresh briefs for the Astra-tuned model was for a coauthor of Lost in the Middle, a paper about how the location of information in a prompt can affect whether a model uses it. I gave both versions of Qwen the same evidence and question. Here are the complete drafts.
Untuned Qwen
Subject: Positional perturbation for evidence testing
Your work on Lost in the Middle demonstrates that performance in multi-document question answering depends on information position.
As a Stanford GSB MBA student who evaluated a local fine-tuned cold-email drafting workflow, I am interested in inference economics. Which positional perturbation would you include in a small test of whether a drafting model uses its evidence?
David Lee
15-example fine-tune
Subject: Evidence position in cold emails
Hi Nelson,
Your work on Lost in the Middle shows that where relevant information appears can affect performance.
I’ve been exploring how context and memory affect agent behavior, and I’m curious whether evidence position matters when a model is drafting cold emails.
Which positional perturbation would you include in a small test of whether a drafting model uses its evidence?
David Lee
The fine-tuned version gives the researcher a reason I’m asking about this paper and tells them what I’m trying to figure out in my own work. Across the 30 fresh briefs, an LLM judge ranked three drafts with the model names hidden. Here’s how the versions compared.
| Model | Ranked first (out of 30) | Kept exact question (out of 30) |
|---|---|---|
| 15-example fine-tune | 25 | 30 |
| 8-example fine-tune | 3 | 19 |
| Untuned Qwen | 2 | 27 |
Even with those gains, 5 of the new tune’s drafts contained unsupported claims or name errors. Untuned Qwen had only one such draft. In another email, the new tune called LoRA coauthor Yelong Shen “Elong.”
So yes, I’d send an email an LLM helped write. The better brief got me to a draft I’d send to the FlashAttention researcher. The tune gave me a better reason to reach out to the coauthor of Lost in the Middle. Before sending, I still need to be "in the loop".
So what?
Sample efficiency matters even when you’re training on a handful of emails. 8 examples gave me drafts I liked on familiar cases. Adding 7 more initially made the model worse at following my ask. Rewriting what those same 15 examples demonstrated helped. I’d spend the next round writing more emails myself and making sure each one teaches the choice I want the model to make.
Getting more improvement from limited data matters across post-training, whether through supervised fine-tuning like mine or reinforcement learning. In RL, Charles Arnal and coauthors found that reusing generated attempts through a well-designed replay buffer reduced generation compute while maintaining or improving performance in their experiments. I expect better sample efficiency across both approaches to make specialized training more accessible, alongside tools like Tinker that already make it easy to try.
Training and generating the early drafts were surprisingly easy. I signed into only Tinker once, hooked it up from my terminal, and used it for the training runs and inference while I worked on the briefs and examples. Once I exported the weights, I could generate drafts on my Mac too and upload the model to Hugging Face. The two 15-example training runs used about 17 cents in estimated token charges. Most of my effort went into deciding what a good email looked like, finding examples, and reading what came back.
My bet is that post-training becomes critical infrastructure for model providers to own. Base-model leadership keeps changing, and I’ll want to try whichever model is better for the job. The examples I’ve collected, the tests I run, and the process for improving and serving a model keep accumulating. A provider that owns the post-training framework, compute, and serving can become the place I return to for each improvement. The model gets the company in the door. Helping that company build on its own work gives it a reason to stay.