AI Tips Ladder · Full Guides
The complete walkthroughs from that post
SAT · AUG 01 · ED 577
BEGINNER
PROMPTING
01 · Let OpenAI Rewrite Your Worst Prompt
A long production prompt accumulates contradictions, dead instructions, and an unstated output format — the usual cause of answers that are right 8 times out of 10 and malformed the other 2. The Prompt Optimizer rewrites it against GPT-5's prompting conventions in about a minute and shows you what it changed, so you get a structural audit for free. Use it on any prompt over ~30 lines that has been edited by more than one person, or whenever output shape is inconsistent run-to-run.
- 01Copy your worst-behaving production prompt (system/developer message plus any few-shot examples) into a scratch file, and alongside it save 3-5 real inputs where the current prompt already fails or wobbles — you need these to judge the rewrite.
- 02Sign in at platform.openai.com and open https://platform.openai.com/chat/edit?optimize=true (the Playground prompt editor with the optimizer panel open). If that URL redirects, go to platform.openai.com/chat and use the Optimize action on the prompt editor.
- 03Paste the prompt into the editor and run Optimize. If the panel offers a field for task context or evaluation criteria, state what the prompt must produce and what 'wrong' looks like — the optimizer only sees the text, not your app.
- 04Read the diff line by line. Sort each change into: (a) removes a real contradiction, (b) adds a missing output spec, (c) tightens wording, (d) drops a constraint the optimizer had no way to know mattered. Reject every (d) — those are your business rules.
- 05Save the accepted rewrite as a new prompt version rather than overwriting: in the Playground use Save on the prompt so it gets a version id you can reference from the API and roll back to.
- 06Open the original in a second Playground tab, set both tabs to the same model, same reasoning effort, and temperature 0, then run the same real input through each and diff the two outputs.
- 07Repeat step 6 across all 3-5 saved inputs plus one input the old prompt handled correctly, to confirm the rewrite did not trade a new failure for the fixed one.
- 08Ship the new version behind whatever your rollout mechanism is, and keep the old prompt id for one week so a regression is a config revert, not a re-edit.
You are auditing a prompt rewrite. I will give you OLD and NEW versions of a system prompt.
For EVERY substantive difference, output one row:
| # | change (one line) | type | keep? | why |
type is exactly one of:
CONTRADICTION_FIX - old text told the model two incompatible things
OUTPUT_SPEC - adds/clarifies format, schema, length, or refusal behaviour
TIGHTENING - same meaning, fewer or clearer words
CONSTRAINT_DROPPED - a rule, edge case, example, or domain fact present in OLD and absent in NEW
NEW_ASSUMPTION - NEW asserts something OLD never said
Rules:
- Do not summarise. One row per change, in document order.
- Default keep? to NO for CONSTRAINT_DROPPED and NEW_ASSUMPTION.
- Ignore pure whitespace and heading-style changes.
- After the table, list every behaviour the OLD prompt guaranteed that the NEW one no longer does.
OLD:
<<<
[paste original prompt]
>>>
NEW:
<<<
[paste optimized prompt]
>>>
Watch out: The optimizer sees only the prompt text — not your tools, retrieved context, or downstream parser. It will happily delete a weird-looking constraint that exists because of a customer escalation, and add an output format that breaks whatever regex or JSON parser consumes the response. · It rewrites toward GPT-5 conventions (explicit instruction hierarchy, less chain-of-thought coaxing, reasoning-effort assumptions). Pasting the result into a different model or an older one unchanged can make things worse, not better. · 'It reads better' is not evidence. A rewrite that fixes your three failure cases and quietly breaks a case that used to work is a net loss — always re-run at least one known-good input, at temperature 0, before shipping.
INTERMEDIATE
PROMPTING
02 · Make Vision Models Quote Their Sources
Extraction errors from scanned documents are usually invisible: a wrong invoice total looks exactly like a right one. Forcing every field to carry the verbatim source text plus a page number turns silent hallucinations into a mechanical check — you grep the quote against the document's text layer, and anything that isn't found is wrong. Use it whenever extracted numbers feed accounting, payments, or any downstream system where a plausible-but-invented value is expensive.
- 01Render each page to its own full-resolution image, one file per page — e.g. `pdftoppm -r 300 -png invoice.pdf page` produces page-1.png, page-2.png. Keep the long edge around 1500-2200px; do not feed a shrunken preview or a single stitched image.
- 02Attach the page images in order (Playground: platform.openai.com/playground, attach via the paperclip/image button on the user message; API: one image content block per page) and set the image detail parameter to "high" so small print is actually tokenized.
- 03Paste the prompt below as the system/instructions message, replacing the field list with your own field names and dropping line_items if your document has none.
- 04Set temperature to 0 and turn on JSON mode / Structured Outputs (Playground: right-hand settings panel → Response format → json_object or json_schema) so you get parseable output instead of prose.
- 05Run it and save the raw response to a file, e.g. out.json — do not hand-edit it before verification.
- 06Verify mechanically: extract the document's text layer with `pdftotext -layout invoice.pdf -` and, for each returned quote, check it appears there — `pdftotext -layout invoice.pdf - | grep -Fq "<quote>" && echo OK || echo MISSING`. Any MISSING quote means that field is fabricated; discard the value.
- 07Spot-check three quotes by eye against the rendered page — pick the highest-risk fields (grand total, due date, tax ID) and confirm both the text and the reported page number are right.
- 08Route every field that came back null, confidence "low", or MISSING to a human review queue instead of letting it default to zero or empty string.
You are a document data extractor. Extract ONLY what is literally printed in the attached page images.
Return a single JSON object. For EVERY field, emit an object with exactly these keys:
"value" - the normalized value (string, number) or null
"quote" - the verbatim text you read, copied character-for-character from the page, including currency symbols and punctuation
"page" - the 1-based index of the image the quote appears on
"confidence" - "high" or "low"
Rules:
- If a field is not printed on the page, set value, quote and page to null. NEVER infer, compute, or guess a value. A missing field is a correct answer.
- "quote" must be an exact substring of the text on the page. Do not paraphrase, reformat, expand abbreviations, or silently fix typos inside "quote".
- Do all normalization (date reformatting, stripping currency symbols, parsing numbers) in "value" only.
- If the page shows two plausible candidates, pick the one physically nearest its field label and set confidence to "low".
- Never merge values from two different pages into one field.
- Output JSON only. No prose, no explanation, no markdown fences.
Fields:
invoice_number, invoice_date, due_date, vendor_name, vendor_tax_id,
bill_to_name, currency, subtotal, tax_total, grand_total
Also return "line_items": an array where each element has the same
{value, quote, page, confidence} shape for: description, quantity, unit_price, amount.
Watch out: A quote that exists in the document is not proof the field is right — models copy real text from the wrong region (subtotal pasted as grand_total). The grep check catches invention, not misattribution; that is what the page number and the three-quote eyeball pass are for. · Exact string matching fails on legitimately correct quotes because of OCR whitespace, ligatures, and non-breaking spaces. Normalize whitespace on both sides before comparing, and for scanned PDFs with no text layer at all (`pdftotext` returns nothing) fall back to visual verification. · Thumbnails and low detail settings are the main driver of guessed values — the model can't read the small print, so it produces something shaped like an invoice number. Also send one image per page: with a stitched or cropped composite, the page field becomes meaningless.
ADVANCED
AGENTS
03 · Refactor Legacy Code One File At A Time
One prompt over a whole legacy repo produces a 40-file diff that nobody can review, and a single bad conversion poisons the entire change. Scoping the agent to one file per run in its own git worktree makes each patch small enough to actually read, lets the test suite pass/fail verdict decide what you keep, and lets failures be retried without blocking the passes. Use it for mechanical, repo-wide migrations — callbacks to async/await, PropTypes to TypeScript, moment to date-fns, deprecated API swaps — where the target pattern is identical across files.
- 01Commit or stash everything, then run your real test command on the clean checkout and record the result — a suite that is already red or flaky makes the whole gate meaningless.
- 02Enumerate the target files into a plain list, one path per line, and save it as files.txt: e.g. `grep -rl 'function(err,' src --include='*.js' > files.txt`, then open it and delete anything you do not want touched (vendored code, generated files, tests).
- 03Confirm the agent runs headless: `claude -p 'reply with OK' ` from the repo root. If `claude` is not found, install it with `npm i -g @anthropic-ai/claude-code` and authenticate by running `claude` once interactively.
- 04Copy the script below into refactor.sh, set TEST_CMD to your real command (`npm test`, `pytest -q`, `go test ./...`), edit the migration sentence in the prompt to describe your actual change, then `chmod +x refactor.sh && ./refactor.sh`.
- 05Watch the PASS/FAIL lines. Each file gets its own worktree at ../wt-<slug> on branch refactor/<slug>; failures are appended to .refactor/failed.txt and their worktrees are left in place for inspection.
- 06Review each passing branch before merging: `git diff --name-only main...refactor/<slug>` must list only the one target file (no test edits), then `git diff main...refactor/<slug>` to read the change, then `git merge --no-ff refactor/<slug>` and re-run TEST_CMD on the merged trunk.
- 07Requeue the failures: `cp .refactor/failed.txt files.txt`, re-run the script after the passing merges have landed so the retry starts from updated code, and hand-fix anything that fails twice.
- 08Clean up when done: `git worktree remove ../wt-<slug>` for each finished worktree (add `--force` if the run left dirty files), then `git worktree prune` and delete the merged branches.
#!/usr/bin/env bash
# refactor.sh — one file per worktree, merged only if tests pass.
set -uo pipefail
TEST_CMD="npm test" # <-- your real test command
MIGRATION="convert every callback-style function to async/await" # <-- your real migration
BASE="$(git rev-parse --abbrev-ref HEAD)"
mkdir -p .refactor && : > .refactor/failed.txt
while read -r FILE; do
[ -z "$FILE" ] && continue
SLUG="$(echo "$FILE" | tr '/.' '--')"
WT="../wt-$SLUG"
git worktree add -b "refactor/$SLUG" "$WT" "$BASE" || { echo "SKIP $FILE"; continue; }
(
cd "$WT" || exit 1
# deps are untracked, so a fresh worktree has none — reuse the main checkout's:
[ -d ../"$(basename "$OLDPWD")"/node_modules ] && ln -s "$OLDPWD/node_modules" node_modules 2>/dev/null
claude -p "Refactor ONLY the file $FILE: $MIGRATION.
Hard rules:
1. Edit no file other than $FILE. Do not edit tests, config, or lockfiles.
2. Do not change exported names, function signatures, or observable behaviour.
3. Preserve error semantics exactly: an error that used to reach a callback must still reach the caller by the path the caller already handles.
4. Add no new dependencies.
5. If the file cannot be migrated without breaking rules 1-4, change nothing and explain why.
When the edit is done, run: $TEST_CMD
If it fails, fix $FILE only until it passes. Finish with a one-paragraph summary of what changed and any behaviour you were unsure about." \
--permission-mode acceptEdits \
--allowedTools "Read,Edit,Grep,Glob,Bash"
# the gate: our verdict, not the agent's
$TEST_CMD
)
if [ $? -eq 0 ]; then
echo "PASS $FILE -> branch refactor/$SLUG (worktree $WT)"
else
echo "$FILE" >> .refactor/failed.txt
echo "FAIL $FILE -> left at $WT for inspection"
fi
done < files.txt
echo "---"; echo "failed files:"; cat .refactor/failed.txt
Watch out: A fresh worktree only contains git-tracked files — node_modules, .env, venv, and build caches are missing, so the test command fails instantly for reasons unrelated to the refactor. Symlink or reinstall them inside each worktree before trusting a FAIL. · An agent told to make tests pass may edit the tests. Always check `git diff --name-only` shows exactly one changed file before merging; treat any test change as an automatic reject. · Green tests on legacy code prove much less than on well-covered code — double-invoked callbacks, swallowed rejections, and changed error timing often slip through. Read every diff yourself, and merge one branch at a time so a regression is bisectable. · All worktrees branch from the same base, so the Nth merge can conflict with or silently duplicate work from the first. Re-run the suite on the trunk after each merge, and re-run failures only after the passing merges have landed.
PRO
TOOLS
04 · Run A 20B Model On Your Own GPU
A local gpt-oss-20b gives you an unmetered model for high-volume, low-stakes text work — summarising support inboxes, tagging tickets, drafting first-pass replies — where a per-token API bill would dominate the cost and where the text itself shouldn't leave your network. It is worth it once you have thousands of items to process or a compliance reason to keep data on-premises; for a handful of one-off calls a hosted API is still cheaper than the setup time. The 20B checkpoint ships MXFP4-quantised at roughly 13GB on disk, so it is the largest open model that realistically fits a single consumer card.
- 01Check the card and its compute capability: `nvidia-smi --query-gpu=name,memory.total,compute_cap --format=csv`. You need 16GB+ VRAM. Note the compute capability — 9.0 (H100) or 10.0 (B200) gets native MXFP4 at ~13GB; 8.9 (RTX 4090) and below silently dequantise to bfloat16 and need ~40GB+.
- 02Create an isolated environment: `python3 -m venv gptoss && source gptoss/bin/activate` (Windows: `gptoss\Scripts\activate`).
- 03Install the stack: `pip install --upgrade pip` then `pip install --upgrade torch transformers accelerate`. On Linux + NVIDIA, install torch from the CUDA index that matches your driver — see pytorch.org/get-started/locally for the exact `--index-url`.
- 04Only on Hopper/Blackwell (compute_cap 9.0+), add the MXFP4 kernels so the model stays at ~13GB: `pip install --upgrade 'triton>=3.4' kernels`. Skip this on other hardware; it will not help.
- 05Pull the weights before you step away — roughly 13GB: `hf download openai/gpt-oss-20b`. Set `HF_HOME=/path/with/space` first if your home partition is small. Resume by re-running the same command.
- 06Save the snippet below as `summarize.py`, put one email per line in `emails.txt`, and run `python summarize.py`. First run loads weights from the local cache in 1-3 minutes.
- 07Watch memory in a second terminal with `nvidia-smi -l 5` (or Activity Monitor on macOS) during the first generation. If it climbs past your VRAM and stalls, the MXFP4 path is not active — see pitfalls.
- 08Scale up by raising `batch_size` in the pipeline call until VRAM is ~85% used, and cap `max_new_tokens` to the shortest length that still produces a usable summary. Both change throughput far more than any other knob.
# summarize.py - bulk-summarize one email per line of emails.txt
from transformers import pipeline
pipe = pipeline(
"text-generation",
model="openai/gpt-oss-20b",
dtype="auto", # older transformers (<4.56): use torch_dtype="auto"
device_map="auto",
)
with open("emails.txt") as f:
emails = [line.strip() for line in f if line.strip()]
for i, email in enumerate(emails, 1):
messages = [
{"role": "system", "content": "Summarize the customer email in one sentence, then output SENTIMENT: positive|neutral|negative. Reasoning: low"},
{"role": "user", "content": email},
]
out = pipe(messages, max_new_tokens=160, do_sample=False)
print(f"--- {i} ---")
print(out[0]["generated_text"][-1]["content"])
Watch out: The 16GB figure only holds where the MXFP4 kernels actually run (Hopper/Blackwell, compute capability 9.0+). On an RTX 4090, an A100, or Apple Silicon the weights are dequantised to bfloat16 and the model wants ~40GB+ — it will either OOM or spill to CPU/disk and crawl. Verify with nvidia-smi during the first generation rather than trusting the headline number. · gpt-oss uses the harmony response format and emits a chain-of-thought channel before its answer. Pass a list of message dicts (as in the snippet) so the chat template is applied — a raw prompt string produces malformed, rambling output. Also strip the reasoning channel before showing anything to a customer. · `dtype` replaced `torch_dtype` in transformers 4.56; on an older install the snippet raises a TypeError. Either upgrade or swap the keyword. Separately, the first `pipeline(...)` call downloads ~13GB if step 5 was skipped — do not run it on a metered connection expecting a fast start.
🛠️ We build the AI systems & automations behind tips like these.
Follow @conon.ai for the daily brief — news + tips, every day.