Marketing Engineering with AI · Coming soon
Stop reading every AI output by eye and hoping you catch the bad one
For marketers running an AI agent or prompt on customer-facing work who check the output by reading it. You build a labelled set of your real examples, checks for facts, tone, policy and format that run on every output, and a test that fails loudly when a prompt or model change makes the output worse.
The course: Evals for marketing agents: knowing the output is good
$25 $49 early-bird price
This course is being built now. Join the early-access list to reserve the early-bird price and shape what it covers.
Joining the list is free. You only pay if you buy at launch, at the $25 early-bird price.
If this sounds like your week
- You read outputs before they go out, and you know you will miss one when volume doubles.
- A prompt edit that fixed one problem quietly broke another, and nothing told you.
- The model changed underneath you, and you cannot say whether the output got better or worse.
- Your boss asks how you know the AI is good enough, and the honest answer is that it looks fine.
What you do now, and where it stops working
Today you sample outputs, read them, and patch the prompt when something looks off. That holds while one person reads everything. It breaks when volume grows, when a second person edits the prompt, or when the vendor updates the model and nothing flags the change.
What changes after the course
- You have a set of real examples, each labelled pass or fail with a reason, that your agent is graded against.
- Checks for facts, tone, banned claims and format run on every output, so your reading time goes to the cases they flag.
- You can name the ways your agent fails most often, ranked by how often each one happens.
- When you change a prompt or a model, a test run shows whether quality dropped before the change ships.
How it works: Look, label, check, gate
- 1Look: read a batch of real outputs and write down every failure in plain words. This error analysis decides what is worth checking.
- 2Label: turn those notes into a set of examples marked pass or fail, with the reason, in a spreadsheet template.
- 3Check: write one grader per failure type — code for format and banned phrases, a model-graded judge for tone — and test each grader against your labels so you know it fails when it should.
- 4Gate: run the set on every prompt or model change, so a drop in quality blocks the change instead of reaching a customer.
What you get
- Eval harness code with example graders
- Labelling template and error-analysis worksheet
- Example eval set for a review-reply agent
- Video walkthrough of a full error analysis
This is for you if
- You run AI on customer-facing marketing work and approve it by reading it.
- You are about to change a prompt, a model or a vendor and want to know if the output gets worse.
- You need to show a boss or client how you know the output is good.
Skip it if
- You want software to buy. This course teaches a method and gives you code; it sells no tool.
- You train or fine-tune models. This course checks the output of agents you already run.
- You have no AI output in use or planned yet; build the agent first.
Before you join
- Do I need to code?
- Not for the first half. Reading outputs, labelling them and the error analysis use a spreadsheet template. The automated checks use the included harness code; if you can follow and edit a script, you can adapt it. If you write no code at all, you still leave with a labelled set and a ranked failure list a developer can turn into checks.
- Is quality not too subjective to test?
- Tone feels like taste until you write down why you rejected your last twenty outputs. Those reasons become pass-or-fail criteria. Research on model-graded evaluation finds people define their criteria by grading real outputs, which is why the course starts with reading, not with a rubric.
- I do not have any labelled examples.
- That is the normal starting point. You build the set from your own outputs during the course, and an example set for a review-reply agent is included to practise on first.
- Who is teaching this, and is it an engineering course in disguise?
- Jay West leads DevOps for the marketing organisation of a multi-location specialty retailer, where he builds AI agent systems and the checks around them. He built an automated check suite for AI-generated web pages and tested that the checks themselves fail when they should. The examples are marketing work: review replies, brand voice, claims.
- The course is not out yet. What am I signing up for?
- An early-access list. You get updates as modules are built, and at launch you can buy at the $25 early-bird price instead of the $49 list price, or not buy at all.
Rather have it built for you? Describe the project and the same work can be done for your team.