Michael Taylor · AI evals
I build evals and optimization loops for LLM products — the fixed judges, gold datasets and pairwise comparisons that turn “vibes” into a number you can hill-climb. Co-author of O'Reilly's Prompt Engineering for Generative AI and before that 8,000+ growth experiments across 200+ startups.
Latest
AI spending ranges from $0 to $20,000/month — and the gap is growing. Explore the "permanent underclass theory," why Silicon Valley fears being priced out of AI, and what economics says about whether human labor really becomes worthless.
The Pairwise Gold Triad is an eval pattern for subjective AI tasks. Compare challenger outputs against an incumbent using gold references and a fixed LLM judge—then hill climb with autoresearch.
Discover hard-won lessons from scaling AI persona research to 5,000 simulated consumers. Learn practical tips for managing costs, ensuring statistical accuracy, and avoiding common pitfalls when using LLMs for market research.
AI excels at strategy and execution but fails at creative problem-solving. Explore why humans will remain essential as "NCOs" in the post-AGI economy, bridging the gap between AI planning and AI action through adaptive intelligence and real-world judgment.
AI art is here, and prompt engineering is the job of the future. Take my prompt template and learn how to find all the art styles, quality boosters and magic words to get the generative art you need.
How to (actually) write engaging LinkedIn posts with ChatGPTOne of the things people get super excited about with ChatGPT is it’s ability to write content for you.
What I work on
Evals, prompt optimization, synthetic research and the economics of working with LLMs — from 2022 to today.
When there is no exact-match metric you need a fixed judge, gold references and pairwise comparisons. The Pairwise Gold Triad is the pattern I use to hill-climb on writing style, personas and creative work.
Trial and error gets you a prompt that works once. Test sets, A/B tests and automated optimizers get you one that keeps working when the model underneath changes.
I simulate thousands of consumers to pressure-test ideas before they ship — and document where LLMs are too agreeable to be trusted without calibration.
From the archive
Before AI this was a growth experimentation lab: 42 essays on marketing mix modeling, virality, content and agency economics. All still here.
Browse the whole blogYou need to choose where to allocate your marketing budget, but don't know if you can trust the reports you're getting from Facebook and Google... and how do you even measure TV? Econometrics is the key to answering these strategic questions and making data-driven marketing decisions.
How do you know if you have enough data for an MMM? What do you do if you don't? Geo-level models are a great way to increase the certainty in your model, by introducing variance so you get to the truth of channels actually drove performance.
Where does all the money go when you hire an agency? I reveal the economics of how an agency makes money, and what it spends it on. You'll see why agency employees are always underpaid and overworked, and why agency owners always look fraught.
Projects
VEXPOWER Marketing Mix Modeling Training Simulator
I worked with former VP of Growth at Eaze Yousuf Bhaijee and Reforge to define a method for calculating Word of Mouth
Active and well-reviewed mentor in the Growth Mentor community.
Kind words
The real value in this course is that Michael is an actual practitioner - he's not just teaching code, but a better way to handle large marketing data sets.
It's rare to have a marketing genius like Michael taking the time to share one of his secret weapons which allowed him to scale a 45+ person world-class agency. Trust me, his course is worth every cent.
The marketing process requires a) smart strategy, b) well thought out tests, then c) collection and analysis of data. Part c) quickly becomes very time-consuming - and there’s only so much you can do in Excel. That’s where Python for Marketers comes in - a key part of an advanced marketer’s tool kit!
I help teams put a number on LLM output quality and then improve it — evals, judges, optimization loops, and the honest conversation about what can't be measured yet.
