Thumbtack · AI content design · Public summary

AI call summaries

I led the content design and prompt engineering for AI recaps of phone calls between customers and pros. This is the short, public version.

Role
Content design and prompt lead
Partners
Applied science, engineering, product design, product, Legal
Status
🚀 Launched to all users

The problem

After a phone call, the customer and the pro each get a short AI recap in their messages. Both recaps have to tell the same story. If the pro’s says Thursday and the customer’s says Friday, someone shows up on the wrong day.

People also trust what’s in the thread. A price in a message preview reads like a promise, and a personal detail repeated by AI can feel like a breach.

The decision: refine after launch, or pause and fix a privacy risk first

We were still testing, and the plan was to keep refining the prompt after launch, which is common with AI features. The day before launch, I called for one more round of testing. My applied scientist and I built a way to check whether the AI would repeat sensitive personal details from a call. In a few test samples, it did.

How we kept quality high

What happened

The final round of testing came back clean, and the feature launched to all users. Now, 90%+ of the time, the customer’s and pro’s recaps tell the same story. AI doesn’t always get it right, so we built in a feedback system and keep testing.

“She helped create clarity in an otherwise complex problem space and enabled engineering, Applied Science, and the broader team to move faster and with greater confidence.”Engineering partner, Thumbtack

What I delivered

Want the full case study, with screens and a behind-the-scenes video? It’s password-protected. See the full version, or email me for the password.