An AI product can be useful and still lose people before they try it. In an OpenAI case study published October 16, Stack Overflow’s VP of Product Alexandra Lato says registrations for Stack Overflow for Agents tripled overnight after the company added a ChatGPT plugin. That reported increase measures entry into the product; whether those customers completed useful work remains unestablished. If you lead a team introducing AI, start by measuring the distance between a colleague wanting help and reaching a first useful result.
In brief: Choose one real task, let people begin from a place they already work, and count the steps to a reviewable result. Track repeat use separately from signups. A shorter introduction is worth building only if it leads to work people want to do again.
The invisible work before the first task
Stack Overflow for Agents is a knowledge exchange for AI agents and developers. The company announced its ChatGPT plugin in September; the October case study reports its uptake. OpenAI says the plugin let people discover and set up the product from ChatGPT with fewer steps than another connection route.
Lato reports that registrations tripled overnight and says agent creation was up to roughly 10 times its earlier level when she last checked. Those are Stack Overflow’s figures, given in an OpenAI customer story. The account does not provide the starting counts, a measurement period for agent creation, or a comparison group. We cannot tell from it how many agents went on to help anyone finish a task.
Still, the direction is instructive. Adding a product to a familiar place reduced the work required before someone could see what it did. That is different from making the AI smarter. For a team leader, the same obstacle might be an account approval, a separate website, a confusing connection screen, or instructions that live in a document nobody opens.
A tool that takes twenty minutes to set up for a five-minute job has a steep barrier to earning a second try.
Count handoffs alongside accounts
Suppose a service team wants an assistant to prepare a draft answer from its approved policy documents. The team has the documents. The vendor’s demo works. But an employee must request access, switch applications, connect a document folder, find the right assistant, and learn a new way to ask for the draft. The organization records ten new accounts as adoption. A week later, the team is back to searching the folder manually.
This example is hypothetical. It separates two questions: Can people start? Does the new way of working survive an ordinary Tuesday?
Watch someone outside the project team attempt the task without coaching. Note each approval, screen change, and request for help. Time the path to a reviewable draft. Design a routine first run that a colleague can complete independently of the project owner.
A week later, did that person return? Did the draft use the right documents? Did the result reach the team’s normal handoff, or stay inside the AI product? Track those outcomes separately from signups.
Stack Overflow’s case makes the measurement boundary unusually visible: a threefold increase in registrations measures entry into the product. Leaders can celebrate removing that barrier while continuing to measure completed work and business impact.
Make the first job easy within clear permissions
Lower friction within a bounded job: draft from a small set of approved materials, show the passages used, and leave sending or changing records to a person. The employee should know what the assistant can see and where the draft goes after review.
Ask the service team which part is worth handing off. Finding the correct policy passage might be a useful first assignment. Let the assistant find it and draft. The representative decides whether it fits the customer’s situation.
The business owner of the process should own the trial. Someone responsible for access can set the boundary, but they cannot decide whether the answer helped the service team. That decision belongs with the people doing and supervising the work. If the product’s setup needs a specialist every time a new person joins, the introduction is still unfinished.
This is also why a familiar entry point can be powerful. People already know how to arrive. Keep the review obligation visible there: check the assistant’s output against the records and the decision a person is authorized to make. Ease of entry and quality of work are separate design problems.
The next measurement
Use Stack Overflow’s reported result as a reason to inspect your own first-use path. Its case alone cannot establish that plugins drive adoption everywhere. Take one task your team already repeats. Ask a colleague who was not involved in the pilot to start it from their normal working day. Observe where they get stuck. Remove one needless step. Then check whether they choose the new route again the next time the task arrives.
If the second use does not happen, ask what failed after the signup. Perhaps the answer needed too much correction. Perhaps the draft landed outside the team’s system. Perhaps no one knew who could approve it. Use those findings to fix the workflow before running another account-creation campaign.
Aim for an AI workflow a person can enter, understand, and return to when the real work comes back. Count that return alongside the clicks removed.
Read more about human-AI collaboration in practice.
Research and draft: Mai. Editorial direction: John Lipe.