An AI product can be useful and still lose people before they try it. In an OpenAI case study published October 16, Stack Overflow’s VP of Product Alexandra Lato says registrations for Stack Overflow for Agents tripled overnight after the company added a ChatGPT plugin. That is not a measure of customers completing useful work. It is evidence that the first step mattered enough to change how many people took it. If you lead a team introducing AI, start by measuring the distance between a colleague wanting help and reaching a first useful result.
In brief: Choose one real task, let people begin from a place they already work, and count the steps to a reviewable result. Track repeat use separately from signups. A shorter introduction is worth building only if it leads to work people want to do again.
The invisible work before the first task
Stack Overflow for Agents is a knowledge exchange for AI agents and developers. The company announced its ChatGPT plugin in September; the October case study reports its uptake. OpenAI says the plugin let people discover and set up the product from ChatGPT with fewer steps than another connection route.
Lato reports that registrations tripled overnight and says agent creation was up to roughly 10 times its earlier level when she last checked. Those are Stack Overflow’s figures, given in an OpenAI customer story. The account does not provide the starting counts, a measurement period for agent creation, or a comparison group. We cannot tell from it how many agents went on to help anyone finish a task.
Still, the direction is instructive. Adding a product to a familiar place reduced the work required before someone could see what it did. That is different from making the AI smarter. For a team leader, the same obstacle might be an account approval, a separate website, a confusing connection screen, or instructions that live in a document nobody opens.
A tool that takes twenty minutes to set up for a five-minute job has to be unusually good to earn a second try. Most tools are not.
Count the handoffs, not just the accounts
Suppose a service team wants an assistant to prepare a draft answer from its approved policy documents. The team has the documents. The vendor’s demo works. But an employee must request access, switch applications, connect a document folder, find the right assistant, and learn a new way to ask for the draft. The organization records ten new accounts as adoption. A week later, the team is back to searching the folder manually.
That is an illustrative scenario, not a Stack Overflow customer story. It separates two questions: Can people start? Does the new way of working survive an ordinary Tuesday?
Watch someone outside the project team attempt the task without coaching. Note each approval, screen change, and request for help. Time the path to a reviewable draft. A routine first run should not require the project owner in the room.
A week later, did that person return? Did the draft use the right documents? Did the result reach the team’s normal handoff, or stay inside the AI product? Signups answer none of these questions.
Stack Overflow’s case makes the distinction unusually visible: three times as many registrations is a result at the entrance, not at the finish line. Leaders should celebrate removing the entrance barrier without calling it business impact yet.
Make the first job easy, not permissionless
Lowering friction does not mean giving an assistant broad access to company records. A simple introduction can begin with a bounded job: draft from a small set of approved materials, show the passages used, and leave sending or changing records to a person. The employee should know what the assistant can see and where the draft goes after review.
Ask the service team which part is worth handing off. It might be finding the correct policy passage, not writing the entire response. Let the assistant find it and draft. The representative decides whether it fits the customer’s situation.
The business owner of the process should own the trial. Someone responsible for access can set the boundary, but they cannot decide whether the answer helped the service team. That decision belongs with the people doing and supervising the work. If the product’s setup needs a specialist every time a new person joins, the introduction is still unfinished.
This is also why a familiar entry point can be powerful. People already know how to arrive. But familiarity should not disguise a new obligation: the assistant’s output must still be checked against the records and the decision a person is authorized to make. Ease of entry and quality of work are separate design problems.
The next measurement
Stack Overflow’s reported result is a good reason to inspect your own first-use path, not a reason to install its plugin or assume that plugins drive adoption everywhere. Take one task your team already repeats. Ask a colleague who was not involved in the pilot to start it from their normal working day. Observe where they get stuck. Remove one needless step. Then check whether they choose the new route again the next time the task arrives.
If the second use does not happen, ask what failed after the signup. Perhaps the answer needed too much correction. Perhaps the draft landed outside the team’s system. Perhaps no one knew who could approve it. Those findings tell you what to fix next. Another account-creation campaign will not.
The simplest AI workflow is not the one with the fewest clicks. It is the one a person can enter, understand, and return to when the real work comes back.
Read more about human-AI collaboration in practice.
Research and draft: Mai. Editorial direction: John Lipe.