[24]7.ai · 2017–2018 · Enterprise AI tooling

Safe changes for the people who build AI

[24]7 deployed chatbots for some of the largest brands in the US, UK and Australia. Building and changing those bots took specialists, handoffs and a lot of manual testing. I led design for a suite of tools to let deployment teams, and eventually clients, make changes themselves without breaking what was live.

Role: Design lead, Omni Services & Tools. Concept through buildable prototypes, in close partnership with product management
Team: Two designers (visual and UX), product managers, engineering leads
Users: [24]7 deployment teams, analysts and data scientists; client administrators

The problem

A single enterprise bot touched a dozen disciplines: conversation design, intent and entity modeling, content, integrations, testing and deployment. Each lived in a different tool, owned by a different team. Product's own analysis put quality assurance alone at 20–40% of the labor on a deployment.

The hardest moment was change. Adding one new question to a live bot could quietly lower the accuracy of the language model for every other question, and nobody would know until customers did.

Approach

With product management, we mapped the whole enterprise bot lifecycle and each tool's place in it, then wrote a story map from the deployment team's point of view: set up a client, start a bot from a reference template, edit content, add an intent, test, and push to production.

That map became a storyboard, revised through nine versions with product and engineering, and then interaction and visual specs engineers could build from.

Key ideas

One home for the whole bot

Intents, models, content cards, knowledge base and code repositories in one overview, with a live preview of the bot beside them. Every item shows when it last changed.

Tool Suite bot overview for a client bot, showing intents, models, active cards, knowledge base and repositories, with a live chat preview — Bot overview. The development version is separate from what's live, with a count of pending changes and a single path to QA.

Development, live and archived

Every bot has a working version and a live version. Changes accumulate in development, are counted, go to QA as a set, and can be reset to production at any point. Earlier versions stay archived. It's version control for people who don't use version control.

See the impact before you commit

Before a new intent is added, the tool retrains the model in a sandbox and shows the effect on accuracy, with the affected intents one click away. The person making the change decides with the consequence in front of them.

Dialog: adding this intent will impact model accuracy by -3%, before 81%, after 78%, with links to view impacted intents, cancel or add intents — Change impact preview. The model is retrained on a copy, so production is untouched until someone confirms.

Live performance, close to the edit

The live view shows how each intent performs, so the next change starts from evidence.

Live bot view with usage and performance metrics per intent — Live bot view with per-intent performance.

Fixing what already existed

Alongside the new suite, we audited the existing modeling workbench used by analysts and data scientists, screen by screen, with specific recommendations for navigation, discoverability and consistency.

Outcome

The work set the design direction and specs for [24]7's self-serve bot tooling. Capabilities very close to these, an enterprise conversation builder and an AI model management environment for analysts and data scientists, later launched as part of [24]7 Engagement Cloud in 2020.

Looking back

The idea I'd keep is the change impact preview: show people the consequence of a change at the moment they make it, and make production safe by default. It applies anywhere operators change configurations that others depend on.

What I'd do differently: instrument the tools from day one, and measure time to change and errors that reach production, so the case for the design rests on numbers rather than storyboards.