LaunchDarkly previews generative AI-powered product experimentation feature

LaunchDarkly, a feature management platform that helps developers and software engineers manage and launch products, released a new experimentation capability today in preview powered by generative AI that will allow teams to quickly generate product experiments at scale.

At the core of building and releasing products is understanding, measuring and testing small changes at scale then iterating on those tests. Oftentimes these tests are developed from a hypothesis or a goal based on a milestone or a plan and then built from data and metrics by a data scientist who can turn them into an actionable set of features that can be tested as variations.

For many teams, this is a complex process that can be problematic at the best of times and it’s difficult to yield good results. To ease this process, LaunchDarkly has combined its already existing Product Experimentation feature with generative AI capabilities powered by Amazon Bedrock, a fully managed service from Amazon Web Services that makes it easy to access large language models such as OpenAI’s GPT-4 and others.

The hallmark of generative AI is its capability of understanding conversational text prompts similar to a person can and then automating processes such as building and recommending product experiment variations.

Speaking to SiliconANGLE in an interview, Cody De Arkland, director of developer relations, and Robert Neal, experimentation product lead at LaunchDarkly, explained that one of the leading reasons that the company turned to generative AI for experimentation is that it’s a hard problem facing the industry.

“What we decided to do is figure out how we could use generative AI with experimentation to solve some of these problems,” said Neal. “One big problem in experimentation is when you’re trying to run an experiment and what you’re trying to do is figure out what is the best variation of this product that you’re building.”

Neal likened it to going out to choose a new Indian restaurant to eat at. A person doesn’t just want to go out and try one new restaurant and if that’s better than the last one stick with that one forever. They want to try out a bunch of them and choose the best.

It’s similar to software development experimentation, except the complexity is broader. Developers want to test as many variations as possible, but those variations need to hit home on the metrics and measures that the developer wants to focus on.

This is where the generative AI comes into play. It takes their hypothesis – written out by the users – and their current features and historical data — metrics, information about the product, feature flags and other internal elements — into account to offer them a set of experimental variation recommendations that they can choose from.

“We’re simplifying a lot of this thing where maybe it’s a product manager setting up the experiment, but it was an engineer who set up a feature flag,” said Neal. “Now that product manager doesn’t have to go talk to the engineer because the LLM is doing a lot of that work.”

That opens up a lot of opportunities for members of the team who would not have gotten access to experimentation and testing to offer more input than before. That’s especially good for less technical members and developers who would need to visit engineers or data scientists to build out variations.

“Or, for example, if Robert and I were building software and I was the one implementing this, I’d be calling Robert and saying, ‘Hey, which tests should I test here?’” De Arkland said. “We’d have a separate meeting to figure out the different variations we’d build manually. A week later we’d come back and decide that wasn’t enough, have a catch-up meeting to figure out another ten.”

Using the same generative AI system, developers could take a burden off their team and have the LLM simply offer them variations on their own existing sample set whenever they wanted, letting them build out even more recommendations. De Arkland explained that this would help reduce the number of “return to brainstorming” meetings that would take away from actual development time and let developers just do their job by having the AI just do that for them so that they can get on with their lives.

“What we want to do here is speed up the way engineering teams are able to iterate on changes and round out some of the complexity and make it a lot faster to get to a place where we’re getting valid data answers,” said De Arkland.

Product experimentation with generative AI capability is rolling out now in early access preview, more information is available on LaunchDarkly’s website for users interested in the feature.

Image: Pixabay