Highlights:
- Comet said that data scientists working on artificial intelligence for natural language processing no longer spend that much time training their models.
- Prompt Playground is one of the new tools that enables developers to iterate more quickly with various templates and comprehend how prompts affect various scenarios.
Comet ML Inc., a European machine learning operations startup, is adapting its MLOps platform to deal with large language models such as those that enable ChatGPT.
The startup announced adding many “cutting-edge” LLM operations features to its platform to assist development teams in expediting engineering, managing LLM workflows, and improving overall efficiency.
Comet, a startup founded in 2017, positions itself as accomplishing machine learning and artificial intelligence what GitHub accomplished for programming. Data scientists and engineers may automatically track their datasets, code modifications, experimental history, and production models using the company’s platform. Comet says this results in efficiency, transparency, and reproducibility due to its platform.
Comet said that data scientists working on artificial intelligence for natural language processing no longer spend that much time training their models. Instead, they spend much more time developing the appropriate instructions to address newer, more challenging challenges. Existing MLOps systems don’t have the tools necessary to monitor and analyze the performance of these prompts well enough, which is an issue for data scientists.
Gideon Mendels, Chief Executive of Comet, reported, “Since the release of ChatGPT, interest in generative AI has surged, leading to increased awareness of the inner workings of large language models. A crucial factor in the success of training such models is prompt engineering, which involves the careful crafting of effective prompts by data scientists to ensure that the model is trained on high-quality data that accurately reflects the underlying task.”
According to Mendels, prompt engineering is a method of natural language processing used to develop and perfect prompts necessary to elicit accurate responses from models. They are required to prevent “hallucinations,” which occur when AI creates responses.
The CEO stated, “As prompt engineering becomes increasingly complex, the need for robust MLOps practices becomes critical, and that is where Comet steps in. The new features built by Comet help streamline the machine learning lifecycle and ensure effective data management, respectively, resulting in more efficient and reliable AI solutions.”
Vice President and Principal Analyst at Constellation Research Inc. Andy Thurai said that because LLMs are still in the early stages of research, the majority of MLOps systems do not offer any tools for controlling workflows in that area. This is because LLM engineering entails modifying prompts for pre-trained models rather than training new models.
“The challenge is that, because LLMs are so big, the prompts need to be fine-tuned to get proper results. As a result, a huge market for prompt engineering has evolved, which involves experimenting and improving prompts that are inputted to LLMs. The inputs, outputs and the efficiency of these prompts need to be tracked for future analysis of why a certain prompt was chosen over others,” Thurai added.
Comet claimed that its new LLMOps tools are made to perform two tasks. One benefit is that they will speed up iteration for data scientists by giving them access to a playground for quick tuning integrated with experiment management. Additionally, they offer debugging capabilities that allow prompt chain visualization to trace prompt experimentation and decision-making.
Mendels said, “They address the problem of prompt engineering and chaining by providing users with the ability to leverage the latest advancements in prompt management and query models, helping teams to iterate quicker, identify performance bottlenecks, and visualize the internal state of the prompt chains.”
A prompt Playground is a new tool that enables developers to iterate more quickly with various templates and comprehend how prompts affect multiple scenarios. Prompt Usage Tracker, which teams may use to track their usage of prompts to understand their impact on a more granular level, is another debugging tool for prompts, responses, and chains.
Comet also disclosed new partnerships with LangChain Inc. and OpenAI LP, the company behind ChatGPT. According to the company, the OpenAI integration will make it feasible to use GPT-3 and other LLMs, while LangChain will make it easier to construct multilingual models.
“These integrations add significant value to users by empowering data scientists to leverage the full potential of OpenAI’s GPT-3 and enabling users to streamline their workflow and get the most out of their LLM development,” Mendels mentioned.
