Managing Generative AI Projects

Data Usage: Traditional Data Science Projects vs Generative AI projects

Before we delve into the specifics of managing a Generative AI project, let’s take a step back to compare and contrast the use of data in traditional data science projects versus Generative AI projects.

Traditional data science projects, such as building a machine learning predictive model, typically focus on making sense of existing data. They employ structured methods to analyze and interpret data, extracting valuable insights to drive decision-making and predictive capabilities. This is why data preparation is such an important phase of the data science life cycle. People often describe Generative AI as focused on creating data (such as an email or an image), implying that data preparation is not required.

However, in reality, this is not totally accurate. Many Generative AI tools need access to data. For example, to develop a Generative AI chatbot for a tech company, it might need to answer customer queries using the company’s own knowledge repository. To do this effectively, the AI will need access to that repository. However, this data might be unstructured, full of jargon, or mixed with irrelevant information. In this situation, the team would need to clean the data, by organizing it and removing unnecessary details, to ensure that the AI can understand and use it properly.

Exploring an Appropriate Process Framework

The data science process typically consists of two complementary frameworks. First, a data science life cycle, which defines the steps or phases of a project. Next is the collaboration framework, which defines how to coordinate and collaborate.

Applying CRISP-DM to Manage Generative AI Projects

A commonly used data science life cycle framework is CRISP-DM (Cross-Industry Standard Process for Data Mining). The CRISP-DM life cycle includes six phases: Business Understanding, Data Understanding, Data Preparation, Modeling, Evaluation, and Deployment.

To keep things simple, lets map the typical steps in a Generative AI project to the CRISP-DM life cycle framework.

Business Understanding

This phase entails understanding the objectives, the constraints, and defining the specific problem the AI model needs to solve. Hence, the following is an example of a task that would be in this phase:

• The team works to understand the goals of the project, and how the tool might be used. This typically requires multiple discussions with stakeholders.

Data Understanding

This phase entails understanding what data might be available and useful. Hence, the following task would be an example of work done during this phase:

• The team explores and determines what (if any) data (such as proprietary knowledge captured in documents) might be needed for the bot.

Data Preparation

While traditional data science projects often rely on structured, labeled datasets, Generative AI projects typically use unstructured data. Hence, the following would be examples of tasks that would be part of this phase:

• For a project that needs to build a vector database, the team will likely need to clean and organize this data so it can be effectively used. The vector database is essentially a collection of data that the Generative AI model will leverage to gain proprietary knowledge.
• For a project that requires fine-tuning a model, the appropriate data must be collected and cleaned for the fine-tuning process.

Modeling

In a typical data science project, the team will need to build a model, typically using Python or R libraries (ranging from a simple linear model to a deep learning neural network). When using a Generative AI model, due to the amount of data and computation required, the model used will typically be “pre-trained”. Many projects can use the model “out-of-the-box”, while other projects might need to fine-tune the model. While a team often leverages a pre-built model, the team still needs to ensure the model is useful within this project context. As one can see in the list of example tasks below, there is significant work that is required in the modeling phase of a project:

• For a project that requires fine-tuning a model, the team will need to build a new, fine-tuned, model.
• Prompt engineering, which is about crafting the right instructions (or prompts) that need to be fed into the Generative AI model, is a key aspect of working with a model to get the desired results.
• The project might need to connect multiple AI agents together, to leverage their strengths and improve the overall performance of the system.
• The model’s output will need to be formatted in a way that’s useful and understandable.
• Note that a model often needs to be integrated into a larger system or application (such as a chat interface), this work might be done by the data science team, or a software engineering team.

Evaluation

Evaluating the output of a Generative AI model is a major challenge. In a conventional data science project, the metrics are typically known (like accuracy, precision, recall). But in Generative AI, how does one determine if a generated draft email or image is ‘correct’? This might require a more subjective and novel evaluation metric. Hence, the following task is a typical example of what needs to be done during evaluation:

• The team evaluates and validates the model’s resulting output. Note that this might require analysis and feedback from human evaluators.

Deployment

This phase involves integrating the trained model into a production environment. However, this might be more complex for Generative AI due to the increased computational requirements and the potential need for continuous learning. The following is an example task that is focused on deployment:

• To ensure the model performs well in real-world conditions, the team needs to establish processes to monitor and maintain the model over time.

Collaboration Frameworks

Just as with more traditional data science / machine learning projects, building a chatbot (or another Generative AI tool) is an iterative process that involves a wide range of skills, ranging from data handling to programming to ethical considerations. In other words, the team will likely need to go through these steps multiple times, making improvements along the way, until the model performs as desired.

Furthermore, as with other data science projects, teams must consider potential ethical issues such as bias throughout the life cycle. More specifically, the team needs to ensure their model does not generate biased or harmful outputs. This might involve creating specific evaluation protocols and considering the ethical implications of the model’s outputs.

Typical collaboration frameworks used by data science teams include Data-Driven Scrum (DDS) and Scrum. These frameworks are iterative in nature, allowing the team to accommodate changes and improvements in small increments. These frameworks also facilitate communication and collaboration among the team and stakeholders. The key difference is that DDS has capability-based iterations, and Scrum has time-boxed iterations.

Key Insights on Managing Generative AI Projects

In exploring the tasks required to create a Generative AI bot, just as for traditional data science projects, creating a Generative AI model is an iterative process that involves a wide range of tasks (and associated skills), from business understanding, to data preparation, to modeling and deployment.

While the specific tasks, and expertise required to do the tasks might be different, the underlying phases of the project are very similar.

Furthermore, despite the distinctive nature of Generative AI and traditional Data Science projects, another common thread is the importance of team collaboration. Harmonious and effective team dynamics are instrumental in propelling any project toward its objectives.

As we push forward into this exciting frontier of Generative AI, project managers, data scientists, and AI engineers need to equip themselves with the understanding and skills to navigate the unique challenges posed by Generative AI projects. This dynamic field is ever-evolving, and as such, our approaches to managing these projects must evolve too. To thrive in this domain, teams should adopt an appropriate project life cycle and coordination framework.

Learn More

• To learn about what Agile means for data science and machine learning projects, read the Agile data science post.
• Consider asking 10 questions before starting a data science project.