Practical AI: Machine Learning, Data Science, LLM

A developer's toolkit for SOTA AI

Wed Jul 12 2023

Generative AISoftware DevelopmentGPU VirtualizationCode AutocompleteInfrastructureWorkflow Improvement

Description

This episode of the Practical AI Podcast explores the challenges and advancements in using generative AI for software development. The guests, Varun and Anjul, discuss their experiences with Codium, a tool designed for engineers using generative AI. They delve into topics such as GPU virtualization, Cody M's unique approach to language models, OpenAI's infrastructure and code generation, fine-tuning workflows, Codeium's features and advantages, and the role of AI in software development.

Insights

GPU Scarcity and Cost

Companies face difficulties in accessing GPUs due to NVIDIA scarcity issues, with a single H100 chip costing over $30,000.

Cody M's Unique Approach

Cody M focuses on a specific application and modality of language models (LMs) for real-time code instruction, providing an AI toolkit for software development beyond autocomplete.

OpenAI's Infrastructure and Privacy

OpenAI processes a billion lines of code daily, leveraging their chat application to optimize infrastructure while respecting privacy concerns.

Workflow Improvements

Fine-tuning locally, automated diff application, and chat functionality streamline the generative AI workflow, improving efficiency and reducing frustrations.

Codeium's Advantages

Codeium offers small buttons for code generation, resolves merge conflicts automatically, provides extensive context for autocomplete, and aims to continually improve model knowledge and helpfulness.

The Role of AI in Software Development

AI tools aim to automate frustrating and time-consuming tasks, but human involvement and practicality should guide their development to ensure usefulness for software developers.

Chapters

Introduction to the Practical AI Podcast
GPU Virtualization and Challenges
Cody M and Generative AI for Software Development
OpenAI's Approach to Code Generation
Fine-tuning and Workflow Improvements
Codeium's Features and Advantages
The Role of AI in Software Development

Summary

Transcript

Introduction to the Practical AI Podcast

00:05 - 06:54

Guests: Varun, CEO and co-founder of Codium, and Anjul, lead of their enterprise and partnership
Varun's background in working at Nuro, an AV company, and developing large-scale deep learning workloads
Xafunction's GPU virtualization software to effectively use GPU hardware
Creation of Codium as an application for engineers using generative AI
Challenges with virtualizing GPUs compared to CPUs
Scarcity and cost issues with GPUs

GPU Virtualization and Challenges

06:27 - 12:58

Companies have trouble getting access to 10 GPUs due to NVIDIA scarcity issues.
The cost of a single H100 chip is well over $30,000.
Ex-Function built software to help deep learning applications leverage GPUs.
The industry has seen a significant increase in GPU usage for training workloads.
Ex-Function managed over 10,000 GPUs on GCP in a single region.
Generative AI, like GitHub Co-pilot, became one of the largest GPU workloads.
Codeium was positioned specifically for code autocompletion and had low latency requirements.

Cody M and Generative AI for Software Development

12:43 - 19:21

The model used for Cody M and Generative AI needs to be relatively small and low latency due to the real-time nature of code instruction.
Context before and after the cursor in code is important, not just what comes before.
Cody M focuses on a specific application and modality of language models (LMs) that is unique.
Cody M aims to build an AI toolkit for software development rather than just an autocomplete tool like CoPilot.
Cody M provides an IDE chat, natural language search over code base using embeddings, and vector stores to address the entire software development lifecycle.
Cody M has ML infrastructure expertise, allowing enterprises to use it confidently and safely.
The transition from infrastructure-focused services to Cody M involved a pivot in product focus and building out an individual tier with feedback from developers.
Over 100,000 active developers are using Cody M's individual plan, providing valuable feedback for product improvement.
Having expertise in ML infrastructure differentiates Cody M from other tools in the marketplace.
Codeium sets a high bar by offering its product entirely for free with unlimited usage, forcing efficient infrastructure design.
Codeium processes over 10 billion tokens of code per day and has one of the world's largest chat applications integrated into its IDE platform.

OpenAI's Approach to Code Generation

18:56 - 25:00

OpenAI processes over a billion lines of code a day for its own developers.
OpenAI has one of the world's largest chat applications, which allows them to learn from their users and optimize their infrastructure.
Codeium, OpenAI's product, will always have a free individual plan.
OpenAI focuses on building the best infrastructure to deliver better experiences to users.
By self-hosting, OpenAI can leverage a company's knowledge base for personalization and create a better product.
Fine-tuning locally in a company's tenant is an infrastructure problem that OpenAI has solved.
OpenAI's solution can take advantage of hidden IP in code and provide value to companies while respecting privacy concerns.

Fine-tuning and Workflow Improvements

24:40 - 31:07

IP never leaves coding for enterprises
Training can be done on the customer's site without sharing IP
Meta trained their own autocomplete model using their own code
Fine-tuning locally is possible with a smaller amount of data
No additional hardware provisioning is required for fine-tuning
Idle time on GPU can be used for continuous model improvement
Generative AI workflows and user experience are still being figured out
Copying and pasting code blocks in the workflow is frustrating
Chat functionality streamlines the workflow by providing buttons for common tasks like explaining functions, refactoring, writing unit tests, etc.
Automated diff application eliminates the need to copy and resolve merge conflicts

Codeium's Features and Advantages

30:41 - 36:39

Codeium provides small buttons and preset prompts for generating code
Merge conflicts are automatically resolved when applying diffs in Codeium
Auto-complete accounts for more than 90-95% of the usage of Codeium
Codeium recognizes two modalities of typing code: accelerating the developer and exploration phase
Codeium aims to make the exploration case more efficient and helpful
Codeium allows users to easily validate and correct generated code completions
Codeium provides double the amount of context compared to CoPilot for autocomplete
Codeium pulls context throughout the code base, including search functionality
Fine-tuning is used in Codeium to semantically understand enterprise codebases
The vision for Codeium is to continually increase knowledge and helpfulness of models
CoPilot lacks codebase-aware chat functionality that Codeium offers
Generative AI technologies are progressing rapidly, leading to faster development of tools like Codeium

The Role of AI in Software Development

36:17 - 41:46

AI will be the next step in a series of tools that have made developers focus on more interesting parts of software development.
Human involvement and guidance will still be necessary in the AI development process.
The goal is to use AI to automate tasks that are frustrating, boring, and slow down engineers.
There are more capabilities to build beyond search and chat.
Educating software developers on how to best use AI tools is important.
The worry is that there will be exciting new demos without actual usefulness for users.
Strategic bets need to be made on what areas to invest in for future development.
Building tools ahead of technological capability can lead to diminishing returns.
Practicality and usefulness should guide the development of AI tools.