Latent Space: The AI Engineer Podcast — Practitioners talking LLMs, CodeGen, Agents, Multimodality, AI UX, GPU Infra and al
Code Interpreter == GPT 4.5 (w/ Simon Willison, Alex Volkov, Aravind Srinivas, Alex Graveley, et al. — AUDIO FIXED)
Mon Jul 10 2023Description
The episode covers the new feature in Jambotron called code interpreter, the capabilities of the AI model used in the podcast, timeouts and disconnections in code interpreter, running a containerized environment, Python libraries for generating files, data analysis capabilities, exploits and system performance, using code interpreter as a fellow debugger, uploading code to Code Interpreter, different behaviors observed in the model, interfacing with external plugins, and the potential use of code interpreter for business analysts.
Insights
Code interpreter can be used as a coding intern
Code interpreter functions like a coding intern, both smart and stupid at the same time. It can intake and run code in a secure environment, allowing users to upload files and run code on them. It has built-in libraries like pandas and map plotlib. However, uploading new Python packages to code interpreter is no longer supported.
The AI model used in the podcast is like a coding intern
The AI model used in the podcast is like a coding intern that is both smart and stupid. It never gets frustrated or gives up, making it very fast. Experienced programmers can develop a mental model of what the AI model can do and use that to coach it. However, there are limitations to the AI model, such as no web access and a maximum upload limit of 100 megabytes.
Timeouts and disconnections can cause loss of state and data
Timeouts and disconnections in code interpreter can cause loss of state and data. Errors can occur when files are lost or the chat thinks it has files that it doesn't. Sometimes timeouts can be beneficial, aborting long operations and suggesting shorter code. Refactoring code into smaller functions can help manage token limits and speed.
Running a containerized environment with code interpreter
The speaker suspects that the other person in the conversation is running a containerized environment. The home directory is located at /home/sandbox. The speaker mentions the virtualization and network access capabilities of the containerized environment. There is a hunch that the containerized environment is secure and unlikely to have exploits for breaking out of the network sandbox.
Python libraries for generating files
Python libraries in requirements.txt can generate HTML, CSS, and JavaScript files. JavaScript file can embed all the data needed by GPT. Additional possibilities open up if you download and open the file. GPT can write Excel files, PDFs, and more. Non-programmers can use this feature effectively.
Exploits and system performance
Exploits can be found in the files and run for privilege escalation. The system performance may vary, possibly due to shared instances. Benchmarking micro tasks is quick and easy on the system. Timeouts can occur during execution, similar to spot instances. Running phase detection and image recognition tasks is possible using libraries like OpenCV.
Using code interpreter as a fellow debugger
The code interpreter can act as a fellow debugger in software companies. It can reproduce issues and provide steps to correct them. The agent design of the code interpreter allows it to decide whether to proceed on its own or ask for more instructions. Asking for multiple options when using GPT can speed up the process and provide different visualizations or answers.
Uploading code to Code Interpreter
Uploading code to Code Interpreter requires the code to be known somehow. Code can be copied and pasted into Code Interpreter for evaluation. There is no workaround to upload a full GitHub repository or zip file of Python code without spending tokens. Code Interpreter allows streaming tokens into it by updating previous messages.
Different behaviors observed in the model
Different behaviors observed in the model may come from different models with continuous improvement capabilities. Training the tool by using it helps it understand our needs and bridge gaps in its capabilities. The model keeps getting better with each use, allowing us to solve new problems. Code interpreter can be used for extracting audio from video files and performing OCR on images.
Interfacing with external plugins
Code interpreter can interface with external plugins that call external APIs, allowing for additional functionality and integration. Plugins in JBBB without web access are a way to access external services via APIs. Code Interpreter doesn't have access to plugins yet due to security concerns.
Code interpreter for business analysts
Code interpreter could be a valuable tool for business analysts. To improve code interpreter's usability for business analysts, OpenAI should provide API access to the fine-tuned model and allow data to be provided through an API key. Uploading up to 100 megabytes of data can already take users a long way in code interpreter, but it falls short when users don't know what specific data they need at the beginning of their analysis.
Chapters
- New feature in Jambotron called code interpreter
- The AI model used in this podcast
- Timeouts and disconnections
- The speaker suspects that the other person in the conversation is running a containerized environment
- Python libraries in requirements.txt can generate HTML, CSS, and JavaScript files
- Used the haversine formula for latitude and longitude distances to filter data
- Exploits can be found in the files and run for privilege escalation
- The code interpreter can act as a fellow debugger in software companies
- Uploading code to Code Interpreter requires the code to be known somehow
- Using different models in code interpreter can result in different behaviors
- Code Interpreter can interface with external plugins that call external APIs
- Code interpreter could be a valuable tool for business analysts
New feature in Jambotron called code interpreter
00:05 - 06:01
- Code interpreter is now available for all paying users
- It can intake and run code in a secure environment
- Users can upload files and run code on them, including SQLite queries
- Code interpreter has built-in libraries like pandas and map plotlib
- Uploading new Python packages to code interpreter is no longer supported
- Code interpreter can interpret file formats it doesn't have libraries for based on its knowledge of the world
- Files can be compressed to bypass the upload limit
- Code interpreter functions like a coding intern, both smart and stupid at the same time
The AI model used in this podcast
05:51 - 12:02
- The AI model is like a coding intern that is both smart and stupid
- It never gets frustrated or gives up, making it very fast
- Experienced programmers can develop a mental model of what the AI model can do and use that to coach it
- Tricking the AI model by giving it specific instructions can make it forget certain capabilities
- The AI model can be prompted to act as a senior developer and execute code with actual execution powers
- The AI model can be used to write code, test it, and fix bugs quickly
- When using the AI model for code, it may invent APIs that don't exist but will fix mistakes before providing the final result
- There are limitations to the AI model, such as no web access and a maximum upload limit of 100 megabytes
- The AI model used to be able to run sub-processes but seems to have been locked down now
- There are time limits on how long each line of code can run in the AI model's container
Timeouts and disconnections
11:44 - 17:56
- Timeouts and disconnections can cause loss of state and data
- Errors can occur when files are lost or the chat thinks it has files that it doesn't
- Sometimes timeouts can be beneficial, aborting long operations and suggesting shorter code
- Refactoring code into smaller functions can help manage token limits and speed
- Uploading large amounts of text is slow, better to upload to a file
- Context window limit for GPT-4 is unknown
- Git bindings would be useful for making changes and committing them
- Python diff lib can be used for generating diffs
- Network vulnerabilities were found in some requirements files
The speaker suspects that the other person in the conversation is running a containerized environment
17:32 - 24:04
- The home directory is located at /home/sandbox
- The speaker mentions the virtualization and network access capabilities of the containerized environment
- There is a hunch that the containerized environment is secure and unlikely to have exploits for breaking out of the network sandbox
- The speaker expresses a desire to have an exploit that allows executing binary files again
- They discuss prompting the system to print CRL statements instead of making network connections
- The limitations of using code interpreter for data augmentation are mentioned, as it cannot call OpenAI to fill in blanks with existing models due to lack of network access
- Using raw GPT-3 may be better for certain tasks like data augmentation or uploading JSON files
- Capabilities and use cases of code interpreter are discussed, including visualization libraries like mapplotlib and generating images from Python libraries
- It is mentioned that code interpreter can render results as images but does not support SVG or JavaScript-based visualizations
- Mapplotlib is highlighted as an ancient Python plotting library that GPT-3 has learned how to use effectively
- A hack discovered by Ethan allows generating HTML, CSS, and JavaScript files using GPT-3
Python libraries in requirements.txt can generate HTML, CSS, and JavaScript files
23:44 - 30:14
- JavaScript file can embed all the data needed by GPT
- Additional possibilities open up if you download and open the file
- GPT can write Excel files, PDFs, and more
- Non-programmers can use this feature effectively
- Torch audio and torch are other libraries to explore
- FFmpeg allows interaction with video files
- FFmpeg Python bindings may be used instead of subprocess.call function
- Data analysis capabilities of code interpreter are impressive
- Code interpreter does everything on the roadmap for dataset project
- Dataset plus large language models need to be better than code interpreter
Used the haversine formula for latitude and longitude distances to filter data
29:49 - 36:15
- Filtered data to only include rows within 500 meters
- Calculated and plotted numbers on a comparative chart
- Found that Whole Foods received more calls than Safeway
- Generated a SQLite file of crimes affecting the two supermarkets
- Impressed by how quickly and accurately the AI system provided results
- Discussed options for accessing internet and packages through proxy or reverse engineered API
- Considered rebuilding code and circuit using GPT-4's API functions
- Speculated that the AI model may be fine-tuned for this specific task
- Injected system prompts into the model to gather information about its capabilities
- Suggested that an earlier checkpoint of GPT-4 could be used for different prompt injection techniques
- Confirmed use of Kubernetes cluster by OpenAI
- Acknowledged limitations and troubleshooting efforts mentioned in comments section
Exploits can be found in the files and run for privilege escalation
35:50 - 42:16
- There is a desire to run binaries on the system for additional capabilities
- Acceleration is expected in the future, potentially with GPU support
- The system performance may vary, possibly due to shared instances
- Benchmarking micro tasks is quick and easy on the system
- Timeouts can occur during execution, similar to spot instances
- Torch was used for estimating an axle and prompt engineering tweaks
- Consulting services are available for OpenAI's security needs
- System specs include 54 gigabytes of RAM, but access to system-level commands is restricted
- Running phase detection and image recognition tasks is possible using libraries like OpenCV
The code interpreter can act as a fellow debugger in software companies
41:59 - 48:33
- It can reproduce issues and provide steps to correct them
- The agent design of the code interpreter allows it to decide whether to proceed on its own or ask for more instructions
- Asking for multiple options when using GPT can speed up the process and provide different visualizations or answers
- Uploading project documentation to the code interpreter and teaching it how to run searches could allow it to answer questions about the docs
- Running a vector DB with cosine similarity in the code interpreter could enable vector search for pipeline similarity
- Downloading token files from the code interpreter is possible, but caution should be exercised due to terms of service restrictions
- There are differences in parameters and endpoints between the normal GPT-4 model and the model used by the code interpreter
Uploading code to Code Interpreter requires the code to be known somehow
48:14 - 54:51
- Code can be copied and pasted into Code Interpreter for evaluation
- There is no workaround to upload a full GitHub repository or zip file of Python code without spending tokens
- Code Interpreter allows streaming tokens into it by updating previous messages
- Tiny Grad is an alternative to using PyTorch, and there is a podcast episode interviewing George Hodges about it
- Code Interpreter is great for uploading CSV files and plotting graphs
- A plugin that creates a vector database for relevant results would be a game changer in Code Interpreter
- It would be cool if Code Interpreter could work with plugins and store documentation in a vector database
- Uploading an entire GitHub repo to Code Interpreter may not fit in context but running specific files might be possible
- Providing additional context through files instead of prompts could improve the system's understanding
- Shared snippets of known working code are needed in Code Interpreter
- The manual for Code Interpreter is missing, so the community needs to figure out what it can do and how to use it effectively
- Different behaviors observed in the model may come from different models with continuous improvement capabilities
Using different models in code interpreter can result in different behaviors
54:30 - 1:01:23
- Training the tool by using it helps it understand our needs and bridge gaps in its capabilities
- The model keeps getting better with each use, allowing us to solve new problems
- Code interpreter can be used for extracting audio from video files and performing OCR on images
- Comparing code interpreter to other tools, it has shown better performance in OCR tasks
- Code interpreter can reason over tables and reproduce text accurately from image files
- RAM limitations have been observed when running Python scripts on the system
- Plugins in code interpreter allow for additional functionality and integration with external APIs
Code Interpreter can interface with external plugins that call external APIs
1:00:57 - 1:08:06
- Plugins in JBBB without web access are a way to access external services via APIs
- Code Interpreter doesn't have access to plugins yet due to security concerns
- A hack allows ChatGPT to have infinite memory by creating a text file named chatgptmemory.txt
- Teaching code interpreter to grep the file could allow for longer memory storage and retrieval
- The audience is collectively trying to find a way to upload a repo library zip file to extend code interpreter's capabilities
- There is a bug on the phone app where it continuously prompts itself if you open it on iOS and swap between web and app
Code interpreter could be a valuable tool for business analysts
1:59:00 - 2:04:00
- To improve code interpreter's usability for business analysts, OpenAI should provide API access to the fine-tuned model and allow data to be provided through an API key
- Uploading up to 100 megabytes of data can already take users a long way in code interpreter, but it falls short when users don't know what specific data they need at the beginning of their analysis
- Code interpreter allows for different styles of code based on prompts, enabling users to get interpretations from different personas like data engineers or statisticians
- A user shared their experience using code interpreter with Swift code and found that it accurately described the elements of their game and provided suggestions for improvement
- The ability to upload and download files in code interpreter is seen as a valuable feature, with potential interest in zipping multiple files together for download
- Overall, participants encourage others to explore and experiment with code interpreter