Sandboxing AI agents, 100x faster

As artificial intelligence (AI) continues to evolve, the need for secure and efficient execution of AI-generated code becomes increasingly important. In this article, we explore the innovative approach of sandboxing AI agents, which allows for the rapid execution of code while maintaining security and performance.

The Need for Code Execution in AI

In September 2022, we introduced a concept known as Code Mode, which emphasizes that AI agents should perform tasks by writing code that calls APIs rather than making direct tool calls. This shift has shown significant benefits, including a reduction in token usage by up to 81% when converting a Multi-Cloud Provider (MCP) server into a TypeScript API. Furthermore, Code Mode can operate behind an MCP server, exposing the entire Cloudflare API with minimal resource consumption.

Understanding the Importance of Sandboxing

When AI generates code on-the-fly, that code needs to be executed in a secure environment. Directly evaluating AI-generated code could lead to vulnerabilities if a malicious user exploits the system. Therefore, sandboxing is essential. A sandbox is an isolated environment where code can run safely, limiting its access to only the necessary capabilities.

Challenges with Traditional Sandboxing Methods

Many in the AI industry have turned to containers for sandboxing. Containers provide a Linux-based environment to execute code, but they come with significant drawbacks:

• Slow Startup Times: Containers can take hundreds of milliseconds to boot.
• High Resource Consumption: They often require hundreds of megabytes of memory to operate.
• Security Risks: Reusing existing containers for multiple tasks can compromise security.

Given these limitations, especially for consumer-scale agents where each user may have multiple agents, a more efficient solution was needed.

Introducing the Dynamic Worker Loader

The solution lies in the Dynamic Worker Loader, a feature introduced alongside Code Mode. This API allows a Cloudflare Worker to instantiate a new Worker in its own sandbox with code specified at runtime. This innovation is now in open beta, available to all paid Workers users.

How It Works

Here’s a simplified example of how the Dynamic Worker Loader operates:

        let agentCode: string = ` export default { async myAgent(param, env, ctx) { // ... } } `;
        let chatRoomRpcStub = ...; 
        let worker = env.LOADER.load({
            compatibilityDate: "2026-03-01",
            mainModule: "agent.js",
            modules: { "agent.js": agentCode },
            env: { CHAT_ROOM: chatRoomRpcStub },
            globalOutbound: null,
        });
        await worker.getEntrypoint().myAgent(param);
    

Performance Advantages

The Dynamic Worker Loader utilizes isolates, which are instances of the V8 JavaScript execution engine. This technology has been the backbone of the Cloudflare Workers platform for eight years. The benefits of using isolates include:

• Speed: Isolates start in just a few milliseconds.
• Memory Efficiency: They consume only a few megabytes of memory, making them 100x faster and 10x-100x more memory efficient than traditional containers.

This efficiency allows for unlimited scalability, enabling the handling of millions of requests per second without the typical constraints imposed by container-based sandbox providers.

Zero Latency Execution

One-off Dynamic Workers usually run on the same machine and thread as the Worker that created them, eliminating the need for inter-machine communication. This results in zero latency, as isolates can be executed wherever the request originated. Dynamic Workers are supported in all of Cloudflare’s global locations, ensuring rapid response times.

JavaScript as the Language of Choice

While technically, Workers can utilize Python and WebAssembly, JavaScript is the preferred language for small code snippets generated on-demand by agents. JavaScript’s design facilitates sandboxing, making it ideal for this application. Moreover, AI models are well-trained in JavaScript, allowing them to generate code efficiently.

Utilizing TypeScript for API Interaction

To enable agents to interact with external APIs effectively, TypeScript is employed. This language is concise and allows for a clear definition of the APIs that agents can access. For instance, a simple interface for a chat room can be defined in TypeScript, vastly simplifying the process compared to verbose OpenAPI specifications.

Example TypeScript Interface

        interface ChatRoom {
            getHistory(limit: number): Promise;
            subscribe(callback: (msg: Message) => void): Promise;
            post(text: string): Promise;
        }
        type Message = { author: string; time: Date; text: string; };
    

Conclusion

Sandboxing AI agents has made significant strides with the introduction of the Dynamic Worker Loader. This innovative approach offers a secure, efficient, and scalable solution for executing AI-generated code. By leveraging the power of isolates and the flexibility of JavaScript and TypeScript, developers can create responsive AI applications that meet the demands of today’s digital landscape.

Note: The information presented in this article is based on the latest developments in AI sandboxing technology as of October 2023.