Issue Review: cloudflare/workers-sdk#9957

Summary

Sending large data via Vitest's provide configuration causes opaque websocket assertion failures or silent test omissions due to HTTP header size limits.

Findings

• Created: 2025-07-14
• Updated: 2025-08-11
• Version: @cloudflare/vitest-pool-workers@0.8.19 -> 0.12.6 (current)
• Component: vitest-pool-workers
• Labels: bug, vitest
• Comments: 0

Key Evidence

• No merged PRs reference issue #9957
• Changelog has no entries mentioning fixes for this issue
• Code review confirms the root cause: providedContext data is passed via HTTP header MF-Vitest-Worker-Data which has size limits
• The chunking mechanism (createChunkingSocket) is only used AFTER websocket connection is established, not for the initial handshake
• Reproduction repo available at https://github.com/samscott89/cf-workers-bug-repro

Root Cause Analysis

The issue is in packages/vitest-pool-workers/src/pool/index.ts:879-893:

const res = await stub.fetch("http://placeholder", {
  headers: {
    Upgrade: "websocket",
    "MF-Vitest-Worker-Data": structuredSerializableStringify({
      filePath: pathToFileURL(workerPath).href,
      name: method,
      data,  // <-- Contains providedContext which can be large
      cwd: process.cwd(),
    }),
  },
});
const webSocket = res.webSocket;
assert(webSocket !== null);  // <-- Line 894: Fails when header is too large

The data object includes providedContext from project.project.getProvidedContext() (line 863). When users pass large data via test.provide, it's serialized into the MF-Vitest-Worker-Data HTTP header.

HTTP headers have typical size limits:

• Most servers: 8KB-16KB per header
• Cloudflare Workers/workerd: Similar limits apply

The chunking socket implementation in src/shared/chunking-socket.ts handles large messages over websocket, but not for the initial HTTP request that establishes the connection.

Failure Modes

1. 64,000 array elements (~250KB serialized): Websocket upgrade fails, res.webSocket === null, assertion throws opaque error
2. 128,000 array elements (~500KB serialized): Complete silent failure - tests not even registered

Recommendation

Status: KEEP OPEN

Reasoning: This is a confirmed, reproducible bug with a clear root cause. No fix has been merged, and the issue affects users trying to pass fixture data to tests via the standard provide configuration.

Action: Implement a fix to handle large providedContext data.

Proposed Solution

Move the providedContext data transfer from HTTP header to websocket message after connection is established.

Option 1: Post-connection data transfer (Recommended)

File: packages/vitest-pool-workers/src/pool/index.ts

// Current code (line 879-893):
const res = await stub.fetch("http://placeholder", {
  headers: {
    Upgrade: "websocket",
    "MF-Vitest-Worker-Data": structuredSerializableStringify({
      filePath: pathToFileURL(workerPath).href,
      name: method,
      data,
      cwd: process.cwd(),
    }),
  },
});
const webSocket = res.webSocket;
assert(webSocket !== null);

// Proposed change:
const res = await stub.fetch("http://placeholder", {
  headers: {
    Upgrade: "websocket",
    // Only send minimal metadata in header
    "MF-Vitest-Worker-Init": "true",
  },
});
const webSocket = res.webSocket;
assert(webSocket !== null, "Failed to establish websocket connection for test runner");

const chunkingSocket = createChunkingSocket({
  post(message) {
    webSocket.send(message);
  },
  on(listener) {
    webSocket.addEventListener("message", (event) => {
      listener(event.data);
    });
  },
});

// Send the worker data over the established websocket (chunked for large payloads)
chunkingSocket.post(structuredSerializableStringify({
  type: "init",
  filePath: pathToFileURL(workerPath).href,
  name: method,
  data,
  cwd: process.cwd(),
}));

// Wait for acknowledgment before proceeding
// ... (would need corresponding changes in worker/index.ts)

File: packages/vitest-pool-workers/src/worker/index.ts

// Current code (line 253-266):
async handleVitestRunRequest(request: Request): Promise<Response> {
  assert.strictEqual(request.headers.get("Upgrade"), "websocket");
  const { 0: poolSocket, 1: poolResponseSocket } = new WebSocketPair();

  const workerDataHeader = request.headers.get("MF-Vitest-Worker-Data");
  assert(workerDataHeader !== null);
  const wd = structuredSerializableParse(workerDataHeader);
  // ...
}

// Proposed change:
async handleVitestRunRequest(request: Request): Promise<Response> {
  assert.strictEqual(request.headers.get("Upgrade"), "websocket");
  const { 0: poolSocket, 1: poolResponseSocket } = new WebSocketPair();
  
  const port = new WebSocketMessagePort(poolSocket);
  
  // Wait for init message over websocket instead of reading from header
  const wd = await new Promise((resolve) => {
    port.once("message", (msg) => {
      if (msg.type === "init") {
        resolve(msg);
      }
    });
  });
  
  // Send acknowledgment
  port.postMessage({ type: "init-ack" });
  
  // ... rest of the handler
}

Option 2: Add validation with helpful error message (Quick fix)

If option 1 is too complex for immediate implementation, at minimum add validation:

// In packages/vitest-pool-workers/src/pool/index.ts before line 879:
const workerData = structuredSerializableStringify({
  filePath: pathToFileURL(workerPath).href,
  name: method,
  data,
  cwd: process.cwd(),
});

const MAX_HEADER_SIZE = 8 * 1024; // 8KB conservative limit
if (Buffer.byteLength(workerData) > MAX_HEADER_SIZE) {
  throw new Error(
    `The data passed to Vitest's \`provide\` configuration is too large (${Buffer.byteLength(workerData)} bytes). ` +
    `Consider reducing the size of provided data or loading test fixtures from files instead of passing them directly.`
  );
}

Implementation Difficulty

Medium

Justification:

• The chunking infrastructure already exists and is well-tested
• Requires coordinated changes in both pool (sender) and worker (receiver)
• Needs to handle the async handshake properly
• May affect timing/ordering assumptions in existing code
• Should maintain backward compatibility with smaller payloads

Files to Modify

1. packages/vitest-pool-workers/src/pool/index.ts - Change how worker data is sent
2. packages/vitest-pool-workers/src/worker/index.ts - Change how worker data is received
3. packages/vitest-pool-workers/src/shared/chunking-socket.ts - May need minor adjustments (if any)

Testing Recommendations

1. Unit tests:

   • Test with small provide data (existing behavior)
   • Test with medium provide data (~10KB)
   • Test with large provide data (~100KB, ~500KB, ~1MB)

2. Integration tests:

   • Use the reproduction repo as a basis
   • Test all three scenarios from the issue: passing, failing, silently-failing

3. Edge cases:

   • Empty provide configuration
   • provide with circular references (should already fail gracefully)
   • provide with non-serializable data
   • Multiple test files with different provide sizes

4. Performance testing:

   • Ensure large payloads don't significantly impact test startup time
   • Verify memory usage with very large payloads