Troubleshooting Node-API Failures In Second-Level Dependencies Within Workers

Introduction

This article delves into a peculiar issue encountered while using Node-API within both worker_threads and Web Workers, specifically when dealing with second-level dependencies. The problem manifests when a package, in this case, moondream, which depends on sharp, fails to initialize properly within these worker environments. This behavior is inconsistent, as sharp works perfectly fine as a top-level dependency. The focus will be to understand the root cause of this failure and explore potential solutions, providing a comprehensive overview for developers facing similar challenges. We'll examine the conditions under which the failure occurs, the error messages encountered, and the environments in which the issue is observed, including Node.js, Deno, and Bun.

The Problem: Sharp Initialization Failure in Workers

The core issue revolves around the inability of the sharp module to initialize correctly when it is a second-level dependency within a worker thread or a Web Worker. This is illustrated by the following scenario:

1. Scenario: When moondream, a package that depends on sharp, is statically imported, its dependency on sharp fails to work in Bun's Node or Web Workers.
2. Top-Level Dependency: If sharp is imported directly as a top-level dependency, it functions correctly in all worker environments.
3. Dynamic Import: Interestingly, when moondream is dynamically imported, it initializes successfully in a Node Worker but fails in a Web Worker. This inconsistent behavior raises questions about the underlying mechanisms at play.

Code Snippet

Consider the following code snippet, which demonstrates the issue:

// WHEN STATICALLY IMPORTED:
// Its dependency on sharp doesn't work in Bun's Node or Web Workers
import * as moondream from "moondream";

// A top-level dependency on sharp works in all Workers
import sharp from "sharp";

import { Buffer } from "node:buffer";
async function workerFun() {
    // WHEN DYNAMICALLY IMPORTED:
    // It works in Bun's Node Worker, but not in Bun's Web Worker
    // const moondream = await import("moondream"); // ??? why does this work?

    const vl = new moondream.vl({ apiKey: "abcd" }); // valid api key not needed for repro

    const imageResponse = await fetch("https://upload.wikimedia.org/wikipedia/commons/thumb/6/66/SMPTE_Color_Bars.svg/672px-SMPTE_Color_Bars.svg.png");
    const preProcessedImage = Buffer.from(await imageResponse.arrayBuffer());

    const image = await sharp(preProcessedImage).resize(100).toFormat("jpeg").toBuffer();
    const newMetadata = await sharp(image).metadata();
    console.log(`Do I look like I know what a ${newMetadata.format} is?`);

    try {
        await vl.point({ image, object: "green bar" })
    } catch (e) {
        if (e.message.includes("401")) {
            console.log("401 error, moondream was successfully initialized");
        } else throw e;
    }
    process.exit(0);
}

import { Worker as NodeWorker, isMainThread } from "node:worker_threads";
import { setTimeout } from "node:timers/promises";

const inNodeWorker = !isMainThread;
if (inNodeWorker) console.log("I might be in a Node Worker");

const inWebWorker = typeof WorkerGlobalScope !== 'undefined' && self instanceof WorkerGlobalScope;
if (inWebWorker) console.log("I might be in a Web Worker");

const inBunWorker = ("Bun" in globalThis && !Bun.isMainThread);
if (inBunWorker) console.log("I might be in a Bun Worker");


if (!(inNodeWorker || inWebWorker || inBunWorker)) {
    console.log("===== Node Worker =====");
    await setTimeout(1000);
    const nodeWorker = new NodeWorker(new URL(import.meta.url));
    nodeWorker.on('error', (err) => {
        console.error(err.message);
    });
    nodeWorker.on('exit', (code) => {
        // Curiously never fires in Deno 2.4.0
        console.log(`Worker exited with code ${code}`);
    });

    await setTimeout(1000);
    if (!("Worker" in globalThis)) {
        console.log("\nWeb Worker is not available in the truly awesome Node.JS runtime");
        process.exit(0);
    }

    console.log("\n===== Web Worker =====");
    const worker = new Worker(new URL(import.meta.url), { type: "module" });
    worker.onerror = (err) => {
        console.error(err.message);
    };
} else await workerFun();

This code demonstrates the different scenarios under which sharp initialization fails within worker threads. When moondream is statically imported, the sharp dependency fails in both Node and Web Workers in Bun. However, when dynamically imported, it works in Node Workers but not in Web Workers. This inconsistency highlights a potential issue with how Bun handles second-level dependencies in different worker environments. This dynamic import behavior suggests that the timing and context of module loading might play a crucial role in the failure.

Error Messages

When the failure occurs, the following error messages are observed:

• Node Worker:

  Something went wrong installing the "sharp" module
  
  symbol 'napi_register_module_v1' not found in native module. Is this a Node API (napi) module?
  
  Possible solutions:
  - Install with verbose logging and look for errors: "npm install --ignore-scripts=false --foreground-scripts --verbose sharp"
  - Install for the current linux-x64 runtime: "npm install --platform=linux --arch=x64 sharp"
  - Consult the installation documentation: https://sharp.pixelplumbing.com/install
  

• Web Worker:

  error: 
  Something went wrong installing the "sharp" module
  
  symbol 'napi_register_module_v1' not found in native module. Is this a Node API (napi) module?
  
  Possible solutions:
  - Install with verbose logging and look for errors: "npm install --ignore-scripts=false --foreground-scripts --verbose sharp"
  - Install for the current linux-x64 runtime: "npm install --platform=linux --arch=x64 sharp"
  - Consult the installation documentation: https://sharp.pixelplumbing.com/install
        at <anonymous> (breakable/node_modules/moondream/node_modules/sharp/lib/sharp.js:37:9)
        at <anonymous> (breakable/node_modules/moondream/node_modules/sharp/lib/constructor.js:10:41)
        at <anonymous> (breakable/node_modules/moondream/node_modules/sharp/lib/index.js:6:7)
        at <anonymous> (breakable/node_modules/moondream/dist/src/moondream.js:8:17)
  

These error messages indicate that the sharp module, which is a Node-API (NAPI) module, is failing to register properly within the worker environment. The error suggests a missing symbol, napi_register_module_v1, which is a crucial part of the NAPI initialization process. This implies that the module is either not being correctly linked or is not being initialized in the expected manner within the worker context. The proposed solutions in the error message, such as verbose logging during installation and specifying the runtime platform, are standard troubleshooting steps for NAPI module issues.

Environment and Versions

The issue has been observed in the following environments and versions:

• Node.js: Version 24.3.0
• Deno: Version 2.4.0
• Bun: Version 1.2.18

The fact that the issue is reproducible across different versions of Node.js runtimes (Node.js, Deno, and Bun) suggests that the problem might be related to the underlying worker implementation or the way these runtimes handle NAPI modules in worker threads. It's also noteworthy that Deno, unlike Node.js, doesn't exhibit the