Multi-Language Dependency Extraction

WHEN TO USE (Triggers)

1. When analyzing a GitHub repo's tech stack programmatically
2. When building project-to-repo matching based on shared dependencies
3. When parsing dependencies from multiple languages (Node + Python)
4. When pyproject.toml parsing fails silently (complex TOML format)
5. When dependency names have version specifiers that need stripping

FAILED ATTEMPTS

CORRECT PATTERN

Unified Extraction Function

function extractDependencies(packageJson, requirementsTxt, pyprojectToml) {
  const deps = [];

  // 1. Node.js: package.json
  if (packageJson) {
    try {
      const pkg = JSON.parse(packageJson);
      if (pkg.dependencies) deps.push(...Object.keys(pkg.dependencies));
      if (pkg.devDependencies) deps.push(...Object.keys(pkg.devDependencies));
    } catch { /* malformed JSON */ }
  }

  // 2. Python: requirements.txt
  if (requirementsTxt) {
    for (const line of requirementsTxt.split('\n')) {
      const trimmed = line.trim();
      if (!trimmed || trimmed.startsWith('#') || trimmed.startsWith('-')) continue;
      const name = trimmed.split(/[=<>~![\s]/)[0].trim();
      if (name) deps.push(name.toLowerCase());
    }
  }

  // 3. Python: pyproject.toml (regex — no TOML parser needed)
  if (pyprojectToml) {
    const depMatch = pyprojectToml.match(/dependencies\s*=\s*\[([\s\S]*?)\]/);
    if (depMatch) {
      const entries = depMatch[1].match(/"([^"]+)"|'([^']+)'/g);
      if (entries) {
        for (const entry of entries) {
          const raw = entry.replace(/["']/g, '');
          const name = raw.split(/[=<>~![\s]/)[0].trim();
          if (name) deps.push(name.toLowerCase());
        }
      }
    }
  }

  // Deduplicate and normalize
  return [...new Set(deps.map(d => d.toLowerCase()))];
}

Parallel File Fetching (GitHub)

const base = `https://raw.githubusercontent.com/${repo}/HEAD`;
const [pkgRes, reqRes, pyRes] = await Promise.all([
  fetch(`${base}/package.json`).catch(() => null),
  fetch(`${base}/requirements.txt`).catch(() => null),
  fetch(`${base}/pyproject.toml`).catch(() => null),
]);

EVIDENCE

QUICK START (< 5 minutes)

1. Copy extraction function (1 min): The unified function handles all 3 formats
2. Fetch files in parallel (1 min): Promise.all for package.json + requirements.txt + pyproject.toml
3. Deduplicate (30 sec): Set for unique, lowercase for case-insensitive
4. Test (1.5 min): Parse a known repo, verify all deps extracted