✻ | ▟█▙ Claude Code v2.1.47 ▐▛███▜▌ Opus 4.6 · Claude Max ▝▜█████▛▘ C:\Bigscreen\cloud ▘▘ ▝▝ ╭──────────────────────────────────────────────────────────────────────────────╮ │ Plan to implement │ │ │ │ Fix: Wiki Pages Not Synced to KB Documents Library │ │ │ │ Context │ │ │ │ The KB sync system has wiki sync support │ │ (apps/kb/src/ingestion/sources/github.ts), but wiki pages from │ │ https://github.com/BigscreenVR/unity/wiki are not being crawled. After │ │ thorough code review, the root cause is a premature early return in │ │ syncRepo() that skips wiki sync whenever the main repo's tree SHA is │ │ unchanged. Two secondary issues also contribute: silent 404 error │ │ suppression hides failures, and a hardcoded branch name prevents fallback to │ │ "main". │ │ │ │ File to modify │ │ │ │ - apps/kb/src/ingestion/sources/github.ts (single file, three focused fixes) │ │ │ │ Fix 1: Remove early return that blocks wiki sync (lines 385-389) — Core Bug │ │ │ │ Problem: syncRepo() at line 386-389 returns early when the main repo tree │ │ SHA is unchanged. Wiki sync (line 511) and commit sync (line 499) are never │ │ reached. The wiki is a separate git repo ({repo}.wiki) with its own tree SHA │ │ — it can change independently of the main repo. │ │ │ │ Fix: Replace the early return result with an if/else that only skips the │ │ document sync. Wiki and commit sync remain outside the conditional and │ │ always execute (they have their own sync state tracking). │ │ │ │ Before: │ │ if (previousTreeSha === currentTreeSha) { │ │ Logger.info(`Repository ${repoIdentifier} unchanged, skipping`); │ │ return result; ← skips everything including wiki │ │ } │ │ // ... doc sync (lines 391-496) ... │ │ // ... commit sync (lines 498-508) ... │ │ // ... wiki sync (lines 510-524) ... │ │ │ │ After: │ │ if (previousTreeSha === currentTreeSha) { │ │ Logger.info(`Repository ${repoIdentifier} documents unchanged, │ │ skipping document sync`); │ │ } else { │ │ // ... doc sync (lines 391-496) — wrapped in else block ... │ │ } │ │ // ... commit sync (lines 498-508) — always runs ... │ │ // ... wiki sync (lines 510-524) — always runs ... │ │ │ │ Fix 2: Add wiki branch fallback (lines 947-986) │ │ │ │ Problem: Line 953 hardcodes ref: "heads/master". If a wiki uses "main", the │ │ 404 is silently swallowed. │ │ │ │ Fix: Add a getWikiRef() helper that tries "master" then "main". Pass the │ │ resolved branch name through to syncWikiPage() (which also hardcodes │ │ "master" at line 1092). │ │ │ │ Changes: │ │ - New method getWikiRef(octokit, owner, wikiRepoName, wikiIdentifier) — │ │ tries heads/master, then heads/main, returns { branch, commitSha } or null │ │ - syncWiki() — replace the hardcoded getRef("heads/master") try/catch with a │ │ call to getWikiRef. If null, log a warning and return 0. │ │ - syncWikiPage() — add wikiBranch: string parameter, use it at line 1092 │ │ instead of hardcoded "master" │ │ - Call site at line 1037 — pass wikiBranch through │ │ - updateWikiSyncState() — optionally store resolved branch in metadata │ │ │ │ Fix 3: Improve wiki error visibility (lines 515-523) │ │ │ │ Problem: In the syncRepo catch block for wiki sync, 404 errors are excluded │ │ from result.errors. Combined with the info-level log in syncWiki, failures │ │ are invisible in Discord notifications. │ │ │ │ Fix: │ │ - In syncWiki: the "no wiki found" case now returns 0 (via getWikiRef │ │ returning null) with a warn-level log, rather than throwing a 404. │ │ - In syncRepo catch block (lines 515-523): remove the 404 suppression. Any │ │ exception that reaches this catch is a genuine error and should always be │ │ added to result.errors. │ │ │ │ Before: │ │ if (!error?.message?.includes("Not Found") && error?.status !== 404) { │ │ result.errors.push(errorMsg); │ │ } │ │ │ │ After: │ │ result.errors.push(errorMsg); │ │ │ │ Verification │ │ │ │ - Trigger a sync via the admin API and check that wiki pages appear in │ │ kb_documents │ │ - Verify that if only wiki changes (no main repo changes), wiki updates are │ │ picked up on next sync │ │ - Check Discord notification includes wiki sync count │ │ - Check that repos without wikis log a warning but don't produce false │ │ errors │ ╰──────────────────────────────────────────────────────────────────────────────╯ ● I'll start by reading the file to understand the current code before making changes. ● Read 1 file (ctrl+o to expand) ● Now I have the full picture. Let me implement all three fixes. Fix 1: Replace the early return with an if/else that only skips document sync. ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 92 lines, removed 92 lines 382 throw new Error(`Failed to get branch reference: $ {error?.message}`); 383 } 384 385 - // Skip if tree SHA unchanged (no changes in repo) 385 + // Skip document sync if tree SHA unchanged (no change +s in repo files) 386 + // Wiki and commit sync still run — they track their o +wn sync state independently 387 if (previousTreeSha === currentTreeSha) { 387 - Logger.info(`Repository ${repoIdentifier} unchange -d, skipping`); 388 - return result; 389 - } 388 + Logger.info(`Repository ${repoIdentifier} document +s unchanged, skipping document sync`); 389 + } else { 390 + // Get all files in the repository tree (recursive +) 391 + const { data: tree } = await this.withRetry( 392 + () => octokit.git.getTree({ 393 + owner: repoConfig.owner, 394 + repo: repoConfig.repo, 395 + tree_sha: currentTreeSha, 396 + recursive: "true", 397 + }), 398 + `getTree(${repoIdentifier})` 399 + ); 400 391 - // Get all files in the repository tree (recursive) 392 - const { data: tree } = await this.withRetry( 393 - () => octokit.git.getTree({ 394 - owner: repoConfig.owner, 395 - repo: repoConfig.repo, 396 - tree_sha: currentTreeSha, 397 - recursive: "true", 398 - }), 399 - `getTree(${repoIdentifier})` 400 - ); 401 + // Supported documentation file extensions 402 + const DOC_EXTENSIONS = [".md", ".txt", ".rst", ".a +doc"]; 403 402 - // Supported documentation file extensions 403 - const DOC_EXTENSIONS = [".md", ".txt", ".rst", ".adoc" -]; 404 + // Filter to documentation files 405 + let docFiles = tree.tree 406 + .filter(item => 407 + item.type === "blob" && 408 + DOC_EXTENSIONS.some(ext => item.path?.toLo +werCase().endsWith(ext)) 409 + ) 410 + .map(item => ({ 411 + path: item.path!, 412 + sha: item.sha!, 413 + })); 414 405 - // Filter to documentation files 406 - let docFiles = tree.tree 407 - .filter(item => 408 - item.type === "blob" && 409 - DOC_EXTENSIONS.some(ext => item.path?.toLowerC -ase().endsWith(ext)) 410 - ) 411 - .map(item => ({ 412 - path: item.path!, 413 - sha: item.sha!, 414 - })); 415 + // Apply path filters 416 + if (repoConfig.paths && repoConfig.paths.length > +0) { 417 + docFiles = docFiles.filter(file => 418 + repoConfig.paths!.some(includePath => file +.path.startsWith(includePath)) 419 + ); 420 + } 421 416 - // Apply path filters 417 - if (repoConfig.paths && repoConfig.paths.length > 0) { 418 - docFiles = docFiles.filter(file => 419 - repoConfig.paths!.some(includePath => file.pat -h.startsWith(includePath)) 420 - ); 421 - } 422 + if (repoConfig.excludePaths && repoConfig.excludeP +aths.length > 0) { 423 + docFiles = docFiles.filter(file => 424 + !repoConfig.excludePaths!.some(excludePath + => file.path.startsWith(excludePath)) 425 + ); 426 + } 427 423 - if (repoConfig.excludePaths && repoConfig.excludePaths -.length > 0) { 424 - docFiles = docFiles.filter(file => 425 - !repoConfig.excludePaths!.some(excludePath => -file.path.startsWith(excludePath)) 426 - ); 427 - } 428 + Logger.info(`Found ${docFiles.length} documentatio +n files in ${repoIdentifier}`); 429 429 - Logger.info(`Found ${docFiles.length} documentation fi -les in ${repoIdentifier}`); 430 + // Build lookup maps for comparison 431 + const previousFilesMap = new Map(previousFiles.map +(f => [f.path, f])); 432 + const currentFilesMap = new Map(docFiles.map(f => +[f.path, f])); 433 431 - // Build lookup maps for comparison 432 - const previousFilesMap = new Map(previousFiles.map(f = -> [f.path, f])); 433 - const currentFilesMap = new Map(docFiles.map(f => [f.p -ath, f])); 434 - 435 - // Find new and changed files 436 - const filesToSync: GitHubFile[] = []; 437 - for (const file of docFiles) { 438 - const previous = previousFilesMap.get(file.path); 439 - if (!previous || previous.sha !== file.sha) { 440 - filesToSync.push(file); 434 + // Find new and changed files 435 + const filesToSync: GitHubFile[] = []; 436 + for (const file of docFiles) { 437 + const previous = previousFilesMap.get(file.pat +h); 438 + if (!previous || previous.sha !== file.sha) { 439 + filesToSync.push(file); 440 + } 441 } 442 - } 442 444 - // Find deleted files 445 - const filesToDelete: string[] = []; 446 - for (const [path] of previousFilesMap) { 447 - if (!currentFilesMap.has(path)) { 448 - filesToDelete.push(path); 443 + // Find deleted files 444 + const filesToDelete: string[] = []; 445 + for (const [path] of previousFilesMap) { 446 + if (!currentFilesMap.has(path)) { 447 + filesToDelete.push(path); 448 + } 449 } 450 - } 450 452 - Logger.info(`${repoIdentifier}: ${filesToSync.length} -files to sync, ${filesToDelete.length} files to delete`); 451 + Logger.info(`${repoIdentifier}: ${filesToSync.leng +th} files to sync, ${filesToDelete.length} files to delete`); 452 454 - // Sync changed files with concurrency limit 455 - const syncResults = await this.processWithConcurrency( 456 - filesToSync, 457 - async (file) => { 453 + // Sync changed files with concurrency limit 454 + const syncResults = await this.processWithConcurre +ncy( 455 + filesToSync, 456 + async (file) => { 457 + try { 458 + await this.syncFile(repoConfig, branch +, file); 459 + return { success: true, path: file.pat +h }; 460 + } catch (error: any) { 461 + const errorMsg = `Failed to sync ${rep +oIdentifier}/${file.path}: ${error?.message || String(error)}` +; 462 + Logger.error(errorMsg); 463 + return { success: false, path: file.pa +th, error: errorMsg }; 464 + } 465 + }, 466 + 3 // Concurrency limit of 3 467 + ); 468 + 469 + // Count successes and collect errors 470 + for (const syncResult of syncResults) { 471 + if (syncResult.success) { 472 + result.documentsSynced++; 473 + } else if (syncResult.error) { 474 + result.errors.push(syncResult.error); 475 + } 476 + } 477 + 478 + // Delete removed files (these are database operat +ions, not API calls, so we can be more aggressive) 479 + for (const path of filesToDelete) { 480 try { 459 - await this.syncFile(repoConfig, branch, fi -le); 460 - return { success: true, path: file.path }; 481 + const sourceId = `${repoIdentifier}/${path +}`; 482 + const deleted = await deleteDocument("gith +ub", sourceId); 483 + if (deleted) { 484 + result.documentsDeleted++; 485 + Logger.info(`Deleted document: ${sourc +eId}`); 486 + } 487 } catch (error: any) { 462 - const errorMsg = `Failed to sync ${repoIde -ntifier}/${file.path}: ${error?.message || String(error)}`; 488 + const errorMsg = `Failed to delete ${repoI +dentifier}/${path}: ${error?.message || String(error)}`; 489 Logger.error(errorMsg); 464 - return { success: false, path: file.path, -error: errorMsg }; 490 + result.errors.push(errorMsg); 491 } 466 - }, 467 - 3 // Concurrency limit of 3 468 - ); 469 - 470 - // Count successes and collect errors 471 - for (const syncResult of syncResults) { 472 - if (syncResult.success) { 473 - result.documentsSynced++; 474 - } else if (syncResult.error) { 475 - result.errors.push(syncResult.error); 492 } 477 - } 493 479 - // Delete removed files (these are database operations -, not API calls, so we can be more aggressive) 480 - for (const path of filesToDelete) { 481 - try { 482 - const sourceId = `${repoIdentifier}/${path}`; 483 - const deleted = await deleteDocument("github", - sourceId); 484 - if (deleted) { 485 - result.documentsDeleted++; 486 - Logger.info(`Deleted document: ${sourceId} -`); 487 - } 488 - } catch (error: any) { 489 - const errorMsg = `Failed to delete ${repoIdent -ifier}/${path}: ${error?.message || String(error)}`; 490 - Logger.error(errorMsg); 491 - result.errors.push(errorMsg); 492 - } 494 + // Update sync state for files 495 + await this.updateSyncState(repoIdentifier, current +TreeSha, docFiles); 496 } 497 495 - // Update sync state for files 496 - await this.updateSyncState(repoIdentifier, currentTree -Sha, docFiles); 497 - 498 // Sync recent commits if enabled 499 if (repoConfig.syncCommits) { 500 try { ● Fix 3: Remove the 404 suppression in the wiki sync catch block (doing this before Fix 2 since Fix 2 will handle the 404 case properly in syncWiki itself). ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 2 lines, removed 6 lines 513 const wikiPagesSynced = await this.syncWiki(re poConfig); 514 result.wikiPagesSynced = wikiPagesSynced; 515 } catch (error: any) { 516 - // Wiki might not exist or be disabled - this -is not necessarily an error 516 const errorMsg = `Failed to sync wiki for ${re poIdentifier}: ${error?.message || String(error)}`; 518 - Logger.warn(errorMsg); 519 - // Only add to errors if it's not a "wiki not -found" type error 520 - if (!error?.message?.includes("Not Found") && -error?.status !== 404) { 521 - result.errors.push(errorMsg); 522 - } 517 + Logger.error(errorMsg); 518 + result.errors.push(errorMsg); 519 } 520 } 521 ● Now Fix 2: Add the getWikiRef() helper, update syncWiki() to use it, and pass the branch through to syncWikiPage(). First, add the getWikiRef() helper method. I'll place it right before syncWiki(). ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 34 lines 921 } 922 923 /** 924 + * Resolve the wiki branch ref, trying "master" then "main +". 925 + * Returns the branch name and commit SHA, or null if no w +iki exists. 926 + */ 927 + private async getWikiRef( 928 + octokit: Octokit, 929 + owner: string, 930 + wikiRepoName: string, 931 + wikiIdentifier: string 932 + ): Promise<{ branch: string; commitSha: string } | null> { 933 + const branchesToTry = ["master", "main"]; 934 + 935 + for (const branch of branchesToTry) { 936 + try { 937 + const { data: ref } = await this.withRetry( 938 + () => octokit.git.getRef({ 939 + owner, 940 + repo: wikiRepoName, 941 + ref: `heads/${branch}`, 942 + }), 943 + `getRef(${wikiIdentifier}/${branch})` 944 + ); 945 + return { branch, commitSha: ref.object.sha }; 946 + } catch (error: any) { 947 + if (error?.status === 404) { 948 + continue; 949 + } 950 + throw error; 951 + } 952 + } 953 + 954 + return null; 955 + } 956 + 957 + /** 958 * Sync wiki pages for a repository. 959 * GitHub wikis are stored as separate git repositories at {owner}/{repo}.wiki 960 */ ● Now update syncWiki() to use getWikiRef() instead of the hardcoded try/catch block. ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 28 lines, removed 42 lines 969 const previousFiles = (syncState?.syncMetadata?.files as GitHubFile[]) || []; 970 const previousTreeSha = syncState?.lastSyncCursor; 971 972 - // Try to get the wiki repository's default branch 973 - // Wikis typically use "master" as their default bran -ch 974 - let currentTreeSha: string; 975 - let wikiTree: Awaited>["data"]; 972 + // Resolve wiki branch (tries "master" then "main") 973 + const wikiRef = await this.getWikiRef(octokit, repoCo +nfig.owner, wikiRepoName, wikiIdentifier); 974 + if (!wikiRef) { 975 + Logger.warn(`No wiki found for ${repoIdentifier} +(tried master and main branches)`); 976 + return 0; 977 + } 978 977 - try { 978 - // First, try to get the wiki repo reference 979 - const { data: ref } = await this.withRetry( 980 - () => octokit.git.getRef({ 981 - owner: repoConfig.owner, 982 - repo: wikiRepoName, 983 - ref: "heads/master", 984 - }), 985 - `getRef(${wikiIdentifier})` 986 - ); 987 - const commitSha = ref.object.sha; 979 + const wikiBranch = wikiRef.branch; 980 + const commitSha = wikiRef.commitSha; 981 989 - const { data: commit } = await this.withRetry( 990 - () => octokit.git.getCommit({ 991 - owner: repoConfig.owner, 992 - repo: wikiRepoName, 993 - commit_sha: commitSha, 994 - }), 995 - `getCommit(${wikiIdentifier})` 996 - ); 997 - currentTreeSha = commit.tree.sha; 982 + const { data: commit } = await this.withRetry( 983 + () => octokit.git.getCommit({ 984 + owner: repoConfig.owner, 985 + repo: wikiRepoName, 986 + commit_sha: commitSha, 987 + }), 988 + `getCommit(${wikiIdentifier})` 989 + ); 990 + const currentTreeSha = commit.tree.sha; 991 999 - // Get wiki tree 1000 - const { data: tree } = await this.withRetry( 1001 - () => octokit.git.getTree({ 1002 - owner: repoConfig.owner, 1003 - repo: wikiRepoName, 1004 - tree_sha: currentTreeSha, 1005 - recursive: "true", 1006 - }), 1007 - `getTree(${wikiIdentifier})` 1008 - ); 1009 - wikiTree = tree; 1010 - } catch (error: any) { 1011 - if (error?.status === 404) { 1012 - Logger.info(`No wiki found for ${repoIdentifi -er}`); 1013 - return 0; 1014 - } 1015 - throw error; 1016 - } 992 + // Get wiki tree 993 + const { data: tree } = await this.withRetry( 994 + () => octokit.git.getTree({ 995 + owner: repoConfig.owner, 996 + repo: wikiRepoName, 997 + tree_sha: currentTreeSha, 998 + recursive: "true", 999 + }), 1000 + `getTree(${wikiIdentifier})` 1001 + ); 1002 + const wikiTree = tree; 1003 1004 // Skip if tree SHA unchanged (no changes in wiki) 1005 if (previousTreeSha === currentTreeSha) { ● Now update the syncWikiPage call to pass wikiBranch, and update syncWikiPage to accept it. ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 1 line, removed 1 line 1050 filesToSync, 1051 async (file) => { 1052 try { 1053 - await this.syncWikiPage(repoConfig, file) -; 1053 + await this.syncWikiPage(repoConfig, file, + wikiBranch); 1054 return { success: true }; 1055 } catch (error: any) { 1056 Logger.error(`Failed to sync wiki page ${ wikiIdentifier}/${file.path}: ${error?.message || String(erro r)}`); ● Now update syncWikiPage signature to accept wikiBranch and use it instead of hardcoded "master". ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 2 lines, removed 1 line 1091 */ 1092 private async syncWikiPage( 1093 repoConfig: GitHubRepoConfig, 1094 - file: GitHubFile 1094 + file: GitHubFile, 1095 + wikiBranch: string 1096 ): Promise { 1097 const octokit = await this.ensureAuthenticated(); 1098 const repoIdentifier = `${repoConfig.owner}/${repoCon fig.repo}`; ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 1 line, removed 1 line 1106 owner: repoConfig.owner, 1107 repo: wikiRepoName, 1108 path: file.path, 1109 - ref: "master", 1109 + ref: wikiBranch, 1110 }), 1111 `getContent(${sourceId})` 1112 ); ● Now update updateWikiSyncState to store the resolved branch in metadata. ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 1 line, removed 1 line 1080 } 1081 1082 // Update sync state for wiki 1083 - await this.updateWikiSyncState(repoIdentifier, curren -tTreeSha, wikiFiles); 1083 + await this.updateWikiSyncState(repoIdentifier, curren +tTreeSha, wikiFiles, wikiBranch); 1084 1085 Logger.info(`Synced ${pagesSynced} wiki pages for ${r epoIdentifier}`); 1086 return pagesSynced; ● Update(apps\kb\src\ingestion\sources\github.ts) ⎿  Added 3 lines, removed 2 lines 1179 private async updateWikiSyncState( 1180 repoIdentifier: string, 1181 treeSha: string, 1182 - files: GitHubFile[] 1182 + files: GitHubFile[], 1183 + branch: string 1184 ): Promise { 1185 await query( 1186 `INSERT INTO kb_sync_state (source_type, source_i dentifier, last_sync_at, last_sync_cursor, sync_metadata) ... 1189 last_sync_at = NOW(), 1190 last_sync_cursor = EXCLUDED.last_sync_cursor , 1191 sync_metadata = EXCLUDED.sync_metadata`, 1191 - ["github", `${repoIdentifier}/wiki`, treeSha, JSO -N.stringify({ files })] 1192 + ["github", `${repoIdentifier}/wiki`, treeSha, JSO +N.stringify({ files, branch })] 1193 ); 1194 } 1195 ● Let me verify the final state of the file to make sure everything looks correct. ● Read 1 file (ctrl+o to expand) ● Everything looks correct. Let me also verify the updateWikiSyncState at the bottom. ● Read 1 file (ctrl+o to expand) ● All three fixes are implemented. Let me verify TypeScript compiles correctly. ● Bash(cd /c/Bigscreen/cloud && npx tsc --noEmit --project apps/kb/tsconfig.json 2>&1 | head -30) ⎿  apps/kb/src/retrieval/rerank.ts:52:21 - error TS2339: Property 'result s' does not exist on type 'unknown'. … +6 lines (ctrl+o to expand) ● The only error is a pre-existing one in a completely different file (rerank.ts), not related to our changes. Our changes compile cleanly. Here's a summary of the three fixes applied to apps/kb/src/ingestion/sources/github.ts: Fix 1 — Core bug (lines 385-496): Replaced the early return result when the main repo tree SHA is unchanged with an if/else that only skips document sync. Wiki sync and commit sync now always execute since they track their own sync state independently. Fix 2 — Wiki branch fallback (new getWikiRef() + plumbing): - Added getWikiRef() helper that tries heads/master then heads/main, returning null if neither exists (no wiki) - syncWiki() now uses getWikiRef() instead of hardcoding heads/master — returns 0 with a warn log if no wiki found - syncWikiPage() now accepts a wikiBranch parameter instead of hardcoding "master" - updateWikiSyncState() now stores the resolved branch in metadata Fix 3 — Error visibility (lines 515-519): Removed the 404 suppression in the syncRepo catch block for wiki sync. Since syncWiki() now handles the "no wiki" case internally (via getWikiRef returning null), any exception that reaches the catch block is a genuine error and should always surface in result.errors. ✻ Brewed for 2m 21s