Q Labs

← All posts

Following 47 GiB of missing RAM

The long version: how the leak was isolated, what was tried and failed, how the repair was guarded, and what the evidence does and does not show.

  • Case date: October 6, 2026, America/La_Paz (UTC−04:00).
  • Machine: Windows 11 Pro 25H2, build 26200.9457, with 61.56 GiB of usable RAM. Codex desktop package 26.930.7945.0.
  • Status: immediate repair verified, Windows restarted, extended monitoring still in progress at the fixed checkpoint of 22:31:15.

The result

A Windows computer with 61.56 GiB of usable RAM had only 3.21 GiB available. The largest visible applications did not explain the pressure. System measurements showed that approximately 45.33 GiB was in the nonpaged kernel pool. A single NTFS allocation tag, NtFC, dominated that pool and continued growing.

The investigation narrowed from system memory, to a specific kernel allocation tag, to a process, to two browser-cache paths, and finally to a cross-volume directory junction interacting with the app's packaged filesystem view.

A guarded repair replaced the public Codex Roaming junction with a normal directory and rebuilt the implicated caches while preserving the existing data. Before that intervention, the measured tag grew at 119.24 MiB per minute over an 8.5-minute window. After Codex reopened, it grew by 384 bytes total over 135.41 seconds. It was flat in the later steady measurement window.

The repair stopped the observed growth. A subsequent Windows restart reclaimed the accumulated allocation: the tag fell from approximately 46.886 GiB to approximately 27 KiB at the first post-reboot check. Longer monitoring was started to look for recurrence.

This is evidence for an effective local repair on this machine. The investigation did not identify a particular defective line in Windows or Codex source code, and the junction change and cache rebuild were performed together.

Part 1Start with the memory category

The user's opening question was straightforward: “what is taking all my ram”.

The first useful task was to distinguish several measurements that are often confused:

Start with the memory category
MeasurementWhat it tells usWhy it mattered here
Available physical memoryRAM Windows can make available for useConfirmed real memory pressure
Process working setA process's currently resident pages, potentially including shared pagesHelped compare visible apps with the scale of the problem
Private committed memoryPrivate allocations backed by the system's commit capacityCould be much larger than a process's resident RAM
Nonpaged kernel poolKernel allocations that must remain residentAccounted for most of the missing RAM
Pool-tag bytesOutstanding pool allocation associated with a particular tagSupplied a more specific quantity to track

At approximately 20:54, the system had 58.35 GiB used and 3.21 GiB available. The nonpaged pool accounted for approximately 45.33 GiB.

At that point, Claude's aggregate working set was about 1.07 GiB, and the Codex app's was about 0.97 GiB. There were also many Node processes: their combined private commit was much larger than their working set. Those were useful observations, but neither the process count nor a large commit figure explained the dominant resident kernel allocation.

Working sets were not added up and presented as a precise total of unique RAM: shared pages can appear in multiple processes. The mismatch in scale guided the next measurement.

Decision: investigate kernel pool allocation before attempting to reduce ordinary app memory.

First measurement problem: the shell could not see the whole system

The initial restricted shell saw only a narrow process view, and system-management queries were denied. Treating those results as complete would have produced a false diagnosis.

The assistant reran the necessary read-only queries with broader host access. This corrected the measurement environment; it did not change Windows memory settings.

Later, a separate privilege issue appeared: being allowed to query the host did not automatically grant an elevated Windows administrator token. Kernel tracing required an explicit UAC elevation. These were different boundaries and were handled separately.

Part 2Turn “kernel memory” into a measurable suspect

An existing local PowerShell helper, Read-PoolTags.ps1, exposed Windows pool-tag statistics through a small C# native interop wrapper. This avoided installing a large diagnostic toolkit during severe memory pressure.

The helper queried NtQuerySystemInformation with information class 22 and read each tag's paged and nonpaged allocation totals. On the examined x64 environment, it used an 8-byte header and 40-byte entries. It checked table bounds, limited buffer-growth retries, and freed its unmanaged buffer in a finally block.

This was a machine-specific helper using a native interface. The general technique, tracking allocation bytes by pool tag over time, is also documented in Microsoft's PoolMon examples.

The standout entry was:

Turn “kernel memory” into a measurable suspect
Pool tagApproximate nonpaged allocation in the early inspection
NtFC41.81 GiB
NDbf0.87 GiB
NDft0.25 GiB

Microsoft's sample tag mapping associates NtFC with ntfs.sys and Create.c. That established a relevant filesystem allocation area; it did not identify which application or configuration was causing the growth. Microsoft's tag-mapping example.

A small parsing correction that mattered

PowerShell's ordinary -eq string comparison is case-insensitive. An early filter therefore also matched differently capitalized tags. The filter was corrected to:

$tag = [StabilityPoolReader]::Read() |
    Where-Object { $_.Tag -ceq 'NtFC' }

NtFC, Ntfc, and NtfC must not be collapsed into one result. The final measurements use the exact case-sensitive tag.

Measure growth, not just size

A large snapshot shows accumulated usage. It cannot, by itself, tell whether that usage is still accumulating.

The repeated measurement was:

growth MiB/min =
    (ending NtFC bytes − starting NtFC bytes)
    ÷ 1,048,576
    × 60
    ÷ elapsed seconds

This became the main outcome for the experiments. It also avoided a misunderstanding later: the tag could remain enormous after the trigger stopped.

Part 3Test plausible explanations and keep the negative results

Reports of similar symptoms involving other AI desktop software helped form initial hypotheses. They were leads, not proof about this computer.

The user closed Claude Desktop. The assistant verified that its processes were gone, noticed that the Cowork service was still running, and measured NtFC again. The user then stopped that service, and the assistant repeated the measurement.

Test plausible explanations and keep the negative results
InterventionObservation windowIncrease in `NtFC`Measured rateWhat the result supported
Claude Desktop closed40.08 s101.09 MiB151.33 MiB/minClosing the desktop app did not halt the ongoing leak
Claude closed and Cowork service stopped40.05 s127.01 MiB190.26 MiB/minThe stopped service was not required for the ongoing growth
Codex restarted, before repair40.06 s76.008 MiB113.849 MiB/minRestarting Codex alone did not remove the triggering condition

These were short observation windows under a changing workload. The different rates should not be used to claim that closing an app made the leak worse. The useful finding was that substantial growth continued.

Nor did the tests prove that Claude or Cowork could never have contributed earlier. They narrowed the explanation for growth happening during the investigation.

Attribute the process correctly

A new clue appeared in Codex's process activity: its main and Chromium network utility processes showed unusually high IOOtherOperationsPersec values, roughly 150,000 to 200,000 in observed readings.

This counter was a reason to inspect activity more closely. It was not treated as a count of bytes written or as proof of filesystem writes.

The executable was named ChatGPT.exe, but its installed package path identified it as Codex. The process name alone would have been ambiguous. Package path, process ID, and utility-process command line supplied the attribution.

Part 4A failed recorder exposed a flaw in the workflow

Testing whether closing Codex stopped the leak required a recorder that could continue while Codex was closed.

The assistant initially launched a hidden recorder using Start-Process. That was a mistake: the child process did not survive Codex's shutdown in this environment.

It wrote 13 samples, from approximately 21:06:02 to 21:08:03, then stopped. It did not produce a valid interval with Codex fully closed.

The correct interpretation was missing evidence. The absence of later samples could not be reported as zero growth.

The user pointed out that tools dependent on Codex's process lifetime had caused problems across multiple projects. At the user's request, the assistant added a persistent rule to its main instructions. In substance, it requires:

Tools that must continue after Codex closes need an independent launcher or supervisor, durable output, a defined stop mechanism, and a bounded duration where appropriate. Continued operation must be verified across an actual close/restart or equivalent parent teardown.
From the report

The investigation then moved the relevant work to Windows Task Scheduler. This improved two things at once:

  1. The recorder and repair worker could outlive the application being tested.
  2. The diagnostic worker could run outside Codex's packaged process context.

Fresh output after the launching shell exited provided an initial check. The later repair supplied the stronger check: the worker recorded the actual Codex-closed interval, applied the repair, and continued after Codex reopened.

The first failure belongs in this case study because it changed the quality of the evidence and produced a reusable operating rule.

Part 5Capture a small trace to identify the activity

The next question was specific: which process was repeatedly opening which filesystem paths?

The assistant created a short Event Tracing for Windows capture using Microsoft-Windows-Kernel-File. The trace selected filename and create/open-related events. It ran for approximately two seconds, used a circular output cap of 64 MiB, and had explicit cleanup.

The command shape was:

logman create trace <unique-session> -ets `
    -o <evidence-directory>\file-activity.etl `
    -p Microsoft-Windows-Kernel-File 0x90 4 `
    -f bincirc -max 64 -bs 64 -nb 16 32

The script stopped only its uniquely named session in a finally block and recorded the start/stop result. These options correspond to direct trace-session control, provider selection, circular output, and bounded buffers in Microsoft's logman reference.

The first attempt lacked the necessary Windows elevation. An elevated helper subsequently completed the capture. The resulting ETL file was approximately 30.34 MB, within the configured output limit.

What the trace showed

The saved analysis counted 78,064 create/open events across 2.0428 seconds.

What the trace showed
AttributionEventsShare of recorded create/open events
Codex network utility process77,35699.09%
All other processes combined7080.91%

The dominant process had the utility subtype network.mojom.NetworkService. Its 77,356 events split equally between two Codex browser-cache directories: 38,678 at each path.

Using anonymized placeholders:

Public view:
<UserProfile>\AppData\Roaming\Codex\
  web\Codex\Default\Cache\Cache_Data

Package-private view:
<UserProfile>\AppData\Local\Packages\<CodexPackage>\
  LocalCache\Roaming\Codex\
  web\Codex\Default\Cache\Cache_Data

This was the decisive narrowing step. “Something in the filesystem” became one app's utility process repeatedly touching one kind of cache through two path views.

What the trace did not show

These counts describe events, not distinct files successfully created. They do not show 77,356 new files, 77,356 writes, or a volume of transferred data.

The capture did not include allocation stacks needed to tie every retained pool allocation to a precise call chain. It also did not establish completion status for each open. The equal path counts were consistent with a redirection interaction, but they were not enough to prove its internal sequence.

The analysis aggregated events by process and directory while streaming the trace, instead of keeping all decoded event objects in memory. A parser failure on an unnamed event was corrected by handling missing filenames explicitly.

Part 6The same pathname gave different answers in different contexts

The implicated cache produced an unusual combination of observations:

  • Known files such as index and data_0 through data_3 could be inspected by exact name.
  • Enumerating the cache directory from a Codex-launched shell failed.
  • The error referred to a mount point resolving to a volume not attached to the specified device object.
  • A basic inspection inside that context did not reliably reveal the underlying junction.

The saved error was:

The create operation failed because the name contained at least
one mount point which resolves to a volume to which the specified
device object is not attached.

This shifted the investigation toward path resolution, rather than merely cache size or file permissions.

For virtualized packaged desktop apps, Windows can merge public and per-app AppData views. Exact behavior depends on the app's packaging and virtualization configuration. That made the querying process's context relevant. Microsoft's description of packaged desktop AppData behavior.

Inspect outside the package, not merely with more privilege

A Task Scheduler worker checked that it had no package identity using GetCurrentPackageFullName. It then opened directory handles and used GetFinalPathNameByHandle to inspect their resolved destinations.

The result exposed the physical layout:

Inspect outside the package, not merely with more privilege
PathIndependently observed layout
Public Codex Roaming directory on C:A junction targeting a relocated directory on E:
Relocated public browser profile on E:The preserved source data
Codex package-private profileA separate real directory on C:

Directory enumeration worked in the independent context. The discrepancy itself was evidence that the earlier shell view was incomplete.

The working explanation became:

  1. Codex browser-cache activity.
  2. Public and package-private path views.
  3. The public Roaming junction crosses from C: to E:.
  4. Path failures and extreme repeated open activity.
  5. Strongly implicated by the trace and the intervention, not proven: a growing NTFS NtFC allocation.
  6. Restoring a normal public directory and rebuilding the caches was followed by the observed growth stopping after reopen.

The dashed connection is deliberate. The combined observations and intervention strongly implicated this interaction, while the exact internal defect remained unproven.

Part 7Make the repair narrow, reversible, and inspectable

The identified paths contained browser-profile state beyond disposable cache files. A broad profile reset would have had a larger data impact than the evidence justified.

The repair therefore preserved the original E: data, staged a verified copy on C:, and renamed the implicated caches so Codex could recreate them. Projects and the separate Codex home directory remained on E:.

Preconditions checked by the repair script

Before applying changes, the worker required:

  1. Execution outside the Codex package context.
  2. The public path to be the expected junction, resolving to the expected source.
  3. The private profile to resolve to its expected physical directory.
  4. No nested reparse points in the source tree.
  5. Staging and junction-backup names confined to the intended parent directory.
  6. No pre-existing staging or backup target.
  7. A source size no greater than 2 GiB and enough free space for the copy plus 512 MiB.
  8. Codex to be fully closed before the repair and still closed at critical change points.

A planning run validated the configuration before the applying run. The applying worker waited up to ten minutes for the user to close Codex and supported cancellation through a STOP marker.

These checks encoded the actual proposed operation. They reduced reliance on the assistant remembering every condition while executing separate commands.

Preserve and verify before replacing the path

The closed-app source contained 3,277 files totaling 244,959,688 bytes, approximately 233.61 MiB. This total comes from the final file manifest; the earlier live planning estimate differed slightly because it preceded app closure.

The copy used robocopy with recursive inclusion, file data/attribute/timestamp preservation, junction exclusion, and bounded retries. It did not use a mirroring deletion option.

Three manifests were compared:

Original source before copy
          ↕ same relative paths, lengths, SHA-256 hashes
Staged copy on C:
          ↕ compared with original manifest
Original source after copy

Comparing source-before with source-after also checked whether the source changed while the copy was being made. All 3,277 copied files passed these comparisons.

This verified file content preservation in the copied tree. It was not presented as an exhaustive test of every application feature.

Apply the supported configuration change

With Codex still closed, the worker:

  1. Renamed Cache_Data in the staged public tree and active private profile to timestamped backup names.
  2. Renamed the original public junction to a sibling backup name, retaining its original E: target.
  3. Renamed the staged real directory into the public Codex path.
  4. Verified that the replacement resolved to C: and was no longer a reparse point.
  5. Reopened Codex through its registered application identity.
  6. Continued collecting NtFC measurements.

The original E: profile and renamed caches were retained. Other private-profile data was not reset.

The script included conditional rollback of renames for a failure before the applied state, while the app remained closed. It did not promise that every possible failure could be undone automatically after the app resumed writing new state.

Experimental limitation: this was a combined path-and-cache intervention. It did not test replacing the junction and rebuilding the cache in separate experiments. The evidence supports the combined repair; it cannot apportion the benefit between its two parts.

Part 8Verify with the same measurement used to diagnose

The independent worker recorded the app before closure, while fully closed, and after the repair and reopening.

Verify with the same measurement used to diagnose
PhaseSamplesElapsed timeChange in `NtFC`Average net growth
Before exit, original configuration98511.004 s+1,064,851,760 bytes119.238484 MiB/min
Codex fully closed, before repair520.793 s0 bytes0 MiB/min
Reopened with repaired configuration25135.410 s+384 bytes0.000162 MiB/min

The final steady portion of the reopened window showed no net growth. The earlier cache-enumeration failure was also gone.

The script's automatic success criterion was weaker than “zero forever”: Codex had to remain present during the steady sample window, and the measured rate had to be below 5 MiB/min. The actual observed result was substantially below that threshold. The saved RepairVerified state should be read together with these measurements.

At approximately 21:37, the temporary repair task had removed itself successfully. It had survived the actual Codex exit and reopen, so its independent lifetime was demonstrated rather than assumed.

Stopping growth and reclaiming memory were separate outcomes

Approximately 46.886 GiB remained allocated under NtFC after the repair. Total nonpaged pool was still approximately 50.65 GiB.

That did not contradict the stopped growth. The intervention had removed the observed trigger, but the already-retained allocation had not been released.

The user restarted Windows. The new boot time was approximately 22:18:11.

At 22:25:44:

Stopping growth and reclaiming memory were separate outcomes
MetricPost-reboot reading
NtFC allocationApproximately 27 KiB
Total nonpaged pool2.447 GiB
Available RAM25.30 GiB
Used RAM36.26 GiB

The before/after NtFC readings show the accumulated allocation had been reclaimed. The change in available RAM should not be described as an exact count of memory “saved by the fix,” because other applications and workloads changed across the reboot.

Part 9Continue observing with a bounded, independent monitor

The user requested monitoring after the restart.

A Windows scheduled task began a 24-hour recording window at approximately 22:28 on October 6, ending at approximately 22:28 on October 7, local time. It sampled every 60 seconds and wrote durable logs.

The recorder tracks:

  • NtFC bytes and recent growth.
  • Total nonpaged pool.
  • Available physical RAM and system commit pressure.
  • A small list of the largest nonpaged tags.
  • Start, stop, gap, and collection-error events.

The implementation keeps a bounded recent sample history and a 32 MiB log cap. It supports a STOP marker, removes its task when finished, and can resume at the user's next sign-in if Windows restarts within the recording window.

The alert thresholds are investigation thresholds:

Continue observing with a bounded, independent monitor
ConditionTrigger
Large NtFC allocationAt least 1 GiB
Sustained NtFC growthAt least 256 MiB allocated, at least 64 MiB growth, and at least 10 MiB/min over a window of at least 150 seconds
Large total nonpaged poolAt least 6 GiB
Memory pressureAvailable RAM below 4 GiB or commit at least 90%, for three consecutive samples
Collection failureThree consecutive failed samples

A separate Codex follow-up checks the logs every 15 minutes and is instructed to notify only on meaningful changes, failures, recovery, completion, or required user action.

The recorder is independent of Codex. The AI review and notifications still depend on Codex being available. Neither component can record measurements while Windows is off or asleep.

At the fixed reporting checkpoint, 22:31:15 on October 6, the monitor had four samples. NtFC was 53,504 bytes, total nonpaged pool 2.502 GiB, and available RAM 25.345 GiB. No alert condition was active.

The full 24-hour observation was not complete when this report was prepared. A quiet early window supports the immediate result; it does not establish permanent resolution.

Part 10What makes this useful as an example of AI assistance

The useful contribution was a sequence of concrete operations that made the problem narrower and the outcome easier to audit.

Evidence changed the direction of the work

The investigation started with several plausible suspects. Continued growth after Claude and Cowork stopped moved attention elsewhere. The filesystem trace then implicated the same Codex application hosting the assistant.

That willingness to follow inconvenient evidence is central to the case. A report blaming a more convenient application would have contradicted the measurements.

The assistant built tools for specific questions

The assistant built tools for specific questions
QuestionTool or technique
Which memory category is large?System counters and process measurements
Which allocation is growing?Pool-tag reader and timed samples
Which process and paths are active?Short kernel-file ETW trace
What do the paths really resolve to?Unpackaged scheduled inspection and native directory handles
Can the repair preserve existing data?Staged copy, hashes, path assertions, renamed backups
Does the result survive app reopening?Independent before/closed/after recorder
Does the problem recur?Bounded scheduled monitor and change-based review

These were adaptations to the observed problem, with saved outputs that could be checked later.

The human supplied context, authority, and corrections

The user reported the symptom, closed applications and stopped a service, authorized the repair, completed the actual app shutdown, restarted Windows, and requested follow-up monitoring.

The user also identified the recurring persistence problem. That feedback improved the implementation and became a standing rule for later projects.

This was collaborative debugging. The assistant did not independently know the user's tolerance for interruption or silently restart the machine.

Reliability included admitting failed experiments

The report retains failures that affected interpretation:

  • A restricted process view could not support a system-wide conclusion.
  • A case-insensitive filter was unsuitable for pool tags.
  • A hidden child recorder died with the application.
  • Kernel tracing required a different privilege level.
  • An event parser needed to handle missing filenames.
  • A packaged shell gave a misleading view of a junction.

Each failure prompted a specific correction. None was relabeled as a successful test.

The outcome is inspectable, with limits

The strongest supported conclusion is:

On this machine, extreme Codex browser-cache activity coincided with a growing NTFS pool allocation and a broken cross-volume AppData path configuration. After a guarded path-and-cache repair, observed growth stopped with Codex reopened. A Windows restart reclaimed the retained allocation, and independent monitoring began.
From the report

The evidence does not establish:

  • A universal Codex or Windows defect affecting every installation.
  • The exact allocator call chain or ownership of a particular code bug.
  • That removing the junction alone, or rebuilding the cache alone, would have been sufficient.
  • That no other software contributed to memory use earlier.
  • A completed 24-hour stability result.
  • A measured productivity advantage over a human expert; there was no comparison study.

The reproducible lesson is the investigation method: select a discriminating measurement, test a specific explanation, retain contradictory evidence, change only the implicated configuration, and verify using the original outcome metric.

Technical appendix: timing and evidence

Timeline

All times are October 6, 2026, America/La_Paz (UTC−04:00). Approximate times are marked “~”.

Timeline
Local timeMilestone
~20:54Severe pressure confirmed; nonpaged pool identified as dominant
~20:55NtFC isolated as the largest tag
21:02 to 21:04Growth continues after Claude closure and Cowork service stop
21:06 to 21:08First recorder captures pre-close data, then dies with Codex
21:10 to 21:11Post-restart test still shows substantial growth
~21:13Two-second elevated filesystem trace
21:20Independent inspection reveals the physical junction layout
21:25Guarded repair worker starts and records the pre-close baseline
21:34:01Codex fully closed; tag stays flat in the closed measurement window
~21:34:41Preservation/cutover complete
~21:34:59Codex reopens
21:35 to 21:37Reopened app observed; only 384 bytes added
21:37:20Verification saved; temporary repair task removed
22:18:11Windows boot after user-controlled restart
22:25:44Small post-reboot tag allocation confirmed
22:28Independent 24-hour monitor starts
22:31:15Fixed checkpoint used in this report; no active alert

Saved evidence and provenance

The private evidence remains in the local investigation directory. Filenames below are identifiers, not invitations to publish their raw contents.

Saved evidence and provenance
EvidenceWhat it supportsPublication handling
FINDINGS-2026-10-06.mdChronology, early readings, closure experiments, trace findingsPersonal paths removed in this article
capture-20261006-210601.csvInitial recorder's limited coverageIts missing closed interval is explicitly acknowledged
trace-20261006-2113-admin/analysis.jsonEvent totals and process/path concentrationOnly relevant aggregate counts exported
trace-20261006-2113-admin/file-activity.etlOriginal captured filesystem eventsKept private; contains unrelated file activity
standalone-inspection-20261006/inspection.jsonPackage identity, junction layout, canonical pathsPaths replaced with placeholders
repair-20261006-2125/memory.csvBefore/closed/reopened allocation measurementsSanitized samples supplied separately
repair-20261006-2125/verification-summary.jsonPhase statistics and task cleanup statusWhitelisted fields exported
repair-20261006-2125/legacy-source-manifest.csvPreserved file count, lengths, and SHA-256 hashesRaw browser filenames and hashes kept private
repair-20261006-2125/copy.logCopy completion and repair scopeAggregate file count/size only
Monitor latest.json and samples.jsonlPost-reboot observationsFixed checkpoint exported; completion still pending

Early baseline figures come from the contemporaneous investigation record and session measurements. Repair-phase results can additionally be recomputed from the supplied sanitized CSV. They should not be represented as having identical provenance.

Measurement limitations

Rates use endpoint differences divided by elapsed wall-clock time; they are not regression estimates. Sampling was approximately every five seconds during the repair experiment, with real timestamps used for calculations. Time spent applying the repair was not filled with invented samples.

No full kernel allocation-stack trace was taken. No lost-event audit is included in the saved ETW summary, so 99.09% describes the create/open events counted in the saved trace, not a certified count of every filesystem operation on the machine.

The repair was not followed by deliberately reintroducing the bad configuration. Doing so would have risked recreating severe pressure. The confidence assessment rests on the path evidence, process concentration, closed-app result, and successful reopen after intervention.