Skip to content

Add correlation-trap analysis (analyze_traps) with heuristics, plotting, and tests#342

Open
charlesmartin14 wants to merge 7 commits intomasterfrom
codex/add-analyze_traps-method-to-weightwatcher
Open

Add correlation-trap analysis (analyze_traps) with heuristics, plotting, and tests#342
charlesmartin14 wants to merge 7 commits intomasterfrom
codex/add-analyze_traps-method-to-weightwatcher

Conversation

@charlesmartin14
Copy link
Copy Markdown
Member

Motivation

  • Provide a dedicated workflow to detect and inspect randomized/permuted "correlation traps" in layer weight matrices using MP-fit and rank-1 isolation.
  • Expose a reproducible API to analyze traps layer-by-layer, evaluate trap vectors in original basis, and classify trap severity with a simple heuristic.

Description

  • Implemented analyze_traps on WeightWatcher along with helpers: apply_analyze_traps, apply_trap_mp_fit, identify_trap_mode_indices, compute_original_basis_for_traps, analyze_single_trap, assess_trap_diffuseness, plot_trap_analysis, _trap_vector_metrics, and _trap_vector_order_invariant_stats to perform permutation, MP bulk-fitting, trap isolation, unpermutation, metric computation, and plotting.
  • Added trap-specific metrics to the returned row dictionary (e.g., sigma_perm, eval_perm, mp_bulk_max, various u_/v_ metrics, trap_diffuseness_score, trap_risk_score, and trap_assessment) and ensured details ordering is consistent.
  • Added README example usage for trap analysis showing watcher.analyze_traps(...) and image saving.
  • Added unit tests: tests/test_analyze_traps.py to validate API, dataframe columns, reproducibility and vector/order-invariant statistics, and tests/test_trap_diffuseness.py to validate assess_trap_diffuseness behavior and outputs.

Testing

  • Added tests/test_analyze_traps.py and tests/test_trap_diffuseness.py and executed them with pytest -q against the modified codebase; tests passed where dependencies were available.
  • Torch-dependent tests in tests/test_analyze_traps.py are guarded with @unittest.skipUnless(TORCH_AVAILABLE, ...) and will be skipped automatically when torch is not installed.
  • The new unit tests exercise metric computation, ordering, reproducibility with fixed seeds, and the diffuseness/risk heuristics and reported results during the run.

Codex Task

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant