PayrunTest: subsequent runs timeout after tenant deletion — possible AssemblyCache not cleared

[gsayer]

Summary

When I run multiple PayrunTest cycles sequentially, the first 2 runs succeed but all subsequent runs timeout with Missing results for payrun job. Restarting the backend resolves it until the pattern repeats.

Steps to reproduce

1. Fresh backend container (v0.10.0-beta.3, SQL Server 2022, Docker on macOS arm64 → linux/amd64 emulation)
2. PayrunTest batch-1.pt.json /bulk /cleantest (20 employees) → ✅
3. PayrunTest batch-2.pt.json /bulk /cleantest (20 employees) → ✅
4. PayrunTest batch-3.pt.json /bulk /cleantest (20 employees) → ❌ timeout
5. All subsequent runs timeout regardless of batch size
6. POST /api/admin/application/clearcache between runs does NOT fix it
7. Restarting the backend container fixes it for 2 more runs

What I've ruled out

• AssemblyCache: Calling /api/admin/application/clearcache between runs does not help
• Batch size: Fails with 20, 50, or 100 employees
• MaxParallelEmployees tuning: Tested with 2 and default (11), same result
• SQL Server timeouts: No SQL errors in logs

What I observe

• The payrun job stays in Process status and never reaches Complete
• Backend logs show no explicit error for the stuck job
• It is always the 3rd run that fails, consistently
• Only a full container restart resolves it (not just cache clear)

Environment

• Backend: v0.10.0-beta.3
• PayrollConsole: v0.10.0-beta.3
• SQL Server 2022 (Docker, linux/amd64)
• macOS arm64 host (Rosetta emulation for all containers)
• Each test creates a fresh tenant with regulation, employees, cases, and payrun

Question

Is there a known limitation with running multiple PayrunTest /cleantest cycles in sequence? Could there be an internal state (beyond AssemblyCache) that accumulates and isn't fully reset when a tenant is deleted?

I'm using PayrunTest for batch payroll validation (~1000 employees split into groups). The first 200 employees always validate correctly (99.5% conformity), but I cannot complete the remaining batches without container restarts.

Thank you for any guidance.

[Giannoudis]

Hi Guillaume,

Thank you for the detailed report. I was unable to reproduce the timeout on Windows with a local backend process — all three sequential /cleantest runs completed successfully with the same batch structure (20 employees, minimal regulation, single WageType).

This strongly suggests the issue is environment-specific rather than a general backend bug.

Key difference: macOS arm64 + Rosetta 2 emulation

Your setup runs the backend container as linux/amd64 under Rosetta 2 emulation on macOS arm64. The .NET ThreadPool behaves differently in this configuration — specifically, thread injection (adding new threads when existing ones are blocked) is slower under emulation than on native x64. After two full /cleantest cycles, enough internal threads may be in a lingering state that the third payrun job is started but never picked up by a worker, causing the polling loop to time out.

This also explains why:

• POST /api/admin/application/clearcache has no effect (it clears the script assembly cache, not thread state)
• A container restart fixes it (the ThreadPool is fully reset)
• The failure is always at run 3, not random (deterministic resource pressure)

Suggested next steps to confirm:

1. Run the backend on native linux/amd64 (e.g., a Linux VM or an x64 Docker host) — if the issue disappears, Rosetta emulation is the root cause
2. Set MaxParallelEmployees: 1 in appsettings.json and retry the 3-batch sequence — this removes parallel thread pressure entirely and will help isolate whether thread pool exhaustion is the trigger

If the issue reproduces on native x64 Linux, please share the backend logs from the failing run (especially any entries around the payrun job status transition from Process → Complete) — that would point to a different underlying cause worth fixing in the backend.

Thanks again for the thorough report.