Skip to content

feat(medcat): CU-869cw9zmj Improve inference speed#410

Open
mart-r wants to merge 7 commits intomainfrom
feat/medcat/CU-869cw9zmj-improve-inference-speed
Open

feat(medcat): CU-869cw9zmj Improve inference speed#410
mart-r wants to merge 7 commits intomainfrom
feat/medcat/CU-869cw9zmj-improve-inference-speed

Conversation

@mart-r
Copy link
Copy Markdown
Collaborator

@mart-r mart-r commented Apr 13, 2026

This PR improves inference speed somewhat (around 10%) by:

  • Using a more efficient unitvec calculation
    • This gave a nice boost to speed (around 20% on this segment)
    • But it was doing the change in place so that polluted other bits which broke a bunch of stuff
    • However, this speedup came from a very small section of the overall time
    • So the overall speedup of this would have been closer to 1%
  • Reusing smaller context vectors mutiple times
    • We've cont multiple context windows (small, medium, large, extra larget)
    • So far we've been recalculating the vectors for all of these for every context window
    • However, the smaller ones will always need to be a subset of the bigger ones
    • So this change will take advantage of that

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant