fix: support nested tags in structured compression regex (#201)

[ousamabenyounes]

What does this PR do?

Fixes #201

The regex pattern used in segment_structured_context to parse <llmlingua>…</llmlingua> blocks used [^<]+ for the content group. When a block contained an inner HTML/XML tag (e.g. <tag>nested</tag>), the < from the inner tag terminated the match early, silently dropping that segment and all subsequent ones.

Before (broken):

pattern = r"...(  [^<]+  )</llmlingua>"

# Input with <tag>…</tag> inside a block → last segment silently dropped
[('', 'False', '', '', 'Speaker 4:'), ..., ('', 'False', '', '', '\nSpeaker 4:')]  # missing last!

After (fixed):

pattern = r"...(  (?:[^<]*(?:<(?!/llmlingua>)[^>]*>)?)*?  )</llmlingua>"

# All 6 segments captured, inner tags treated as plain text
[('', 'False', '', '', 'Speaker 4:'), ..., ('0.6', '', '', '', ' We have <tag>…</tag> here.')]

The new alternation (?:[^<]*(?:<(?!/llmlingua>)[^>]*>)?)*? reads:

• zero or more non-< characters, then optionally an inner tag that is not </llmlingua>
• non-greedy so it still stops at the first </llmlingua>

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Was this discussed/approved via a Github issue? Please add a link to it if that's the case.
  Issue: [Bug]: regex pattern does not handle nested tags in prompt (structured compression) #201
• Did you make sure to update the documentation with your changes? (not applicable — internal parsing fix)
• Did you write any new necessary tests?
  Added tests/test_nested_tag_regex.py with 4 unit tests: plain content regression, nested-tag fix, full multi-segment scenario, and rate-attribute preservation. Tests read the pattern directly from source to stay in sync with the code.

Who can review?

@iofu728 @QianhuiWu

Generated by Ora Studio
Vibe coded by ousamabenyounes

[ousamabenyounes]

@microsoft-github-policy-service agree