Skip to content

fix(classifier): feature verb wins over debug keyword (part of #196)#289

Merged
iamtoruk merged 1 commit intomainfrom
fix/classifier-feature-debug-priority
May 10, 2026
Merged

fix(classifier): feature verb wins over debug keyword (part of #196)#289
iamtoruk merged 1 commit intomainfrom
fix/classifier-feature-debug-priority

Conversation

@iamtoruk
Copy link
Copy Markdown
Member

First of three PRs addressing the issues flagged in #196 and #159. Independent of the others; can land alone.

The bug

Messages like `add error handling`, `create an issue tracker`, or `implement the 404 page` were landing in the Debugging bucket because the classifier checked the debug regex before the feature regex. Real users from #196 saw their feature work mislabeled.

```ts
// Before: order of checks decided the result
if (DEBUG_KEYWORDS.test(userMessage)) return 'debugging' // wins on "add error handling"
if (REFACTOR_KEYWORDS.test(userMessage)) return 'refactoring'
if (FEATURE_KEYWORDS.test(userMessage)) return 'feature'
```

The fix

Pick whichever keyword pattern matches earliest in the user message. The position of the leading verb is a much stronger intent signal than the order of the checks in code.

```ts
// After: position decides
return firstMatchingCategory(userMessage, [
{ regex: REFACTOR_KEYWORDS, category: 'refactoring' },
{ regex: FEATURE_KEYWORDS, category: 'feature' },
{ regex: DEBUG_KEYWORDS, category: 'debugging' },
]) ?? 'coding'
```

Tie-break (same start index) goes to the candidate listed first. Ordering is `refactoring > feature > debugging` so existing behavior for plain bug reports (`login is broken, traceback below`) is preserved — DEBUG word leads in those cases.

Same logic applied to `classifyConversation` for chat-only turns.

Examples

Message Before After
`add error handling` debugging feature
`create an issue tracker` debugging feature
`implement the 404 page` debugging feature
`fix the layout for the new feature` debugging debugging (unchanged)
`login is broken, traceback below` debugging debugging (unchanged)
`refactor the error handling` refactoring refactoring (unchanged)

Tests

8 new regression tests under `describe('classifyTurn — feature vs debugging precedence (#196)')`. Full suite: 45 files / 609 tests, all pass.

Out of scope

Messages like "add error handling", "create an issue tracker", or
"implement the 404 page" were landing in the Debugging bucket
because the classifier checked DEBUG_KEYWORDS (which matches
`error`, `issue`, `404`) before FEATURE_KEYWORDS in both
`refineByKeywords` (tool-bearing turns) and `classifyConversation`
(chat-only turns). The position of the matched word in the
sentence is a much stronger intent signal than the order of the
checks in code, so we now pick whichever pattern matches earliest.

The new helper `firstMatchingCategory` runs each candidate regex
once with `RegExp.exec` and keeps the match with the lowest
`index`. Ties (rare in practice — same start position) break by
the order the candidates were listed, which is `refactoring >
feature > debugging` for coding turns. That ordering preserves
existing behavior for plain bug reports (e.g. "login is broken,
traceback below") while flipping mislabeled feature work to its
correct category.

8 regression tests in `tests/classifier.test.ts` cover the
mislabel cases from #196 plus tie-break / chat-only cases. Full
suite: 45 files / 609 tests, all green.

Closes the activity-misattribution half of #196. The Cursor
provider attribution half (single 'cursor' project for all
sessions) is addressed in a separate PR.
@iamtoruk iamtoruk merged commit 7a878f4 into main May 10, 2026
3 checks passed
@iamtoruk iamtoruk deleted the fix/classifier-feature-debug-priority branch May 10, 2026 05:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant