261111_01_ENH: Kiwi 형태소 분석 정확도 개선 및 품사 합산·복합명사·치환·불용어 옵션 추가

This commit is contained in:
2026-11-11 10:37:13 +09:00
parent fb76c9dcb3
commit 6e1d57414e
13 changed files with 572 additions and 36 deletions

View File

@@ -112,7 +112,7 @@
- Follow existing question organization. Return N and percent together, including Total. Preserve repeated rank/matrix occurrences, original numeric missing/mean/statistic rules, scale/reverse/NPS, grouped values and TABULATION denominators. Frontend chooses PRTMODE presentation without requery.
- Keep original empty-base semantics: ordinary cell/Total percentages are 0; undefined group/statistic values are null with '-' display. Use 15-significant-digit C# double-format behavior and midpoint-away rounding, validated by legacy fixtures.
- StatisticsMapper uses survey sessions, parameterized complete/TEST/date conditions, original distinct derived/profile joins and selected storage-group projection. Filter data is private and separate from projected aggregation data. AST filters never inject user SQL.
- User approved the latest Kiwi model after finding the original DLL was 0.15.2. Use matching official KiwiJava/model 0.24.0 CONG instead of pinning SBG. Model-result changes are allowed; preserve Report NLP dictionary/tag scopes, word scores 10/50/100, tag/length/SL selection and stable top-100 frequency logic. Add no new stopword rules.
- User approved the latest Kiwi model after finding the original DLL was 0.15.2. Use matching official KiwiJava/model 0.24.0 CONG instead of pinning SBG. Model-result changes are allowed; preserve Report NLP dictionary/tag scopes, word scores 10/50/100, tag/length/SL selection and stable top-100 frequency logic. Stopwords exist only as the REPORT_NLP_WORD STOP rows described under Kiwi NLP accuracy; add no other stopword rules.
- Configure kri.kart.nlp.model-directory as ./nlp_model/v0.24.0, install model separately, ignore runtime models in Git, and choose the platform-specific official jar at build time. Do not claim Ubuntu execution from Windows tests. Missing native/model infrastructure is explicit 503, not empty successful NLP output.
- Per user decision, the legacy external-Referer condition is removed: every general statistics batch writes Get Report Data once through LogService. Quota results write QUOTA VIEW only, never Get Report Data. Do not add a separate logging endpoint.
- Validate through original C# synthetic calculation fixtures, authenticated API/Swagger tests, source-bound MyBatis tests and opt-in actual native analysis with -PkiwiModelDirectory. Actual SQL Server equivalence and deployment performance remain separate checks.
@@ -187,3 +187,14 @@
- Apply team/assignment before candidate TOP 50 and ordering, never after Java merge/TOP 6. Must-view does not override authorization. Linked list retains its business query team and adds separately named authenticated-team/role/assignment constraints.
- Derived management list: role 1001 sees all project definitions; others see matching GKEY only. Keep normal questions/report metadata unfiltered by this management rule.
- Scope is these read conditions only. Do not add generic per-project/mutation/owner authorization, change original aggregations/Excel, or weaken External scope. Validate actual bound mapper SQL and H2 results on both sources, empty assignments, precise id routing, trusted JWT/API Key context versus forged query fields; no shared DB execution.
## Kiwi NLP accuracy (user decisions 2026-11-10)
- The legacy C# report services are retired; REPORT_NLP_WORD/TAG are no longer read by them.
- KiwiNlpService builds with BuildOption.default_ and analyzes with Match.allWithNormalizing | oovChrModel plus basicTypoSetWithContinual typo correction (threshold 2.5). Do not revert to AnalyzeOption(0).
- User dictionary: register each (text, tag) once with its highest scope score, because Kiwi keeps the last duplicate score and the query has no order. Trim/uppercase tags. Log only registered/skipped/rejected counts, never words.
- Statistics options: wordMergeMode 1=ALL, 2=NOUN (default; NNG/NNP/NNB/NR/NP/SL/SH), 3=NONE sums the same text across tags, keeping the most frequent tag (first on ties) and first-appearance order. compoundNounMaxTokens 1-5 (default 3; 1 disables) joins noun prefixes/suffixes and merges NNG/NNP/SL runs attached within one word when they contain a Korean noun; NNP if any part is NNP; longer runs stay separate.
- Synonyms (user decision 2026-11-11, replacing the separate REPORT_NLP_SYNONYM table): a REPORT_NLP_WORD row with SOURCE_TEXT (NVARCHAR(128), nullable; WORD_TEXT is NVARCHAR(128)) replaces SOURCE_TEXT with WORD_TEXT/WORD_TAG. WORD_TEXT always means the result word. Scope priority question > project > ALL uses the 100/50/10 scores. The column must exist before deployment. Sources are pre-analyzed (score 200, above every user word) after registering the target as a word with score 0 unless a plain row for the same target already registered it with its scope score, because Kiwi rejects a target morpheme missing from its dictionary such as LG/NNP; a failing row is skipped, never a 503; a spaced source also covers its spaceless form and replaces whole token runs only when no noun of the same word continues the run.
- Stopwords (불용어): REPORT_NLP_WORD rows with WORD_TAG STOP (case-insensitive) are never registered in Kiwi. Any scope excludes the final word (after synonyms and compound merging) for every tag, comparing trimmed, space-collapsed text case-insensitively; SOURCE_TEXT is ignored on STOP rows.
- User dictionary words and synonym targets take precedence over everything: they are never merged with neighbors.
- The original analyze(project, qid, responses) and tokenize(texts, dictionary) overloads keep the previous behavior (synonym rows ignored; no stopwords or merging). Only the statistics API uses the new options.