CALLAII
Language modelThe language model powering the AI surfaces in the Callmenta panel: call analysis, coaching, live support, CRM, and CALLAII Legal.
CALLAII's eighth version and new primary model. It scores calls based on the company's own criteria and writes summaries using only what was discussed. It prepares legal texts strictly adhering to library sources and procedure. Training was completed on October 15, 2026. CALLAII v8 passed the same exams as v7, results are below.
What improved in this version
- In call analysis, it identifies the request type with 81.8% accuracy across 40 real calls, up from 22.7% in v7
- It produces valid output in all 534 analyses
- Even with thinking mode off, it starts legal answers with a procedure check: 61 out of 70 questions, compared to 22 in v7
- Critical citation and quote errors in legal documents dropped from 8 to 4 across 70 documents
- Scored 90 on map questions and 95 on shape counting
- Route plan evaluation on energy screens improved from 20 to 93.3
Limitations and responsible use
- Accuracy in resolution status remains limited: 76 correct out of 123 questions in targeted calls. This area should remain as a suggestion on screen.
- The quality of legal documents was not scored by a judge in this round. Every document must undergo lawyer review.
- Legal knowledge does not come from the model's memory, sources are fetched from the current library each time.
- Numbers and dates must be verified.
Training and measurements
| Training data | 61,461 samples, 8,000 of which are visual |
|---|---|
| Validation set | 186 samples |
| Step | 3,842 · 1 epoch · 16 samples per step |
| Training context | 32,768 tokens |
| Training date | October 2026 |
| Data for each screen was generated by the live system's own prompts. The model is trained on the prompts it sees | |
| Petitions, formal notices, and legal chats draw sources from the library. Procedure checks rely solely on the provided articles | |
| Call scoring was taught using examples where the missing behavior was known in advance. In real calls, the median of three separate evaluations was used, and classification fields were corrected based on human labels | |
| Not sharing internal rules, speaking in the company's own name in every language, and not inventing unknown products were additionally taught | |
| Sales, cross-selling, and competency scenarios were verified against known truths | |
| Real calls reserved for exams were excluded from training |
Each point is the average of 7 consecutive records.
Show values as a table
| Training loss | |
|---|---|
| 35 | 0.7007 |
| 210 | 0.5514 |
| 385 | 0.5294 |
| 560 | 0.4948 |
| 735 | 0.5226 |
| 910 | 0.5209 |
| 1,085 | 0.4995 |
| 1,260 | 0.4866 |
| 1,435 | 0.4916 |
| 1,610 | 0.4829 |
| 1,785 | 0.4963 |
| 1,960 | 0.4516 |
| 2,135 | 0.4679 |
| 2,310 | 0.4778 |
| 2,485 | 0.4750 |
| 2,660 | 0.4592 |
| 2,835 | 0.4596 |
| 3,010 | 0.4480 |
| 3,185 | 0.4427 |
| 3,360 | 0.4400 |
| 3,535 | 0.4327 |
| 3,710 | 0.4643 |
| 3,840 | 0.4225 |
| Validation loss | |
| 100 | 0.4600 |
| 300 | 0.3948 |
| 500 | 0.3773 |
| 700 | 0.3684 |
| 900 | 0.3574 |
| 1,100 | 0.3421 |
| 1,300 | 0.3386 |
| 1,500 | 0.3330 |
| 1,700 | 0.3276 |
| 1,900 | 0.3223 |
| 2,100 | 0.3152 |
| 2,300 | 0.3126 |
| 2,500 | 0.3068 |
| 2,700 | 0.3018 |
| 2,900 | 0.2982 |
| 3,100 | 0.2957 |
| 3,300 | 0.2947 |
| 3,500 | 0.2932 |
| 3,700 | 0.2930 |
| 3,842 | 0.2929 |
Comparison with CALLAII v7
v8 and the previous live main model v7 entered the same exams under the same conditions. The measurement was conducted on the same exam server on October 15, 2026. Call analysis and thinking mode procedure exam were run three times, others once. Calls and prompts were identical for both models. Verification losses were measured with different sets, so they are not comparable between versions.
- Invalid output in call analysis dropped from 3 to 0 out of 534 analyses.
- Request type increased from 22.7% to 81.8% on 40 real calls with pre-determined correct answers. The biggest jump was in this area.
- Resolution status increased from 54.5% to 68.2% on the same calls. Scorable call separation increased from 84.2% to 94.2%, and risk level from 77.3% to 81.8%.
- Request type increased from 70.4% to 97.2% on 138 targeted calls. Scorable call separation increased from 64.7% to 78.4%, and risk level from 66.7% to 74.2%.
- Resolution status on targeted calls barely moved. It increased from 73 to 76 on 123 questions. This area remains the weakest.
- In legal procedure audit, both v7 and v8 started the answer with a procedure check on all 70 questions in thinking mode. In without thinking mode, this increased from 22 to 61.
- Citation and quote errors deemed critical by the audit dropped from 8 to 4 on 70 documents with live setting. In without thinking run, it dropped from 8 to 3. Cut documents dropped from 1 to 0. The sample is small, and document quality was not separately scored in this round.
- Score on map questions increased from 60 to 90, and shape counting from 80 to 95.
- Route plan evaluation on energy screens increased from 20 to 93.3, and field task preparation from 53.3 to 90. Manager route remained the same at 93.3.
Same calls and same prompt, sum of 3 runs. Analyses not producing valid output were counted as incorrect.
Show values as a table
| CALLAII v7 | CALLAII v8 | |
|---|---|---|
| Scorable | 84.20 | 94.20 |
| Risk level | 77.30 | 81.80 |
| Resolution status | 54.50 | 68.20 |
| Request type | 22.70 | 81.80 |
Same calls and same prompt, sum of 3 runs. Analyses not producing valid output were counted as incorrect.
Show values as a table
| CALLAII v7 | CALLAII v8 | |
|---|---|---|
| Scorable | 64.70 | 78.40 |
| Risk level | 66.70 | 74.20 |
| Resolution status | 59.30 | 61.80 |
| Request type | 70.40 | 97.20 |
In thinking mode, the median of 3 runs. Without thinking, one run.
Show values as a table
| CALLAII v7 | CALLAII v8 | |
|---|---|---|
| Start with procedure check, thinking mode | 100.00 | 100.00 |
| Procedure points, thinking mode | 85.70 | 88.60 |
| Start with procedure check, without thinking | 31.40 | 87.10 |
Lower is better. Compares audit decision number, law article, and quote with the actual text. Document quality was not additionally scored in this round.
Show values as a table
| CALLAII v7 | CALLAII v8 | |
|---|---|---|
| Critical error, live setting | 8.00 | 4.00 |
| Critical error, without thinking | 8.00 | 3.00 |
| Truncated document, live setting | 1.00 | 0.00 |
Definitive answer questions, one run. Map 100 questions, other image 18-point question, shape counting 60 questions.
Show values as a table
| CALLAII v7 | CALLAII v8 | |
|---|---|---|
| Map | 60.00 | 90.00 |
| Other view | 82.30 | 100.00 |
| Shape counting | 80.00 | 95.00 |
30 tasks per screen, 38 addresses, single run.
Show values as a table
| CALLAII v7 | CALLAII v8 | |
|---|---|---|
| Route plan evaluation | 20.00 | 93.30 |
| Field assistant | 96.70 | 100.00 |
| Field task preparation | 53.30 | 90.00 |
| Agent assistant | 80.00 | 96.70 |
| Manager route | 93.30 | 93.30 |
| Address parsing | 92.10 | 96.20 |
Previous versions
CALLAII v7 · Previous version
CALLAII v7 is live. The previous live version, v6, was put through the same exam under live settings. On legal questions, it checks procedure almost every time: competent and authorized court, conditions of the lawsuit, and deadlines. It works alongside an additional audit that catches and extracts information the lawyer did not mention in petitions.
What improved in this version
- Procedure check in legal mode: performed in 69 out of 70 questions; live v6 did not perform it in any of the same exam.
- Procedure check points in legal mode rose from 8 to 57 across 70 questions.
- Procedure check in general mode with thinking off rose from 0 to 23.
- Answers are more concise: answers cut off at the limit in general mode dropped from 40 to 11.
- Text extraction from document images and image description were fully preserved after training.
Limitations and responsible use
- Legal knowledge does not come from the model's memory; laws, regulations, and rulings are fetched from the current library every time.
- The text it generates is a draft; petitions, contracts, and official documents must be approved by an authorized person.
- Numbers and dates must be verified.
- Quality in languages other than Turkish has not been measured in as much detail as Turkish.
Training and measurements
| Training data | 45,847 samples |
|---|---|
| Validation set | 200 samples never seen during training |
| Steps | 2,859 · 1 epoch · 16 samples per step |
| Training context | 4,096 tokens |
| Validation loss | 0.629 → 0.513 |
| Exam | 70 procedure questions; same questions as live v6, live settings |
| Training date | September 2026 |
Each point is the average of 5 consecutive records.
Show values as a table
| Training loss | |
|---|---|
| 0.01 | 0.8421 |
| 0.05 | 0.6483 |
| 0.10 | 0.5975 |
| 0.14 | 0.5828 |
| 0.18 | 0.5622 |
| 0.23 | 0.5602 |
| 0.27 | 0.5509 |
| 0.31 | 0.5387 |
| 0.36 | 0.5504 |
| 0.40 | 0.5220 |
| 0.45 | 0.5358 |
| 0.49 | 0.5406 |
| 0.53 | 0.5296 |
| 0.58 | 0.5333 |
| 0.62 | 0.5053 |
| 0.66 | 0.5165 |
| 0.71 | 0.5090 |
| 0.75 | 0.5308 |
| 0.80 | 0.5334 |
| 0.84 | 0.5043 |
| 0.88 | 0.5118 |
| 0.93 | 0.5051 |
| 0.97 | 0.5022 |
| 1.00 | 0.4946 |
| Validation loss | |
| 0.07 | 0.6293 |
| 0.14 | 0.5990 |
| 0.21 | 0.5781 |
| 0.28 | 0.5642 |
| 0.35 | 0.5544 |
| 0.42 | 0.5433 |
| 0.49 | 0.5368 |
| 0.56 | 0.5298 |
| 0.63 | 0.5238 |
| 0.70 | 0.5194 |
| 0.77 | 0.5163 |
| 0.84 | 0.5139 |
| 0.91 | 0.5129 |
| 0.98 | 0.5127 |
| 1 | 0.5126 |
Same 70 legal questions, live settings. Thinking on in legal mode, off in general mode.
Show values as a table
| Live v6 | v7 | |
|---|---|---|
| Procedure check · legal | 0.00 | 98.60 |
| Procedure points · legal | 11.40 | 81.40 |
| Procedure check · general | 0.00 | 32.90 |
| Procedure points · general | 11.40 | 45.70 |
Technical sheet
| Version | v7 |
|---|---|
| Release | September 2026 |
| Input | Text and image |
| Output | Text |
| Language | Turkish-focused; responds in the same language for other languages |
CALLAII v6 · Previous version
CALLAII v6 powers all AI surfaces within the Callmenta panel. It was trained on data generated from scratch for this purpose; every sample passed rule-based checks and a judge evaluation.
What improved in this version
- Petition and contract style was learned from real documents of the law firm we work with
- Contract drafting and evidence review were added
- Multi-turn conversation and tool use were strengthened
- Training was regularly evaluated on 740 unseen samples; validation error decreased at every measurement, and the model did not memorize
What it does
- Call analysis: summary, action items, missed opportunities, and criteria scorecard from call transcripts
- Coaching: evidence-based feedback for agent performance
- Live support: answers visitor questions and escalates to a human when needed
- CRM and reporting: customer card summary, metric interpretation, and commentary
- CALLAII Legal: drafting petitions and contracts, document analysis, and legal/regulatory/case-law queries
- KVKK masking: extracting personal data from text
- Tool use: search, record creation, and document retrieval within the panel
- Visual reading: understanding documents and screenshots
Limitations and responsible use
- Legal knowledge does not come from the model's memory; laws, regulations, and rulings are retrieved from the current library each time
- Generated text is a draft; petitions, contracts, and official correspondence must be approved by an authorized person
- Numbers and dates must be verified; the model was trained to state missing information rather than fabricate it
- Very long documents are processed in segments
- Quality in languages other than Turkish has not been measured with the same granularity as Turkish
Training and measurements
| Training data | 37,102 samples, 118 million tokens |
|---|---|
| Quality filtering | 37,600 of 47,872 generated samples were accepted |
| Validation set | 740 samples never seen during training |
| Steps | 3,101 · 2 epochs · 96,000 tokens per step |
| Training context | 12,288 tokens |
| Validation loss | 0.7435 → 0.4729 |
| Training date | September 2026 |
Only measured points; records of intermediate steps were not retained.
Show values as a table
| Training loss | |
|---|---|
| 0 | 0.7800 |
| 2,700 | 0.3951 |
| 3,101 | 0.3763 |
| Validation loss | |
| 0 | 0.7435 |
| 2,700 | 0.4730 |
| 3,101 | 0.4729 |
Every generated sample passed rule-based checks and judge evaluation; those that failed were discarded.
Show values as a table
| Samples | |
|---|---|
| Generated | 47,872.00 |
| Accepted | 37,600.00 |
| Training | 37,102.00 |
| Validation | 740.00 |
Technical sheet
| Version | v6 |
|---|---|
| Release | September 2026 |
| Input | Text and images |
| Output | Text |
| Language | Turkish-focused; other languages answered in the same language |
| Customer data | not used for training |
| Hosting | servers managed by Callmenta; your data is not sent to a third-party AI provider |
CALLAII v5 · Previous version
CALLAII's first comprehensive release.
What improved in this version
- Call center, coaching, and live support surfaces unified in a single model.
Training and measurements
| Training data | 45,000 samples, 15 categories, 9 languages |
|---|---|
| Steps | 411 · 1 epoch |
| Tokens processed | 26.7 million |
| Final validation loss | 0.4867 |
| Validation token accuracy | 85.4% |
Show values as a table
| Training loss | |
|---|---|
| 10 | 2.7335 |
| 30 | 1.2499 |
| 50 | 0.8402 |
| 70 | 0.7024 |
| 90 | 0.6135 |
| 110 | 0.5847 |
| 130 | 0.5689 |
| 150 | 0.5584 |
| 170 | 0.5385 |
| 190 | 0.5599 |
| 210 | 0.5412 |
| 230 | 0.5371 |
| 250 | 0.5122 |
| 270 | 0.5038 |
| 290 | 0.5125 |
| 310 | 0.5011 |
| 330 | 0.4956 |
| 350 | 0.4728 |
| 370 | 0.4874 |
| 390 | 0.4968 |
| 410 | 0.4969 |
| Validation loss | |
| 40 | 0.8849 |
| 80 | 0.6208 |
| 120 | 0.5651 |
| 160 | 0.5402 |
| 200 | 0.5221 |
| 240 | 0.5106 |
| 280 | 0.5010 |
| 320 | 0.4927 |
| 360 | 0.4880 |
| 400 | 0.4867 |
| 411 | 0.4867 |
Show values as a table
| Training | |
|---|---|
| 10 | 57.20 |
| 30 | 70.07 |
| 50 | 78.02 |
| 70 | 80.74 |
| 90 | 82.59 |
| 110 | 83.02 |
| 130 | 83.40 |
| 150 | 83.64 |
| 170 | 84.01 |
| 190 | 83.63 |
| 210 | 83.78 |
| 230 | 83.92 |
| 250 | 84.41 |
| 270 | 84.64 |
| 290 | 84.49 |
| 310 | 84.78 |
| 330 | 84.86 |
| 350 | 85.33 |
| 370 | 85.09 |
| 390 | 84.94 |
| 410 | 84.81 |
| Validation | |
| 40 | 77.40 |
| 80 | 82.75 |
| 120 | 83.79 |
| 160 | 84.25 |
| 200 | 84.61 |
| 240 | 84.85 |
| 280 | 85.04 |
| 320 | 85.23 |
| 360 | 85.32 |
| 400 | 85.37 |
| 411 | 85.37 |
Technical sheet
| Version | v5 |
|---|---|
| Release | September 2026 |
| Status | Superseded by v6 |