Guide

Test your AI on Arabic and mixed-language input

Models that work well in English often slip on Arabic names, mixed scripts and Gulf formats. BoxAI's AI test sets give you realistic inputs with the right answer attached, so you can measure that, not guess.

Three kinds of test case

• Field extraction (KYC): a customer message with a name, Emirates ID, phone, IBAN and emirate in it. The expected answer is the fields as JSON.
• Personal data redaction: text with personal data in it. The expected answer lists each item with its type and exact character positions.
• Invoice questions: an invoice and a question about it, with the right answer.

Each comes in Arabic, English, or a mix of both, the way real messages arrive.

Get a test set

curl -X POST https://api.boxai.dev/api/generate/ai-tests \
  -H "Content-Type: application/json" \
  -d '{"tasks": ["kyc_extraction", "pii_redaction"], "count": 50, "lang": "mixed"}' -o tests.jsonl

One JSON object per line, with instruction, input and expected:

{"id": "kyc_extraction-0001", "task": "kyc_extraction", "language": "en",
 "instruction": "Extract the customer's name, Emirates ID, phone, IBAN and emirate as JSON.",
 "input": "Hi, I'd like to open a new account. My name is Shamma Al Suwaidi, Emirates ID 784-2001-6765063-7 ...",
 "expected": {"name": "Shamma Al Suwaidi", "emirates_id": "784-2001-6765063-7", ...}}

Score your model

1. Send each instruction and input to your model.
2. Compare its answer with expected, field by field.
3. Report the share correct per task and per language. A gap between English and Arabic is the finding.

Use the same seed each time, and a change in the score means your model or prompt changed, not the test.

Or do it inside BoxAI

With a free key, fill a box with an AI test set and call askModel() from a script to run the cases against a model, or ask Ask AI "what share of the KYC cases did the model get right?". BoxAI also has an OpenAI-compatible model API for comparing prompts.