Local LLM Benchmarking: Summarization Task

CONCLUSIONS

This benchmark evaluates models on summarization accuracy, coherence, and relevance using standardized metrics and human evaluation. Mistral, Llama 2, Code Llama, and Perplexity’s API will be compared on a diverse dataset, with results prioritizing factual fidelity and readability.

METHODOLOGY

1. Task Selection

Summarization is chosen as the benchmark task due to its prevalence in real-world applications (e.g., news aggregation, document analysis) and the availability of standardized evaluation metrics (ROUGE, BLEU).

2. Dataset

• Source: Use the CNN/Daily Mail dataset (100,000+ articles) for general text, supplemented with XSum for longer documents.
• Preprocessing: Split into training/validation/test sets; anonymize sensitive data.

3. Model Setup

• Mistral: Use the 7B parameter version with default hyperparameters (e.g., temperature=0.7, max tokens=512).
• Llama 2: Use the 70B parameter version with quantization (e.g., 4-bit) for efficiency.
• Code Llama: Fine-tune on code-summarization datasets (e.g., CodeSearchNet) for technical accuracy.
• Perplexity API: Use the summarize endpoint with identical parameters (e.g., length=300, model=llama-3).

4. Evaluation Metrics

• Automated Metrics:

- ROUGE-1, ROUGE-2, ROUGE-L (for precision/recall/f1 scores).

- BLEU-4 (for multilingual consistency).

• Human Evaluation:

- 500 summaries rated by 10 evaluators on a 5-point scale for:

- Factual Accuracy (does the summary capture key facts?).

- Coherence (logical flow and grammar).

- Relevance (avoiding extraneous details).

5. Testing Procedure

1. Run each model on the test set, saving outputs with timestamps and metadata.
2. Calculate automated metrics for all outputs.
3. Randomly select 500 summaries for human evaluation, ensuring diversity in domain and length.
4. Aggregate results into a comparative table (e.g., average ROUGE scores, human ratings).