Insights

1. Natural language is redundant, amount of information varies.

2. LLMs can understand compressed prompt.

3. GPT-4 can perform high quality, extractive prompt compression using carefully designed instruction and chunking. (LLMLingua-2)

4. Prompt compression can be formulated as a token classification problem and accomplished by a BERT sized compressor. (LLMLingua-2)

5. Meetingbank-trained compressors generalize well to other domain data. (LLMLingua-2)

For more details, please refer to the paper LLMLingua-2.

Methodology

To address the challenges in existing entropy-based prompt compression methods, LLMLingua-2 employs a data distillation procedure, which involves extracting knowledge from an LLM (GPT-4) to compress texts without losing crucial information or introducing hallucinated content.

Instruction Design

A well-crafted instruction is the key to unveiling the compression capabilities of GPT-4. To ensure that the generated texts stay faithful to the original, we explicitly instruct GPT-4 to compress the text by discarding unimportant words in the original texts only and not adding any new words during generation.

Annotation & Filtering

Leveraging the distilled knowledge from the LLM, we design a data annotation algorithm, which assigns labels to each word in the original text to indicate whether it should be preserved after compression. To ensure the quality of the constructed dataset, we propose two quality control metrics to filter out low-quality samples. For more details, please refer to the Sec. 3 of our LLMLingua-2 paper.

Compressor

Finally, we formulate prompt compression as a token classification problem and employ a Transformer encoder as the feature extractor to capture all essential information for prompt compression from the full bidirectional context. After training on the constructed dataset, we determine whether to preserve or discard each token in the original prompt based on its probability calculated by our classification model.

Performance

We evaluate LLMLingua-2 on both in-domain and out-of-domain datasets, including various tasks such as in-context learning, summarization, conversation, multi-document QA, single-document QA, code, and synthetic tasks. The results indicate that our approach can achieve minimal performance loss and outperform the SOTA task-agnostic prompt compression methods.

• ●   In-Domain (MeetingBank)
Below, we present the results of LLMLingua-2 compared to the strong baselines on In-Domain data: the test set of MeetingBank. Despite the fact that our compressors are much smaller than the LLaMa-2-7B used in the baselines, our approach achieves significantly better performance on both the QA and Summary tasks, and comes close to matching the performance of the original prompt.

• ●   Out-of-Domain (LongBench, GSM8K and BBH)
As our model is only trained on meeting transcripts data from MeetingBank, here we explore its generalization ability across various benchmarks of long-context scenarios, reasoning, and in-context learning. Although the compressor of LLMLingua-2 is only trained on MeetingBank data, our comprehensive experiments validate its effectiveness on out-of-domain datasets, with its performance comparable to or even surpassing the SOTA task-agnostic compression baselines.

Cases Studies

We conduct a series of case studies on both in-domain and out-of-domain datasets, including various tasks such as document-based QA, summarization, retrieval-augmented QA and in-context learning. We find that LLMLingua-2 effectively preserves the informative words in the original context. Below, we showcase several compression results on both in-domain and out-of-domain datasets to provide readers with a clearer understanding of LLMLingua-2's capability.

• ●   In-Domain
1. 📝 Online Meeting QA
2. ✍️ Online Meeting Summary
• ●   Out-of-Domain
1. 🕸️  RAG, Multi-Documnent QA
2. 💡  Chain-of-Thought, Reasoning