LLMLingua Series | Effectively Deliver Information to LLMs via Prompt Compression

LLMLingua

Identify and remove non-essential tokens in prompts using perplexity from a SLM

Read More

LongLLMLingua

Enhance long-context information via query-aware compression and reorganization

Read More

LLMLingua-2

Utilize data distillation to learn compression targets for efficient and faithful task-agnostic compression

Read More

News

🍩 [24/12/13] We are excited to announce the release of our KV cache-centric analysis work, SCBench, which evaluates long-context methods from a KV cache perspective.
👘 [24/09/16] We are pleased to announce the release of our KV cache offloading work, RetrievalAttention, which accelerates long-context LLM inference via vector retrieval.
🌀 [24/07/03] We're excited to announce the release of MInference to speed up Long-context LLMs' inference, reduces inference latency by up to 10X for pre-filling on an A100 while maintaining accuracy in 1M tokens prompt! For more information, check out our paper, visit the project page.
🦚 We're excited to announce the release of LLMLingua-2, boasting a 3x-6x speed improvement over LLMLingua! For more information, check out our paper, visit the project page, and explore our demo.
👾 LLMLingua has been integrated into LangChain and LlamaIndex, two widely-used RAG frameworks.
🤳 Talk slides are available in AI Time Jan, 24.
🖥 EMNLP'23 slides are available in Session 5 and BoF-6.
📚 Check out our new blog post discussing RAG benefits and cost savings through prompt compression. See the script example here.
👨‍🦯 Explore our './examples' directory for practical applications, including LLMLingua-2, RAG, Online Meeting, CoT, Code, and RAG using LlamaIndex.

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities and have been applied across various fields. Advancements in technologies such as Chain-of-Thought (CoT), In-Context Learning (ICL), and Retrieval-Augmented Generation (RAG) have led to increasingly lengthy prompts for LLMs, sometimes exceeding tens of thousands of tokens. Longer prompts, however, can result in 1) increased API response latency, 2) exceeded context window limits, 3) loss of contextual information, 4) expensive API bills, and 5) performance issues such as “lost in the middle.”
Inspired by the concept of "LLMs is Compressors" we designed a series of works that try to build a language for LLMs via prompt compression. This approach accelerates model inference, reduces costs, and improves downstream performance while revealing LLM context utilization and intelligence patterns. Our work achieved a 20x compression ratio with minimal performance loss (LLMLingua), and a 17.1% performance improvement with 4x compression (LongLLMLingua).
LLMLingua-2, a small-size yet powerful prompt compression method trained via data distillation from GPT-4 for token classification with a BERT-level encoder, excels in task-agnostic compression. It surpasses LLMLingua in handling out-of-domain data, offering 3x-6x faster performance.

Insights

Natural language is redundant, amount of information varies.
LLMs can understand compressed prompt.
There is a trade-off between language completeness and compression ratio. (LLMLingua)
GPT-4 can recover all the key information from a compressed prompt-emergent ability. (LLMLingua)
The density and position of key information in a prompt affect the performance of downstream tasks. (LongLLMLingua)
GPT-4 can perform high quality, extractive prompt compression using carefully designed instruction and chunking. (LLMLingua-2)
Prompt compression can be formulated as a token classification problem and accomplished by a Bert size model. (LLMLingua-2)
Meetingbank-trained compressors generalize well to other domain data. (LLMLingua-2)

For more details, please refer to the project pages, LLMLingua, LongLLMLingua, and LLMLingua-2.

Cases Studies

LLMLingua and LongLLMLingua can be applied to a wide range of scenarios, particularly in Chain-of-Thought, long contexts, and RAG. We have conducted tests on various tasks, including in-context learning, summarization, conversation, multi-document QA, single-document QA, code, and synthetic tasks. The results indicate that our approach can achieve minimal performance loss or even improve performance with up to a 20x compression ratio.

Below, we will showcase several cases from common real-world scenarios to demonstrate the capabilities of LLMLingua and LongLLMLingua.

🕸️ RAG, Multi-Documnent QA
📝 Online Meeting
💡 Chain-of-Thought, Reasoning
🧑‍💻 Code Completion

🕸️ Retrieval-Augmented Generation (RAG) - Multi-Documnent Question-Answer

To test a realistic retrieval-augmented generation setup that closely resembles commercial search and question-answering applications (e.g., Bing Chat), we chose two scenarios: NaturalQuestions and LlamaIndex for evaluation. In these scenarios, we first use SoTA retrieval methods to recall several relevant documents from a candidate set, and then form the corresponding prompts. This approach allows us to assess the performance of our models in a more practical setting and measure their effectiveness in handling real-world search and question-answering tasks. The response generated by GPT-3.5-Turbo.

You can find the example in RAG.ipynb and Colab.

Also you can use LongLLMLingua in LlamaIndex pipeline, like RAGLlamaIndex.ipynb and Colab.


Method	Prompt	Tokens	Response	Accuracy
Original (NaturalQuestions)	Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant). ...... Document [2](Title: OPEC) of "the top 100 most influential people in the shipping industry". However, the influence of OPEC on international trade is periodically challenged by the expansion of non-OPEC energy sources, and by the recurring temptation for individual OPEC countries to exceed production targets and pursue conflicting self-interests. As of June 2018, OPEC has 15 member countries: six in the Middle East (Western Asia), seven in Africa, and two in South America. According to the U.S. Energy Information Administration (EIA), OPEC\'s combined rate of oil production (including gas condensate) represented 44 percent of the world\'s total in 2016, and OPEC accounted for ...... Document [10](Title: OPEC) Organization of the Petroleum Exporting Countries (OPEC, /ˈoʊpɛk/ OH-pek, or OPEP in several other languages) is an intergovernmental organization of 14 nations as of February 2018, founded in 1960 in Baghdad by the first five members (Iran, Iraq, Kuwait, Saudi Arabia, and Venezuela), and headquartered since 1965 in Vienna, Austria. As of 2016, the 14 countries accounted for an estimated 44 percent of global oil production and 73 percent of the world\'s "proven" oil reserves, giving OPEC a major influence on global oil prices that were previously determined by American-dominated multinational oil companies. ...... Question: how many countries are a part of opec in may 2018 Answer:	2,946	15	54.1
LongLLMLingua	Write a high-quality answer for the given question using only the provided search results (some of which might be irrelevant). [5Title: OPEC OPE its two newest, who had joined mid1970s. Ecuador withdrew in December 1992 it was unwilling pay US$2 million membership fee and felt that it needed to produce more it was allowed under the OPEC quota, although rejoined in October 2007. Similar concerns prompted Gabon to suspend membership in January 1995; it rejoined in July 2016 Iraq remained a member of OPEC since the organization\'s founding, Ira production was a part of OPEC quota agreements from 1998 to 216, due to the country\'s daunting political difficulties. 1 the Petroleum Exporting Countries (OPEC, /ˈoʊpɛk/ OH-pek, or OPEP in several other languages) is an intergovernmental organization of 14 nations as of February 2018, founded in 1960 in Baghdad by the first five members (Iran, Iraq, Kuwait, Saudi Arabia, and Venezuela), and headquartered since 1965 in Vienna, Austria. As of 2016, the 14 countries accounted for an estimated 44 percent of global oil production and 73 percent of the world\'s "proven" oil reserves, giving OPEC a major influence on global oil prices that were previously determined by American-dominated multinational oil companies. Question: how many countries are a part of opec in may 2018 Answer:	313 (9x)	14	75.5 (+21.4)


Original (LlamaIndex)	Given the context, please answer the final question In the fall of 1992 I moved back to Providence to continue at RISD. The foundation had merely been intro stuff, and the Accademia had been a (very civilized) joke. Now I was going to see what real art school was like. But alas it was more like the Accademia than not. Better organized, certainly, and a lot more expensive, but it was now becoming clear that art school did not bear the same relationship to art that medical school bore to medicine. At least not the painting department. The textile department, which my next door neighbor belonged to, seemed to be pretty rigorous. No doubt illustration and architecture were too. But painting was post-rigorous. Painting students were supposed to express themselves, which to the more worldly ones meant to try to cook up some sort of distinctive signature style. There were plenty of earnest students too: kids who "could draw" in high school, and now had come to what was supposed to be the best art school in the country, to learn to draw even better. They tended to be confused and demoralized by what they found at RISD, but they kept going, because painting was what they did. I was not one of the kids who could draw in high school, but at RISD I was definitely closer to their tribe than the tribe of signature style seekers. I learned a lot in the color class I took at RISD, but otherwise I was basically teaching myself to paint, and I could do that for free. So in 1993 I dropped out. I hung around Providence for a bit, and then my college friend Nancy Parmet did me a big favor. A rent-controlled apartment in a building her mother owned in New York was becoming vacant. Did I want it? It wasn\'t much more than my current place, and New York was supposed to be where the artists were. So yes, I wanted it! [7] ...... Where did the author go for art school?	1,955	The author went to art school at RISD (Rhode Island School of Design).	-
LongLLMLingua	Given the context, please answer the final question why artists a signature style,ers pay a lot work. I a lot the color class tookD but I myself do for in9 I hung around then my favor. A York. It my be where the wereI for the B, to go only 25, and art schools are full of people of different ages. RISD counted me as a transfer sophomore and said I had to do the foundation that summer. The foundation means the classes that everyone has to take in fundamental subjects like drawing, color, and design. Where did the author go for art school?	125 (16x)	The author went to RISD (Rhode Island School of Design) for art school.	-

📝 Online Meeting

Using generative AI like ChatGPT in online meetings can greatly improve work efficiency (e.g., Teams). However, the context in such applications tends to be more conversational, with a high degree of redundancy and a large number of tokens(more than 40k). By utilizing LLMLingua to compress prompts, we can significantly reduce the length of prompts, which in turn helps to reduce latency. This makes the AI more efficient and responsive in real-time communication scenarios like online meetings, enabling smoother interactions and better overall performance. We use meeting transcripts from the MeetingBank dataset as an example to demonstrate the capabilities of LLMLingua.

You can find the example in OnlineMeeting.ipynb and Colab.


Method	Prompt	Tokens	Response
Original	Madam Court, could you please read docket 1239? Certainly. Docket 1239. The Committee on Government Operations, to which was referred on December 1st, 2021, docket number 1239 message an order authorizing the creation of a sheltered market program in conformity with the requirements of general laws. Chapter 30 B Section 18. This authorization applies to contracts for goods, professional services and support services. This authorization is for no more than six contracts, which must be awarded by June 30th, 2022. This sheltered market program shall be available for disadvantaged, minority and women only vendors, for whom there is a demonstrated substantial disparity in the city's 2020 disparities. Study submits a report recommending the order ought to pass. Thank you so much, Madam Clerk. The Chair recognizes Councilor Edwards, chair of the committee. Councilor Edwards. You have the floor. This is this is actually a matter, I believe, sponsored by the. Mayor in Cannes. In conformance with the recommendations from the disparity study and making sure that we opt in to this this pilot program under mass general laws 30 Section 18. Again, it's really just following the recommendations of an already studied issue, which which demonstrates a disparity between minority contractors or women contractors receiving contracts in the city of Boston. So this would allow for us to shepherd and move these six contracts to those already designated groups who have a disadvantage. And I think it's. Really fulfilling a promise. Of making sure that we go through and make sure all aspects of the city government, including the financial benefits, are accessible to people in the city of Boston. I recommend that this pass and I hope that my colleagues will vote for it. Thank you. Thank you so much. Councilor Edward seeks acceptance of the committee report and passage of Docket 1239. Madam Court, could you please call the roll? Certainly. Docket 1239. Councilor Arroyo. Yes. Councilor Arroyo. Yes. Councilor Baker. Councilor Baker. Councilor. Councilor Barker. Council Braden. Councilor Braden. Councilor Campbell. Councilor Campbell. Yes. Councilor Edwards. Yes. Councilor Sabby. George. Councilor Sabby. George. He has Councilor Flaherty. Councilor Flaherty as Councilor Flynn. Councilor Flynn. Yes. Councilor Jane. Yes. Councilor Janey. As Councilor me here. Councilor me here as Councilor Murphy. Councilor Murphy. Yes. And Councilor O'Malley. Yes. Councilor O'Malley. Yes. Madam President, do I get number 1239 has received unanimous vote. Thank you so much. Dockett 1239 has passed and now we will move on to matters recently heard for possible action. Madam Clerk, if you could please read docket 0863. Certainly Docket 0863 order for hearing to discuss pest control and illegal dumping in the city of Boston. Question: What is the subject of Docket 0863? Answer:	626	The subject of Docket 0863 is an order for a hearing to discuss pest control and illegal dumping in the city of Boston.
LongLLMLingua	Madam, read 12?39 The on Government referred,ocket39 message order a sheltered market in of. Chapter 30 B Section This to, and for six be by This for only a demonstrated substantial the's 2020. Studymits the order to. The Chair recognizesor, the is is matter, the. Mayor in In the the and we this program under mass general laws Section.'s just studied which which a between minority or contractors receiving to and these designated a's. Really Of all city are I this that.or Edward seeks and of.am, the?.or.or.or Council.oror. hasorororor. get 1239 received.ett to matters foram readocket. Certainly3 for discuss and of Boston. Question: What is the subject of Docket 0863? Answer:	152 (4x)	The subject of Docket 0863 is not explicitly mentioned in the provided text. The text appears to be a transcription of a discussion or meeting, but it does not contain information about Docket 0863 or its subject matter.
LLMLingua2	Madam Court read docket 1239? Committee Government Operations referred December 1st 2021 sheltered market program laws Chapter 30 B Section 18. contracts goods services six contracts June 30th 2022 disadvantaged minority women vendors disparity city 2020 disparities order pass Chair recognizes Councilor Edwards sponsored Mayor disparity study pilot program laws 30 Section 18. disparity minority women contractors Boston six contracts groups government financial benefits Boston recommend pass colleagues vote seeks acceptance report passage Docket 1239 call roll? Docket 1239 Councilor Arroyo BakerCouncilor Flaherty Flynn Murphy O'Malley Madam President 1239 unanimous vote 1239 passed Madam Clerk read docket 0863 pest control illegal dumping Boston Question: What is the subject of Docket 0863? Answer:	151 (4x)	The subject of Docket 0863 is "pest control and illegal dumping in Boston."
Original	Speaker 4: We're going to hear all of the presentations at once. And then after we go through all the presentations, we'll do all the all of the questions at once and then any any public comment, and we'll go from there. Thank you. Speaker 7: All right. Thank you, Mr. Mayor. It was our pleasure to give you an overview of the budget last Tuesday, we transmitted that officially to the mayor and city council and the public. And now we're looking to really dove deeper in some of the key departments. Today, we want to go through three key departments, the fire department. Then we'll go through our police department and then Parks, Recreation and Marine. So I will turn it over to Chief Espino and his team to go through the fire department presentation first followed by police, followed by Parks and Rec. Speaker 1: It's. Speaker 6: Thank you, Mr. Monaco. Good evening. Mayor Garcetti and members of the City Council. This evening, I will be providing a brief overview of the fire department, highlighting the key services that we provide to the community and some of our recent accomplishments. ......Over the past 12 months, we've responded to over 80,000 fire, emergency medical services, marine safety and other emergency incidents, a 5% increase from last year. ...... ...... Speaker 9: Thank you, Mr. Modica. Ms.. Tatum. Good afternoon, Honorable Mayor and members of the City Council. I'm very excited to be presenting the Police Department's proposed FY 23 budget. My first as Chief of Police for our great city. ...... Like many major cities, we experienced an increase in violent crime at the beginning of 2022. However, we remain focused and resolute. We developed strategies and our employees adapted to several operational changes. Our efforts have resulted in a notable downward trend in violent crime. And after starting 22, 2022, with a 17.4% increase, we've now seen that number shrink to a 5.4% increase year to date. The accessibility of firearms has also contributed to some of the violence we're experiencing this year. Our officers have worked vigorously to address gun crime in the illegal possession of firearms. Oftentimes, they do this at great risk to their own lives. A.... ...... Speaker 1: ...... And yet, even with the overtime, we're still not meeting the state requirements. California requires that 95% of all night long calls be at within 15 seconds. However, we're falling short of that and too that at a rate of just about 77% for 2022, approximately 4% of our 7000 calls this year are. ...... (Q1) Question: How much did the crime rate increase last year?\nAnswer: (Q2) Question: What is the homicide clearance rate?\nAnswer: (Q3) Question: what are the arrangements the Police Department will make this year?	30,086	Q1: Over the past 12 months, we've responded to over 80,000 fire, emergency medical services, marine safety and other emergency incidents, a 5% increase from last year. Q2: The homicide clearance rate is 77%. Q3: The Police Department will make several arrangements this year. These include addressing the steady increase in call volume and improving response times to fires, emergency medical and other emergency responses. The department will also focus on firefighter safety and behavioral health, increasing diversity in all ranks of the department through recruitment and training opportunities, and maintaining staffing and resources to meet service demands of citywide growth. Additionally, the department will work on improving compliance with mandated fire prevention inspections and streamlining the fire plan check review process.
LongLLMLingua (Q1)	aker3., the. Speaker 9 : Thank youoman Yes,'s. 5.4% increase to date That after this a 1.4 increase in crime in. : you Let keep police:.: Investig : No, Counciloman, it is the federal UCR one violentized onines. can say the.gets day. Our department will continue to evolve and move forward, building on our existing strengths and taking advantage of opportunities for growth and renewal. Our priorities around crime and homelessness, employee and community wellness and open communication will help guide us further into 21st century policing, while also supporting the shared responsibility of public safety in the city of Long Beach. Thank you. Myself and Bureau Chief Josie Murray stand ready to answer any questions they can. Question: How much did the crime rate increase last year? Answer:	172 (175x)	5.4%
LongLLMLingua (Q2)	aker 1: From Police you. T afternoon, and of them very be presenting the Police Department proposed3 budget My for our The lastented policeple, because their some have to the the the spend the. because we' typical a offering specialative training, to close centered around andation Our mental evaluationach as are howvere person,, and no or here each a of listen Theyance all In risingented, serve and' to ensure everyone community has to thrive. Every we services we providegency criminal,, andarding public and locations from domesticats, we the resources we, like Cityutor on In withIP, Tra Department and Communic to link safety responses Or our health as thes team reason the. Here volumecing, officers working. Year to00 calls. Although we' several, our response priority the of about5 minutes many we experienced crime we several notable with weve increase fire therecing this year have vig address inms O at to their lives As of June, had a%, weve mades an.1 these peopleiddening. this seized a 23 the personallyms the second' continueends address crime Theating experienced is. And while every are and toward bringing justice ones ofimsll have aide of77%. national average, a which is approximately a There are manyiveness, a by officers followations efforts ouride In addition, when the, more to become to and of these are to achieve. highlight work,itions known as street takevers. supported efforts street this In police and improve sharing We a pending racingover To our have4 We supportingach the andicians our, Year offered over00 have temporary and housing for over people. well our employees is a priority and be through and.reidesingness and over the with in on a administrator to employeeuma intersection between and supporting informed community available on critical discuss that of thees facing lawcement today ismentention.ing short in a our., more critical and of this police including groups inve the with of by. ourve1 civil new Hruits andades. increase communication and we help shape division commander Our of matchgoing as Current the. And media information effectively health and on transformationalities .' FY3s un for and in the following crime andness supporting and communicationency weing police department our the fund a that and To visionantm the newative Responseagement C shortest. and that. and. relationship community. propose of officersrol division as the immediate addressness officers form two specificrol single altogether life forwide is moving the, to police ofcement to.reing. the The Affairs on warning Administration of nine F sectionatives were. it.mations which the rankscademy and2re support and work is operateations of and additional as police Anaticated our to lead the officers and professional staff of the Long Beach Police Department and how appreciative I am for the dedicated service they provide each day. Our department will continue to evolve and move forward, building on our existing strengths and taking advantage of opportunities for growth and renewal. Our priorities around crime and homelessness, employee and community wellness and open communication will help guide us further into 21st century policing, while also supporting the shared responsibility of public safety in the city of Long Beach. Thank you. Myself and Bureau Chief Josie Murray stand ready to answer any questions they can. Question: What is the homicide clearance rate? Answer:	688 (44x)	The homicide clearance rate is 77%
LongLLMLingua (Q3)	Speaker3: Thank you. Next week, the police. Speaker 3: Thank you. Next, keep the police. Speaker 1 From Police. Speaker2 Pat required complete lot of firear able Speaker1: We responded the of andSpeaker3: Thank you.' keepSpeaker1:Speaker 9: Thank you Councilwoman Yes,'s.54% year date That is after starting this year 17.4% in violent January.Speaker :Spe 1: I.Spe: the next Spe: you Mr. Modica Good, Honorable Mayor and of the City Councilm very to being the PolicesY3 budget. of for our city years haveented. Our police experienced increasedple yet each serve entire Long Beach community. their work has purpose. They this because they needed chosen to the department or altogether vast the vast majority have spend entireers Long Beach Department because we're not your typical city police department.re be a world offering specialized and innov but maintain relationships around andation,ach such as walks are how we, learn and respond to our community' and were person, group, and police department is of the or situation, the police officers and professional staff this here day, providing a of our community listen.age. And st enhance the people in city. In of pandemic, rising violent un attacks police officers commitment un and inspiring willing to everyone in our to thrive. Every year we public safety services provide, emergency forations, victim supportacy, safegu our while protect and resource international domesticats However, the resources we dedicate partners and collaborations, like working with Proutor's on Altern Incer with community organizations such T theumavention to to incidents.ating Fire Department andgency Communication Center link safety responses. Or joining with health as of city'department homelessness. andhips are' able do the work we do. Here you see the call we experien exemplifies hard our working. date, have respond over 0,00 for service, which is nearly 0 calls a day. ... Today'3 budget proposals vision for department priorit in the following key areas andcknowled and internal communication models Built a of accountability and trans Although we're propos many unique andative changes the police department's structural budget requestY 2 our general a strategicorgan thatign the Department and operations to current and anticip and provides coordination of police services. fulfill this Sign focus n The Coll Response, orRC for. bureau will overseen by ourest Deputy focus goals that enhative public models and build meaningful relationships between the department and public Kerry will responsible of the and partnerships currently promote these goals. theureau will consist of two theach Response En and Youth and Community Services Division.ry prioritize aligned with disrupt primary factors relationship baseding models To accomplish this addition of 0 new police of 6 bike officersrolional areas such spacesks in the with immediate of towide addresselessness. ...... Question: what are the arrangements the Police Department will make this year? Answer:	1,836 (16x)	This year, the Police Department will focus on the following key areas: enhancing community engagement and internal communication models, building a culture of accountability and transparency, and prioritizing recruitment and retention. We are proposing the creation of a new bureau in our department, the Community Response Bureau (CRB), which will be overseen by our newest Deputy Chief. The CRB will focus on goals that enhance innovative public safety models and build meaningful relationships between the department and the public. We are also proposing the addition of 10 new police officers, the expansion of the Internal Affairs Division, and the rebranding of the Public Information Division to the Strategic Communications Bureau. We will also be bringing back two back-to-back police academy classes, adding an administrative analyst position, and increasing staffing in the Business Services Division.

💡 In-Context Learning, Chain-of-Thought, Reasoning

LLMs are known for their outstanding Chain-of-Thought and reasoning abilities. To evaluate the preservation of such capabilities in compressed prompts, we conducted tests on GSM8K using complex 9-steps Chain-of-Thought prompts as the original prompts. The results show that a similar performance can be maintained at a compression ratio of up to 20x.

You can find the example in CoT.ipynb and Colab.


Method	Prompt	Tokens	Response	Accuracy
Original	Please reference the following examples to answer the math question, Question: Sam bought a dozen boxes, each with 30 highlighter pens inside, for $10 each box. He rearranged five of these boxes into packages of six highlighters each and sold them for $3 per package. He sold the rest of the highlighters separately at the rate of three pens for $2. How much profit did he make in total, in dollars? Let's think step by step Sam bought 12 boxes x $10 = $120 worth of highlighters. He bought 12 * 30 = 360 highlighters in total. Sam then took 5 boxes × 6 highlighters/box = 30 highlighters. He sold these boxes for 5 * $3 = $15 After selling these 5 boxes there were 360 - 30 = 330 highlighters remaining. These form 330 / 3 = 110 groups of three pens. He sold each of these groups for $2 each, so made 110 * 2 = $220 from them. In total, then, he earned $220 + $15 = $235. Since his original cost was $120, he earned $235 - $120 = $115 in profit. The answer is 115 Question: A group of 4 fruit baskets contains 9 apples, 15 oranges, and 14 bananas in the first three baskets and 2 less of each fruit in the fourth basket. How many fruits are there? Let's think step by step ...... The answer is 146 ...... Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?	2,366	Let's think step by step: Josh bought the house for $80,000. He invested an additional $50,000 in repairs. After the repairs, the value of the house increased by 150%. To calculate the new value, you need to add 150% of the original value to the original value: New Value = Original Value + (150% of Original Value) New Value = $80,000 + (1.5 * $80,000) New Value = $80,000 + $120,000 New Value = $200,000 To calculate the profit, subtract the total cost (purchase price + repair costs) from the new value: Profit = New Value - Total Cost Profit = $200,000 - ($80,000 + $50,000) Profit = $200,000 - $130,000 Profit = $70,000 Josh made a profit of $70,000 from flipping the house.	78.85
LLMLingua	Please reference the following examples to answer the math question, : Sam bought a dozen boxes each 30 highl pens inside, $10 each. He reanged five of boxes into of six each $3 per. He sold the thelters separately at the of three $2. much make total, Lets think step bought boxes x0 oflters He 2 3ters in Sam then boxes 6lters/box 0ters He sold these boxes 5 Afterelling these boxes there 36030lters ese00 of three sold groups2 each so made *2 $20 from In total, he015 Since his he $ - $120 = $115 in profit. The answer is 115 Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?	117 (20x)	Let's think step by step The value of the house increased by 80,0001.5=$<<800001.5=120000>>120,000 So the total value of the house is 80,000+120,000=$<<80000+120000=200000>>200,000 So he made a profit of 200,000-80,000-50,000=$<<200000-80000-50000=70000>>70,000. Answer: 70000.	77.94
LLMLingua2	Please reference the following examples to answer the math question, Question: Sam bought dozen boxes each with 30 highlighter pens for $10 each rearranged five boxes into packages six highlighters sold for $3 per package sold rest highlighters three pens for $2. profit total dollars? step by step Sam bought 12 boxes x $10 = $120 highlighters 12 * 30 = 360 highlighters total took 5 boxes × 6 highlighters/box = 30 highlighters sold boxes for 5 * $3 = $15 After 5 360 - 30 = 330 highlighters remaining 330 / 3 = 110 groups of three pens sold each groups for $2 each made 110 * 2 = $220 earned $220 + $15 = $235. original cost $120 earned $235 - $120 = $115 profit The answer is 115 Question: Josh decides to try flipping a house. He buys a house for $80,000 and then puts in $50,000 in repairs. This increased the value of the house by 150%. How much profit did he make?	178 (14x)	Let's think step by step The repairs increased the value by 80,0001.5=$<<800001.5=120000>>120,000 So the house is now worth 80,000+120,000=$<<80000+120000=200000>>200,000 So his profit was 200,000-80,000-50,000=$<<200000-80000-50000=70000>>70,000. Answer: 70,000.	77.79

🧑‍💻 Code Completion

In addition to natural language, we also evaluate the capabilities of our method on code. For example, in repobench-p, the original prompts can be particularly long, up to 20k characters. LongLLMLingua can achieve a 1.4 point improvement at a 6x compression ratio.

You can find the example in Code.ipynb and Colab.


Method	Prompt	Tokens	Response	Accuracy
Original	Please complete the code given below. Spectaculum-Effect-Immersive/src/main/java/net/protyposis/android/ImmersiveTouchNavigation.java public class ImmersiveTouchNavigation { private static final String TAG = ImmersiveTouchNavigation.class.getSimpleName(); private SpectaculumView mSpectaculumView; private boolean mSpectaculumViewTouchEnabled; ...... Spectaculum-Demo/src/main/java/net/protyposis/android/testeffect/InterlaceEffect.java public class InterlaceEffect extends ShaderEffect { @Override protected TextureShaderProgram initShaderProgram() { final InterlaceShaderProgram shaderProgram = new InterlaceShaderProgram(); ...... Spectaculum-Core/src/main/java/net/protyposis/android/spectaculum/effects/Parameter.java public interface Parameter { ...... public EffectManager(Activity activity, int parameterListViewId, SpectaculumView glView) { mActivity = activity; mParameterListView = (ViewGroup) activity.findViewById(parameterListViewId); mSpectaculumView = glView; mEffects = new ArrayList<>(); mSpectaculumView.setEffectEventListener(this); // Spectaculum-Core filters Next line of code:	21,241	`// Add a no-effect filter as the first option mEffects.add(new NoEffect());`	54.2
LongLLMLingua	Please complete the code given below. Spectaculum-Core/src/main/java/net/protyposis/android/spectaculum/SpectaculumView.java * Creates a sensor navigation instance for the immersive effect. SpectaculumView.this.onEffectError(index, effect, e); SpectaculumView.this.onEffectInitialized(index, effect); ...... public EffectManager(Activity activity, int parameterListViewId, SpectaculumView glView) { mActivity = activity; mParameterListView = (ViewGroup) activity.findViewById(parameterListViewId); mSpectaculumView = glView; mEffects = new ArrayList<>(); mSpectaculumView.setEffectEventListener(this); // Spectaculum-Core filters Next line of code:	1,613 (13x)	`// Add a no-effect filter to disable all effects mEffects.add(new NoEffect());`	56.6 (+2.4)

BibTeX

If you find this repo helpful, please cite the following papers:

@inproceedings{jiang-etal-2023-llmlingua,
    title = "{LLML}ingua: Compressing Prompts for Accelerated Inference of Large Language Models",
    author = "Huiqiang Jiang and Qianhui Wu and Chin-Yew Lin and Yuqing Yang and Lili Qiu",
    editor = "Bouamor, Houda  and
        Pino, Juan  and
        Bali, Kalika",
    booktitle = "Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing",
    month = dec,
    year = "2023",
    address = "Singapore",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2023.emnlp-main.825",
    doi = "10.18653/v1/2023.emnlp-main.825",
    pages = "13358--13376",
}

@inproceedings{jiang-etal-2024-longllmlingua,
    title = "{L}ong{LLML}ingua: Accelerating and Enhancing {LLM}s in Long Context Scenarios via Prompt Compression",
    author = "Huiqiang Jiang and Qianhui Wu and and Xufang Luo and Dongsheng Li and Chin-Yew Lin and Yuqing Yang and Lili Qiu",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.acl-long.91",
    pages = "1658--1677",
}

@inproceedings{pan-etal-2024-llmlingua,
    title = "{LLML}ingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression",
    author = "Zhuoshi Pan and Qianhui Wu and Huiqiang Jiang and Menglin Xia and Xufang Luo and Jue Zhang and Qingwei Lin and Victor Ruhle and Yuqing Yang and Chin-Yew Lin and H. Vicky Zhao and Lili Qiu and Dongmei Zhang",
    editor = "Ku, Lun-Wei  and
      Martins, Andre  and
      Srikumar, Vivek",
    booktitle = "Findings of the Association for Computational Linguistics ACL 2024",
    month = aug,
    year = "2024",
    address = "Bangkok, Thailand and virtual meeting",
    publisher = "Association for Computational Linguistics",
    url = "https://aclanthology.org/2024.findings-acl.57",
    pages = "963--981",
}