 [2305.11186] Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt




























  








Skip to main content





Grab your spot at the free arXiv Accessibility Forum
Forum Schedule

We gratefully acknowledge support fromthe Simons Foundation, Stockholm University, and all contributors. Donate





 > cs > arXiv:2305.11186
  





Help | Advanced Search




All fields
Title
Author
Abstract
Comments
Journal reference
ACM classification
MSC classification
Report number
arXiv identifier
DOI
ORCID
arXiv author ID
Help pages
Full text




Search















open search






GO



open navigation menu


quick links

Login
Help Pages
About












Computer Science > Computation and Language


arXiv:2305.11186 (cs)
    




  [Submitted on 17 May 2023 (v1), last revised 10 Oct 2023 (this version, v2)]
Title:Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt
Authors:Zhaozhuo Xu, Zirui Liu, Beidi Chen, Yuxin Tang, Jue Wang, Kaixiong Zhou, Xia Hu, Anshumali Shrivastava View a PDF of the paper titled Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt, by Zhaozhuo Xu and 6 other authors
View PDF

Abstract:While the numerous parameters in Large Language Models (LLMs) contribute to their superior performance, this massive scale makes them inefficient and memory-hungry. Thus, they are hard to deploy on commodity hardware, such as one single GPU. Given the memory and power constraints of such devices, model compression methods are widely employed to reduce both the model size and inference latency, which essentially trades off model quality in return for improved efficiency. Thus, optimizing this accuracy-efficiency trade-off is crucial for the LLM deployment on commodity hardware. In this paper, we introduce a new perspective to optimize this trade-off by prompting compressed models. Specifically, we first observe that for certain questions, the generation quality of a compressed LLM can be significantly improved by adding carefully designed hard prompts, though this isn't the case for all questions. Based on this observation, we propose a soft prompt learning method where we expose the compressed model to the prompt learning process, aiming to enhance the performance of prompts. Our experimental analysis suggests our soft prompt strategy greatly improves the performance of the 8x compressed LLaMA-7B model (with a joint 4-bit quantization and 50% weight pruning compression), allowing them to match their uncompressed counterparts on popular benchmarks. Also, we demonstrate that these learned prompts can be transferred across various datasets, tasks, and compression levels. Hence with this transferability, we can stitch the soft prompt to a newly compressed model to improve the test-time accuracy in an ``in-situ'' way.
    



Subjects:

Computation and Language (cs.CL); Machine Learning (cs.LG)

Cite as:
arXiv:2305.11186 [cs.CL]


 
(or 
arXiv:2305.11186v2 [cs.CL] for this version)
          
 
 

https://doi.org/10.48550/arXiv.2305.11186



Focus to learn more




                arXiv-issued DOI via DataCite
              







Submission history From: Zhaozhuo Xu [view email]       [v1]
        Wed, 17 May 2023 20:45:13 UTC (247 KB)
[v2]
        Tue, 10 Oct 2023 04:01:30 UTC (248 KB)



 

Full-text links:
Access Paper:


View a PDF of the paper titled Compress, Then Prompt: Improving Accuracy-Efficiency Trade-off of LLM Inference with Transferable Prompt, by Zhaozhuo Xu and 6 other authorsView PDFTeX SourceOther Formats
view license

 
    Current browse context: cs.CL


< prev

  |   
next >


new
 | 
recent
 | 2023-05

    Change to browse by:
    
cs
cs.LG




References & Citations

NASA ADSGoogle Scholar
Semantic Scholar




a
export BibTeX citation
Loading...




BibTeX formatted citation
×


loading...


Data provided by: 




Bookmark





 




Bibliographic Tools

Bibliographic and Citation Tools






Bibliographic Explorer Toggle



Bibliographic Explorer (What is the Explorer?)







Litmaps Toggle



Litmaps (What is Litmaps?)







scite.ai Toggle



scite Smart Citations (What are Smart Citations?)








Code, Data, Media

Code, Data and Media Associated with this Article






Links to Code Toggle



CatalyzeX Code Finder for Papers (What is CatalyzeX?)







DagsHub Toggle



DagsHub (What is DagsHub?)







GotitPub Toggle



Gotit.pub (What is GotitPub?)







Links to Code Toggle



Papers with Code (What is Papers with Code?)







ScienceCast Toggle



ScienceCast (What is ScienceCast?)











Demos

Demos






Replicate Toggle



Replicate (What is Replicate?)







Spaces Toggle



Hugging Face Spaces (What is Spaces?)







Spaces Toggle



TXYZ.AI (What is TXYZ.AI?)








Related Papers

Recommenders and Search Tools






Link to Influence Flower



Influence Flower (What are Influence Flowers?)







Connected Papers Toggle



Connected Papers (What is Connected Papers?)







Core recommender toggle



CORE Recommender (What is CORE?)





Author
Venue
Institution
Topic














        About arXivLabs
      



arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.










Which authors of this paper are endorsers? |
    Disable MathJax (What is MathJax?)
    












About
Help





contact arXivClick here to contact arXiv
 Contact


subscribe to arXiv mailingsClick here to subscribe
 Subscribe











Copyright
Privacy Policy




Web Accessibility Assistance


arXiv Operational Status 
                    Get status notifications via
                    email
                    or slack





 





