 [2305.11175] VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks




























  








Skip to main content





We gratefully acknowledge support fromthe Simons Foundation, Stockholm University, and all contributors.
Donate





 > cs > arXiv:2305.11175
  





Help | Advanced Search




All fields
Title
Author
Abstract
Comments
Journal reference
ACM classification
MSC classification
Report number
arXiv identifier
DOI
ORCID
arXiv author ID
Help pages
Full text




Search















open search






GO



open navigation menu


quick links

Login
Help Pages
About












Computer Science > Computer Vision and Pattern Recognition


arXiv:2305.11175 (cs)
    




  [Submitted on 18 May 2023 (v1), last revised 25 May 2023 (this version, v2)]
Title:VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks
Authors:Wenhai Wang, Zhe Chen, Xiaokang Chen, Jiannan Wu, Xizhou Zhu, Gang Zeng, Ping Luo, Tong Lu, Jie Zhou, Yu Qiao, Jifeng Dai View a PDF of the paper titled VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks, by Wenhai Wang and 10 other authors
View PDF

Abstract:Large language models (LLMs) have notably accelerated progress towards artificial general intelligence (AGI), with their impressive zero-shot capacity for user-tailored tasks, endowing them with immense potential across a range of applications. However, in the field of computer vision, despite the availability of numerous powerful vision foundation models (VFMs), they are still restricted to tasks in a pre-defined form, struggling to match the open-ended task capabilities of LLMs. In this work, we present an LLM-based framework for vision-centric tasks, termed VisionLLM. This framework provides a unified perspective for vision and language tasks by treating images as a foreign language and aligning vision-centric tasks with language tasks that can be flexibly defined and managed using language instructions. An LLM-based decoder can then make appropriate predictions based on these instructions for open-ended tasks. Extensive experiments show that the proposed VisionLLM can achieve different levels of task customization through language instructions, from fine-grained object-level to coarse-grained task-level customization, all with good results. It's noteworthy that, with a generalist LLM-based framework, our model can achieve over 60\% mAP on COCO, on par with detection-specific models. We hope this model can set a new baseline for generalist vision and language models. The demo shall be released based on this https URL. The code shall be released at this https URL.
    


 
Comments:
Technical Report


Subjects:

Computer Vision and Pattern Recognition (cs.CV)

Cite as:
arXiv:2305.11175 [cs.CV]


 
(or 
arXiv:2305.11175v2 [cs.CV] for this version)
          


 
 https://doi.org/10.48550/arXiv.2305.11175


Focus to learn more



                  arXiv-issued DOI via DataCite







Submission history From: Wenhai Wang [view email]       [v1]
        Thu, 18 May 2023 17:59:42 UTC (971 KB)
[v2]
        Thu, 25 May 2023 15:02:07 UTC (4,398 KB)



 

Full-text links:
Access Paper:


View a PDF of the paper titled VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks, by Wenhai Wang and 10 other authorsView PDFTeX SourceOther Formats


view license


 
    Current browse context: cs.CV


< prev

  |   
next >


new
 | 
recent
 | 2023-05

    Change to browse by:
    
cs




References & Citations

NASA ADSGoogle Scholar
Semantic Scholar




a
export BibTeX citation
Loading...




BibTeX formatted citation
×


loading...


Data provided by: 




Bookmark





 




Bibliographic Tools

Bibliographic and Citation Tools






Bibliographic Explorer Toggle



Bibliographic Explorer (What is the Explorer?)







Litmaps Toggle



Litmaps (What is Litmaps?)







scite.ai Toggle



scite Smart Citations (What are Smart Citations?)








Code, Data, Media

Code, Data and Media Associated with this Article






Links to Code Toggle



CatalyzeX Code Finder for Papers (What is CatalyzeX?)







DagsHub Toggle



DagsHub (What is DagsHub?)







GotitPub Toggle



Gotit.pub (What is GotitPub?)







Links to Code Toggle



Papers with Code (What is Papers with Code?)







ScienceCast Toggle



ScienceCast (What is ScienceCast?)











Demos

Demos






Replicate Toggle



Replicate (What is Replicate?)







Spaces Toggle



Hugging Face Spaces (What is Spaces?)







Spaces Toggle



TXYZ.AI (What is TXYZ.AI?)








Related Papers

Recommenders and Search Tools






Link to Influence Flower



Influence Flower (What are Influence Flowers?)







Connected Papers Toggle



Connected Papers (What is Connected Papers?)







Core recommender toggle



CORE Recommender (What is CORE?)





Author
Venue
Institution
Topic














        About arXivLabs
      



arXivLabs: experimental projects with community collaborators
arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.
Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.
Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.










Which authors of this paper are endorsers? |
    Disable MathJax (What is MathJax?)
    












About
Help





contact arXivClick here to contact arXiv
 Contact


subscribe to arXiv mailingsClick here to subscribe
 Subscribe











Copyright
Privacy Policy




Web Accessibility Assistance


arXiv Operational Status 
                    Get status notifications via
                    email
                    or slack





 





