LARGE LANGUAGE MODELS MEET COMPUTER VISION; A BRIEF
SURVEY
RabyHamadi
ArtificialIntelligenceDepartment,Tahakom
rhamadi@tahakom.com
ABSTRACT
Recently,theintersectionofLargeLanguageModels(LLMs)andComputerVision(CV)hasemerged
asapivotalareaofresearch,drivingsignificantadvancementsinthefieldofArtificialIntelligence(AI).
Astransformershavebecomethebackboneofmanystate-of-the-artmodelsinbothNaturalLanguage
Processing(NLP)andCV,understandingtheirevolutionandpotentialenhancementsiscrucial. This
surveypaperdelvesintothelatestprogressionsinthedomainoftransformersandtheirsubsequent
successors,emphasizingtheirpotentialtorevolutionizeVisionTransformers(ViTs)andLLMs. This
surveyalsopresentsacomparativeanalysis,juxtaposingtheperformancemetricsofseveralleading
paidandopen-sourceLLMs, sheddinglightontheirstrengthsandareasofimprovementaswell
asaliteraturereviewonhowLLMsarebeingusedtotacklevisionrelatedtasks. Furthermore,the
surveypresentsacomprehensivecollectionofdatasetsemployedtotrainLLMs,offeringinsights
intothediversedataavailabletoachievehighperformanceinvariouspre-traininganddownstream
tasks of LLMs. The survey is concluded by highlighting open directions in the field, suggesting
potentialvenuesforfutureresearchanddevelopment. Thissurveyaimstounderscorestheprofound
intersectionofLLMsonCV,leadingtoaneweraofintegratedandadvancedAImodels.
Keywords LargeLanguageModels·VisionTransformer·RetentionNetworks
1 Introduction
ThefieldofAIiswitnessingagreatfusionofCVandNLP.AtthemiddleofthisexcitingcombinationareLLMs,
whichhavecometobeakeyparticipantinAI.Thesemodels,thatwerefirstcreatedtounderstandandproducehuman
language,areactuallyexpandingtoencompasstheinterpretationofvisualdata. Thisshowshowflexibleandadaptable
LLMsare,astheybegintocombinetheunderstandingoftextwithseeingandinterpretingvisualdata. LLMs,along
withCVaresettorevolutionizethewaymachinesunderstandanddescribethevisibleworld. ThisfusionofLLMs
andCVisleadingtoanerawhereAIsystemscanseetheworld,understandit,andcommunicateaboutitnearlylike
humansdo. Bybringingtogetherthesetechnologies,wecanmakethewayhumansandmachinesinteractsmootherand
morenatural. Forexample,insecurity,thiscouldmeansystemsthatnotonlyseesomethingunusualbutalsoexplainit
clearly,helpingtodealwiththreatsfaster. Inhealthcare,combiningLLMsandcomputervisioncouldleadtobetter
diagnosisandtreatmentbyjoiningvisualinformationwithawiderangeofmedicalknowledge. Inretail,AIcould
watchoverinventorylevelsanduseLLMstomakeusefulreportsandpredictions. Inmanufacturing,thesetechnologies
couldimprovehowwecheckforproductqualitybyspottingdefectswithcomputervisionandusingLLMstogive
detailedadviceonfixingthem. Lookingahead,thefusionofLLMsandcomputervisionissettobealandmarkin
AI’sprogression,bringingusclosertocreatinghighlyautonomousdigitalassistantsthatcanengagewiththeworld
comprehensively. ThisneweraofAIenvisionsmachinesasproactiveparticipants,interpretingandnarratingthevisual
worldaroundthem.
ApivotaladvancementinthefieldsofNLPandCVwastheinventionofthetransformerarchitecture[1],whichwas
crucialatmanaginglongsequencesinNLPtasksthroughtheinnovativeattentionmechanism[1]. Thismechanism
allowsthemodeltoweightheimportanceofdifferentsegmentsofinputdata. Thebreakthroughcamewhenresearchers
statedthatiftransformerscouldinterpretlanguageassequencesofwords,theymightalsoprocessimagesassequences
3202
voN
82
]VC.sc[
1v37661.1132:viXra