TAHAKOM
enhancingthetrainingdataforVLMs.Additionally,bypromptingLLMs,theauthorsgeneratetextanalogies—sentences
thatconveysimilarmeaningsbutwithvariedwording. Thismethodofstructuredtextualdatamanipulation,poweredby
LLMs,aimstoemphasizetheimportanceofSVLCinVLMtraining,leadingtosignificantimprovementsinthemodel’s
comprehensionoftheseconcepts. Throughevaluationsonmultipledatasets,theapproachdemonstratesimprovements
ofupto15%intheVLMs’understandingofSVLCs,whileensuringminimaldegradationintheirzero-shotobject
recognitioncapabilities.
Yining et Al. [18], made a significant push towards integrating 3D spatial understanding into LLMs. Their work
introduceda3D-LLMframework,designedexplicitlyfortrainingLLMstocomprehendandrespondto3Dscenes
andobjects. Thisframeworkingeniouslyextracts3Dfeaturesfrom2Dmulti-viewimagesandcombinesthemwith
languagepromptstogeneraterelevantresponses. Oneoftheprimarychallengesaddressedbytheauthorswasthe
difficultyintraining3D-LLMsfromscratch,primarilyduetotheabsenceofacomprehensive3D-languagedataset.
Tocircumventthis, theyproposedamethodthatleveragespre-trainedimageencoderstoextractfeaturesfrom2D
imagesandmapthesetothe3Ddomain. Thesemapped3Dfeaturesarethenseamlesslyintegratedintopre-trained2D
vision-languagemodels,whichsubsequentlyserveasthefoundationalarchitectureforthe3D-LLMs. Furthermore,the
studyemployedChatGPT[19]togenerateadiverserangeof3D-languagedata,emphasizingitsabilitytoformholistic
3Ddescriptionsfrommultipleimageviews. Theresultsfromthisresearchunderscoredthepotentialof3D-LLMs,
showcasingtheircapabilitytogenerateanswerscloselyalignedwithground-truthdataandperformintricatevisual
reasoningtasks. Thisinnovativeapproachnotonlybridgestheexistinggapbetween2Dimageunderstandingand3D
spatialreasoningbutalsosetsapromisingdirectionforfutureadvancementsinAI-driven3Dscenecomprehension.
Zhenfeietal.,[20]introduceanadvancementinmulti-modallargelanguagemodels(MLLMs)byincorporatingpoint
clouddata,presentingtheLAMM-DatasetandLAMM-Benchmarkspecificallydesignedfor2Dimageand3Dpoint
cloudunderstanding. Theseresourcesareintendedtosupporthigh-levelvisiontasksacrossbothdimensions. They
validatetheeffectivenessoftheirdatasetandbenchmarkthroughextensiveexperiments. Furthermore,theydetailthe
processofcreatinginstruction-tuningdatasetsandbenchmarksforMLLMs,aimingtoexpeditefutureresearchand
expansionintonewdomains,tasks,andmodalities. ThepaperalsointroducesanoveltrainingframeworkforMLLMs
thatisoptimizedfortheinclusionofadditionalmodalities. Alongsidethisframework,theyprovidebaselinemodelsas
wellascomprehensiveexperimentalobservationsandanalysestoserveasafoundationforacceleratingsubsequent
researchinthefield.
Inthispaper[21], theauthorsaddressthechallengeofproductunderstandinginthecontextofonlineshopping, a
taskthatinvolvesrespondingtoawiderangeofqueriesbasedonmulti-modalproductinformation. Unliketraditional
methodsthatusedifferentmodelarchitecturesforeachsub-task,theauthorsintroducePUMGPT,aunifiedvision-
languagemodeldesignedtohandleallaspectsofproductunderstandingwithinasinglemodelstructure. Toeffectively
combinevisualandtextualdata,theyproposeanoveltechniquecalledLayer-wiseAdapters(LA),whichalignsvision
andtextrepresentationsmoreefficientlywithfewervisualtokensandallowsforparameter-efficientfine-tuning. This
fine-tuningcapabilityenablesPUMGPTtoadaptquicklytonewtasksandproducts. Theauthorsalsocreatediverse
productinstructiondatasetsusinginstructiontemplatesandenhancethemodel’sperformanceandgeneralizationby
incorporatingopen-domaindatasetsinthetrainingprocess. PUMGPTisthoroughlyevaluatedandshowssuperior
performanceacrossavarietyofproductunderstandingtasks,includingproductcaptioning,categoryquestion-answering,
attributeextraction,attributequestion-answering,andfree-formquestion-answeringaboutproducts.
Inthiswork[22],theauthorstacklethelimitationsofvisionfoundationmodels(VFMs)incomputervision,which,
despitetheirpower,lacktheopen-endedtaskflexibilityoflargelanguagemodels(LLMs).TheyintroduceVisionLLM,a
novelLLM-basedframeworkdesignedtohandlevision-centrictasks. VisionLLMtreatsimagesasiftheywereaforeign
language,aligningvisiontaskswithlanguagetasksthatcanbedefinedandmanipulatedthroughlanguageinstructions.
ThisallowsforahighdegreeoftaskcustomizationusinganLLM-baseddecoderthatinterpretstheseinstructions
tomakepredictionsforavarietyofopen-endedtasks. Theauthorsdemonstratethroughextensiveexperimentsthat
VisionLLMcanhandlearangeofcustomizations,fromdetailedobject-leveltobroadertask-leveladjustments,with
impressiveresults. Remarkably,themodelachievesover60%meanAveragePrecision(mAP)ontheCOCOdataset,
whichiscompetitivewithmodelsdedicatedtodetectiontasks. TheauthorsaimforVisionLLMtoestablishanew
standardforgeneralistmodelsthatintegratevisionandlanguage.
5 Datasets
AsuiteofdatasetswasintroducedtoenhancethetrainingandevaluationofLargeLanguageModels(LLMs). These
datasetstargetvariousfunctionalitiesofLLMssuchasAPIinteraction,reasoning,extendedcontextcomprehension,
andlanguageunderstanding. Aconciseoverviewofthelatestdatasetisprovidedbelow:
7