TAHAKOM
Figure1: DualformofRetNet. “GN”isshortforGroupNorm[3].
ofpixelpatches. ThisinsightgaverisetoVT,astepforwardinCVmethodologies. Inoursurvey,weaimtopresentan
exhaustiveexplorationofLLMsastheyventureintotherealmofCV.Wewilldelveintothenuancesoftransformer
architectures andtheir successors, highlighting their rolein shaping anintegrated AI future. Our literature review
willmeticulouslycoverrecentworksandgroundbreakingresearchthathavechartedthecourseforLLMsinvisual
applications. Acomparativeanalysiswillilluminatethestrengthsandpotentialenhancementsofbothpaidandopen-
sourcemodels. Additionally,wewillexplorethediversedatasetsthatareusedintheliteraturetotrainthesemodels.
ThesurveyisthenconcludedbysomeofthechallengesthatfaceLLMsandsomeopenresearchdirections.
2 Background
2.1 RNNs
RecurrentNeuralNetworks(RNNs)[2]areaclassofartificialneuralnetworksdesignedforprocessingsequencesof
data. Unliketraditionalfeedforwardneuralnetworks,RNNspossessloopsthatallowinformationtobepassedfromone
stepinthesequencetothenext,makingtheminherentlysuitedfortaskswheretemporaldynamicsandcontextfrom
earlierstepsarecrucial. ThefoundationalideabehindRNNsistheconceptofmemory. Ateachstepinasequence,an
RNNtakesinanewinputandtheprevioushiddenstate(ormemory)toproduceanewhiddenstateandanoutput. This
recurrentnatureallowsRNNstomaintainaformof"memory"aboutpreviousstepsinthesequence,enablingthemto
capturepatternsovertime.
2.2 Transformers
Introduced in 2017 by Vaswani et al. [4], Transformers are a type of deep learning models that excel at handling
sequentialdata, makingthemparticularlysuitablefornaturallanguageprocessingtasks. Thekeybreakthroughin
Transformers lies within their self-attention mechanism [5], which allows the model to assess the significance of
differentsegmentswithinaninputsequencerelativetooneanother. Thisuniquecapabilityenablesthemodeltocapture
complexpatternsanddependenciesoverlongdistancesinthedata. UnlikeconventionalRNNs,Transformersprocess
inputinformationconcurrentlyratherthansequentially,resultinginnotableenhancementsinbothtrainingefficiency
andscalability. Thus,itenablesresearcherstobuildlargemodelswithbillionsofparametersthatcangeneratelong
coherentandcontextualstories,answerquestionsaccurately,andexecutetasksthatrequirelogicandunderstanding.
Whiletheyareefficientduringtrainingduetotheirabilitytohandleparallelization,oneoftheirdrawbacksisthatthey
canbeslowduringinference. Thisisbecausetheself-attentionmechanisminTransformersrequirescomputationover
allelementsoftheinputsequence,leadingtoaquadraticincreaseincomputationasthesequencelengthgrows. This
canbeparticularlychallengingforreal-timeapplicationsorscenarioswherequickresponsetimesareessential.
2