arXiv:2403.05530 · v5
Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context
Gemini TeamPetko GeorgievVing Ian LeiRyan BurnellLibin BaiAnmol GulatiGarrett TanzerDamien VincentZhufeng PanShibo WangSoroosh MariooryadYifan DingXinyang GengFred AlcoberRoy FrostigMark OmernickLexi WalkerCosmin PaduraruChristina SorokinAndrea TacchettiColin GaffneySamira DarukiOlcan SercinogluZach GleicherJuliette LovePaul VoigtlaenderRohan JainGabriela SuritaKareem MohamedRory BlevinsJunwhan AhnTao ZhuKornraphop KawintiranonOrhan FiratYiming GuYujing ZhangMatthew RahtzManaal FaruquiNatalie ClayJustin GilmerJD Co-ReyesIvo PenchevRui ZhuNobuyuki MoriokaKevin HuiKrishna HaridasanVictor CamposMahdis MahdiehMandy GuoSamer HassanKevin KilgourArpi VezerHeng-Tze ChengRaoul de LiedekerkeSiddharth GoyalPaul BarhamDJ StrouseSeb NouryJonas AdlerMukund SundararajanSharad VikramDmitry LepikhinMichela PaganiniXavier GarciaFan YangDasha ValterMaja TrebaczKiran VodrahalliChulayuth AsawaroengchaiRoman RingNorbert KalbLivio Baldini SoaresSiddhartha BrahmaDavid SteinerTianhe YuFabian MentzerAntoine HeLucas GonzalezBibo XuRaphael Lopez KaufmanLaurent El ShafeyJunhyuk OhTom HenniganGeorge van den DriesscheSeth OdoomMario LucicBecca RoelofsSid LallAmit MaratheBetty ChanSantiago OntanonLuheng HeDenis TeplyashinJonathan LaiPhil CroneBogdan DamocLewis HoSebastian RiedelKarel LencChih-Kuan YehAakanksha ChowdheryYang XuMehran KazemiEhsan AmidAnastasia PetrushkinaKevin SwerskyAli KhodaeiGowoon ChenChris LarkinMario PintoGeng YanAdria Puigdomenech BadiaPiyush PatilSteven HansenDave OrrSebastien M. R. ArnoldJordan GrimstadAndrew DaiSholto DouglasRishika SinhaVikas YadavXi ChenElena GribovskayaJacob AustinJeffrey ZhaoKaushal PatelPaul KomarekSophia AustinSebastian BorgeaudLinda FrisoAbhimanyu GoyalBen CaineKris CaoDa-Woon ChungMatthew LammGabe Barth-MaronThais KagoharaKate OlszewskaMia ChenKaushik ShivakumarRishabh AgarwalHarshal GodhiaRavi RajwarJavier SnaiderXerxes DotiwallaYuan LiuAditya BaruaVictor UngureanuYuan ZhangBat-Orgil BatsaikhanMateo WirthJames QinIvo DanihelkaTulsee DoshiMartin ChadwickJilin ChenSanil JainQuoc LeArjun KarMadhu GurumurthyCheng LiRuoxin SangFangyu LiuLampros LamprouRich MunozNathan LintzHarsh MehtaHeidi HowardMalcolm ReynoldsLora AroyoQuan WangLorenzo BlancoAlbin CassirerJordan GriffithDipanjan DasStephan LeeJakub SygnowskiZach FisherJames BesleyRichard PowellZafarali AhmedDominik PaulusDavid ReitterZalan BorsosRishabh JoshiAedan PopeSteven HandVittorio SeloVihan JainNikhil SethiMegha GoelTakaki MakinoRhys MayZhen YangJohan SchalkwykChristina ButterfieldAnja HauthAlex GoldinWill HawkinsEvan SenterSergey BrinOliver WoodmanMarvin RitterEric NolandMinh GiangVijay BolinaLisa LeeTim BlythIan MackinnonMachel ReidObaid SarvanaDavid SilverAlexander ChenLily WangLoren MaggioreOscar ChangNithya AttaluriGregory ThorntonChung-Cheng ChiuOskar BunyanNir LevineTimothy ChungEvgenii EltyshevXiance SiTimothy LillicrapDemetra BradyVaibhav AggarwalBoxi WuYuanzhong XuRoss McIlroyKartikeya BadolaParamjit SandhuErica MoreiraWojciech StokowiecRoss HemsleyDong LiAlex TudorPranav ShyamElahe RahimtoroghiSalem HaykalPablo SprechmannXiang ZhouDiana MincuYujia LiRavi AddankiKalpesh KrishnaXiao WuAlexandre FrechetteMatan EyalAllan DafoeDave LaceyJay WhangThi AvrahamiYe ZhangEmanuel TaropaHanzhao LinDaniel ToyamaEliza RutherfordMotoki SanoHyunJeong ChoeAlex TomalaChalence Safranek-ShraderNora KassnerMantas PajarskasMatt HarveySean SechristMeire FortunatoChristina LyuGamaleldin ElsayedChenkai KuangJames LottesEric ChuChao JiaChih-Wei ChenPeter HumphreysKate BaumliConnie TaoRajkumar SamuelCicero Nogueira dos SantosAnders AndreassenNemanja RakićevićDominik GreweAviral KumarStephanie WinklerJonathan CatonAndrew BrockSid DalmiaHannah SheahanIain BarrYingjie MiaoPaul NatsevJacob DevlinFeryal BehbahaniFlavien ProstYanhua SunArtiom MyaskovskyThanumalayan Sankaranarayana PillaiDan HurtAngeliki LazaridouXi XiongCe ZhengFabio PardoXiaowei LiDan HorganJoe StantonMoran AmbarFei XiaAlejandro LinceMingqiu WangBasil MustafaAlbert WebsonHyo LeeRohan AnilMartin WickeTimothy DozatAbhishek SinhaEnrique PiquerasElahe DabirShyam UpadhyayAnudhyan BoralLisa Anne HendricksCorey FryJosip DjolongaYi SuJake WalkerJane LabanowskiRonny HuangVedant MisraJeremy ChenRJ Skerry-RyanAvi SinghShruti RijhwaniDian YuAlex Castro-RosBeer ChangpinyoRomina DattaSumit BagriArnar Mar HrafnkelssonMarcello MaggioniDaniel ZhengYury SulskyShaobo HouTom Le PaineAntoine YangJason RiesaDominika RogozinskaDror MarcusDalia El BadawyQiao ZhangLuyu WangHelen MillerJeremy GreerLars Lowe SjosAzade NovaHeiga ZenRahma ChaabouniMihaela RoscaJiepu JiangCharlie ChenRuibo LiuTara SainathMaxim KrikunAlex PolozovJean-Baptiste LespiauJosh NewlanZeyncep CankaraSoo KwakYunhan XuPhil ChenAndy CoenenClemens MeyerKaterina TsihlasAda MaJuraj GottweisJinwei XingChenjie GuJin MiaoChristian FrankZeynep CankaraSanjay GanapathyIshita DasguptaSteph Hughes-FittHeng ChenDavid ReidKeran RongHongmin FanJoost van AmersfoortVincent ZhuangAaron CohenShixiang Shane GuAnhad MohananeyAnastasija IlicTaylor TobinJohn WietingAnna BortsovaPhoebe ThackerEmma WangEmily CavenessJustin ChiuEren SezenerAlex KaskasoliSteven BakerKatie MillicanMohamed ElhawatyKostas AisoposCarl LebsackNathan ByrdHanjun DaiWenhao JiaMatthew WiethoffElnaz DavoodiAlbert WestonLakshman YagatiArun AhujaIsabel GaoGolan PundakSusan ZhangMichael AzzamKhe Chai SimSergi CaellesJames KeelingAbhanshu SharmaAndy SwingYaGuang LiChenxi LiuCarrie Grimes BostockYamini BansalZachary NadoAnkesh AnandJosh LipschultzAbhijit KarmarkarLev ProleevAbe IttycheriahSoheil Hassas YeganehGeorge PolovetsAleksandra FaustJiao SunAlban RrustemiPen LiRakesh ShivannaJeremiah LiuChris WeltyFederico LebronAnirudh BaddepudiSebastian KrauseEmilio ParisottoRadu SoricutZheng XuDawn BloxwichMelvin JohnsonBehnam NeyshaburJustin Mao-JonesRenshen WangVinay RamaseshZaheer AbbasArthur GuezConstant SegalDuc Dung NguyenJames SvenssonLe HouSarah YorkKieran MilanSophie BridgersWiktor GworekMarco TagliasacchiJames Lee-ThorpMichael ChangAlexey GuseynovAle Jakse HartmanMichael KwongRuizhe ZhaoSheleem KashemElizabeth ColeAntoine MiechRichard TanburnMary PhuongFilip PaveticSebastien CeveyRamona ComanescuRichard IvesSherry YangCosmo DuBo LiZizhao ZhangMariko IinumaClara Huiyi HuAurko RoyShaan BijwadiaZhenkai ZhuDanilo MartinsRachel SaputroAnita GergelySteven ZhengDawei JiaIoannis AntonoglouAdam SadovskyShane GuYingying BiAlek AndreevSina SamangooeiMina KhanTomas KociskyAngelos FilosChintu KumarColton BishopAdams YuSarah HodkinsonSid MittalPremal ShahAlexandre MoufarekYong ChengAdam BloniarzJaehoon LeePedram PejmanPaul MichelStephen SpencerVladimir FeinbergXuehan XiongNikolay SavinovCharlotte SmithSiamak ShakeriDustin TranMary ChesusBernd BohnetGeorge TuckerTamara von GlehnCarrie MuirYiran MaoHideto KazawaAmbrose SloneKedar SoparkarDisha ShrivastavaJames Cobon-KerrMichael SharmanJay PavagadhiCarlos ArayaKarolis MisiunasNimesh GhelaniMichael LaskinDavid BarkerQiujia LiAnton BriukhovNeil HoulsbyMia GlaeseBalaji LakshminarayananNathan SchucherYunhao TangEli CollinsHyeontaek LimFangxiaoyu FengAdria RecasensGuangda LaiAlberto MagniNicola De CaoAditya SiddhantZoe AshwoodJordi OrbayMostafa DehghaniJenny BrennanYifan HeKelvin XuYang GaoCarl SaroufimJames MolloyXinyi WuSeb ArnoldSolomon ChangJulian SchrittwieserElena BuchatskayaSoroush RadpourMartin PolacekSkye GiordanoAnkur BapnaSimon TokumineVincent HellendoornThibault SottiauxSarah CoganAliaksei SeverynMohammad SalehShantanu ThakoorLaurent ShefeySiyuan QiaoMeenu GabaShuo-yiin ChangCraig SwansonBiao ZhangBenjamin LeePaul Kishan RubensteinGan SongTom KwiatkowskiAnna KoopAjay KannanDavid KaoParker SchuhAxel StjerngrenGolnaz GhiasiGena GibsonLuke VilnisYe YuanFelipe Tiengo FerreiraAishwarya KamathTed KlimenkoKen FrankoKefan XiaoIndro BhattacharyaMiteyan PatelRui WangAlex MorrisRobin StrudelVivek SharmaPeter ChoySayed Hadi HashemiJessica LandonMara FinkelsteinPriya JhakraJustin FryeMegan BarnesMatthew MaugerDennis DaunKhuslen BaatarsukhMatthew TungWael FarhanHenryk MichalewskiFabio ViolaFelix de Chaumont QuitryCharline Le LanTom HudsonQingze WangFelix FischerIvy ZhengElspeth WhiteAnca DraganJean-baptiste AlayracEric NiAlexander PritzelAdam IwanickiMichael IsardAnna BulanovaLukas ZilkaEthan DyerDevendra SachanSrivatsan SrinivasanHannah MuckenhirnHonglong CaiAmol MandhaneMukarram TariqJack W. RaeGary WangKareem AyoubNicholas FitzGeraldYao ZhaoWoohyun HanChris AlbertiDan GarretteKashyap KrishnakumarMai GimenezAnselm LevskayaDaniel SohnJosip MatakInaki IturrateMichael B. ChangJackie XiangYuan CaoNishant RankaGeoff BrownAdrian HutterVahab MirrokniNanxin ChenKaisheng YaoZoltan EgyedFrancois GalileeTyler LiechtyPraveen KallakuriEvan PalmerSanjay GhemawatJasmine LiuDavid TaoChloe ThorntonTim GreenMimi JasarevicSharon LinVictor CotrutaYi-Xuan TanNoah FiedelHongkun YuEd ChiAlexander NeitzJens HeitkaemperAnu SinhaDenny ZhouYi SunCharbel KaedBrice HulseSwaroop MishraMaria GeorgakiSneha KuduguntaClement FarabetIzhak ShafranDaniel VlasicAnton TsitsulinRajagopal AnanthanarayananAlen CarinGuolong SuPei SunShashank VGabriel CarvajalJosef BroderIulia ComsaAlena RepinaWilliam WongWarren Weilun ChenPeter HawkinsEgor FilonovLucia LoherChristoph HirnschallWeiyi WangJingchen YeAndrea BurnsHardie CateDiana Gage WrightFederico PiccininiLei ZhangChu-Cheng LinIonel GogYana KulizhskayaAshwin SreevatsaShuang SongLuis C. CoboAnand IyerChetan TekurGuillermo GarridoZhuyun XiaoRupert KempHuaixiu Steven ZhengHui LiAnanth AgarwalChristel NganiKati GoshvadiRebeca Santamaria-FernandezWojciech FicaXinyun ChenChris GorgolewskiSean SunRoopal GargXinyu YeS. M. Ali EslamiNan HuaJon SimonPratik JoshiYelin KimIan TenneySahitya PotluriLam Nguyen ThietQuan YuanFlorian LuisierAlexandra ChronopoulouSalvatore ScellatoPraveen SrinivasanMinmin ChenVinod KoverkathuValentin DalibardYaming XuBrennan SaetaKeith AndersonThibault SellamNick FernandoFantine HuotJunehyuk JungMani VaradarajanMichael QuinnAmit RaulMaigo LeRuslan HabalovJon ClarkKomal JalanKalesha BullardAchintya SinghalThang LuongBoyu WangSujeevan RajayogamJulian EisenschlosJohnson JiaDaniel FinchelsteinAlex YakubovichDaniel BalleMichael FinkSameer AgarwalJing LiDj DvijothamShalini PalKai KangJaclyn KonzelmannJennifer BeattieOlivier DousseDiane WuRemi CrockerChen ElkindSiddhartha Reddy JonnalagaddaJong LeeDan Holtmann-RiceKrystal KallarackalRosanne LiuDenis VnukovNeera VatsLuca InvernizziMohsen JafariHuanjie ZhouLilly TaylorJennifer PrendkiMarcus WuTom EcclesTianqi LiuKavya KopparapuFrancoise BeaufaysChristof AngermuellerAndreea MarzocaShourya SarcarHilal DibJeff StanwayFrank PerbetNejc TrdinRachel SterneckAndrey KhorlinDinghua LiXihui WuSonam GoenkaDavid MadrasSasha GoldshteinWilli GierkeTong ZhouYaxin LiuYannie LiangAnais WhiteYunjie LiShreya SinghSanaz BahargamMark EpsteinSujoy BasuLi LaoAdnan OzturelCarl CrousAlex ZhaiHan LuZora TungNeeraj GaurAlanna WaltonLucas DixonMing ZhangAmir GlobersonGrant UyAndrew BoltOlivia WilesMilad NasrIlia ShumailovMarco SelviFrancesco PiccinnoRicardo AguilarSara McCarthyMisha KhalmanMrinal ShuklaVlado GalicJohn CarpenterKevin VillelaHaibin ZhangHarry RichardsonJames MartensMatko BosnjakShreyas Rammohan BelleJeff SeibertMahmoud AlnahlawiBrian McWilliamsSankalp SinghAnnie LouisWen DingDan PopoviciLenin SimicichLaura KnightPulkit MehtaNishesh GuptaChongyang ShiSaaber FatehiJovana MitrovicAlex GrillsJoseph PagadoraTsendsuren MunkhdalaiDessie PetrovaDanielle EisenbudZhishuai ZhangDamion YatesBhavishya MittalNilesh TripuraneniYannis AssaelThomas BrovelliPrateek JainMihajlo VelimirovicCanfer AkbulutJiaqi MuWolfgang MachereyRavin KumarJun XuHaroon QureshiGheorghe ComaniciJeremy WiesnerZhitao GongAnton RuddockMatthias BauerNick FeltAnirudh GPAnurag ArnabDustin ZelleJonas RothfussBill RosgenAshish ShenoyBryan SeyboldXinjian LiJayaram MudigondaGoker ErdoganJiawei XiaJiri SimsaAndrea MichiYi YaoChristopher YewSteven KanIsaac CaswellCarey RadebaughAndre ElisseeffPedro ValenzuelaKay McKinneyKim PatersonAlbert CuiEri Latorre-ChimotoSolomon KimWilliam ZengKen DurdenPriya PonnapalliTiberiu SoseaChristopher A. Choquette-ChooJames ManyikaBrona RobenekHarsha VashishtSebastien PereiraHoi LamMarko VelicDenese Owusu-AfriyieKatherine LeeTolga BolukbasiAlicia ParrishShawn LuJane ParkBalaji VenkatramanAlice TalbertLambert RosiqueYuchung ChengAndrei SozanschiAdam PaszkePraveen KumarJessica AustinLu LiKhalid SalamaBartek PerzWooyeol KimNandita DukkipatiAnthony BaryshnikovChristos KaplanisXiangHai ShengYuri ChervonyiCaglar UnluDiego de Las CasasHarry AskhamKathryn TunyasuvunakoolFelix GimenoSiim PoderChester KwakMatt MiecnikowskiVahab MirrokniAlek DimitrievAaron ParisiDangyi LiuTomy TsaiToby ShevlaneChristina KouridiDrew GarmonAdrian GoedeckemeyerAdam R. BrownAnitha VijayakumarAli ElqurshSadegh JazayeriJin HuangSara Mc CarthyJay HooverLucy KimSandeep KumarWei ChenCourtney BilesGarrett BinghamEvan RosenLisa WangQijun TanDavid EngelFrancesco PongettiDario de CesareDongseong HwangLily YuJennifer PullmanSrini NarayananKyle LevinSiddharth GopalMegan LiAsaf AharoniTrieu TrinhJessica LoNorman CasagrandeRoopali VijLoic MattheyBramandia RamadhanaAustin MatthewsCJ CareyMatthew JohnsonKremena GoranovaRohin ShahShereen AshrafKingshuk DasguptaRasmus LarsenYicheng WangManish Reddy VuyyuruChong JiangJoana IjaziKazuki OsawaCeline SmithRamya Sree BoppanaTaylan BilalYuma KoizumiYing XuYasemin AltunNir ShabatBen BariachAlex KorchemniyKiam ChooOlaf RonnebergerChimezie IwuanyanwuShubin ZhaoDavid SoergelCho-Jui HsiehIrene CaiShariq IqbalMartin SundermeyerZhe ChenElie BurszteinChaitanya MalaviyaFadi BiadsyPrakash ShroffInderjit DhillonTejasi LatkarChris DyerHannah ForbesMassimo NicosiaVitaly NikolaevSomer GreeneMarin GeorgievPidong WangNina MartinHanie SedghiJohn ZhangPraseem BanzalDoug FritzVikram RaoXuezhi WangJiageng ZhangViorica PatrauceanDayou DuIgor MordatchIvan JurinLewis LiuAyush DubeyAbhi MohanJanek NowakowskiVlad-Doru IonNan WeiReiko TojoMaria Abi RaadDrew A. HudsonVaishakh KeshavaShubham AgrawalKevin RamirezZhichun WuHoang NguyenJi LiuMadhavi SewakBryce PetriniDongHyun ChoiIvan PhilipsZiyue WangIoana BicaAnkush GargJarek WilkiewiczPriyanka AgrawalXiaowei LiDanhao GuoEmily XueNaseer ShaikAndrew LeachSadh MNM KhanJulia WiesingerSammy JeromeAbhishek ChakladarAlek Wenjiao WangTina OrnduffFolake AbuAlireza GhaffarkhahMarcus WainwrightMario CortesFrederick LiuJoshua MaynezAndreas TerzisPouya SamangoueiRiham MansourTomasz KępaFrançois-Xavier AubetAnton AlgymrDan BanicaAgoston WeiszAndras OrbanAlexandre SengesEwa AndrejczukMark GellerNiccolo Dal SantoValentin AnklinMajd Al MereyMartin BaeumlTrevor StrohmanJunwen BaiSlav PetrovYonghui WuDemis HassabisKoray KavukcuogluJeff DeanOriol Vinyals
首次提交 2024-03-09 02:54
版本更新 2024-12-17 01:39
雷达观测 2026-11-01 22:46
摘要
In this report, we introduce the Gemini 1.5 family of models, representing the next generation of highly compute-efficient multimodal models capable of recalling and reasoning over fine-grained information from millions of tokens of context, including multiple long documents and hours of video and audio. The family includes two new models: (1) an updated Gemini 1.5 Pro, which exceeds the February version on the great majority of capabilities and benchmarks; (2) Gemini 1.5 Flash, a more lightweight variant designed for efficiency with minimal regression in quality. Gemini 1.5 models achieve near-perfect recall on long-context retrieval tasks across modalities, improve the state-of-the-art in long-document QA, long-video QA and long-context ASR, and match or surpass Gemini 1.0 Ultra's state-of-the-art performance across a broad set of benchmarks. Studying the limits of Gemini 1.5's long-context ability, we find continued improvement in next-token prediction and near-perfect retrieval (>99%) up to at least 10M tokens, a generational leap over existing models such as Claude 3.0 (200k) and GPT-4 Turbo (128k). Finally, we highlight real-world use cases, such as Gemini 1.5 collaborating with professionals on completing their tasks achieving 26 to 75% time savings across 10 different job categories, as well as surprising new capabilities of large language models at the frontier; when given a grammar manual for Kalamang, a language with fewer than 200 speakers worldwide, the model learns to translate English to Kalamang at a similar level to a person who learned from the same content.