@(iUSrSSKJr SSKrSSKrSSKrSSKrSSKrSSKJ r SSK J r J r SSK Jr SSKJr SSKJr SS KJr SS KJr SSKrSSKrSSKrSS KJr SS KJr S SKJrJ r J!r!J"r"J#r#J$r$J%r% \RL"\'5r(\(a0SSK)J*r*J+r+J,r, SS KJr- SSKJ.r.J/r/ SSK0J1r1 SSKJ2r2 SSK3J4r4 S SKJ5r5 Sr6Sr7\"SS9"SS55r8\"SSSS9"SS55r9S'Sjr:\"SS 55r;\;Rx4S(S!jjr=\(a\/"S"5r>S)S#jr?S$r@\@S*S%j5rA\@S+S&j5rBg),a  Sharded repodata subsets. Traverse dependencies of installed and to-be-installed packages to generate a useful subset for the solver. The algorithm developed here is a direct result of the following CEP: - https://conda.org/learn/ceps/cep-0016 (Sharded Repodata) In this algorithm we treat a (channel, package name) as a node, its dependencies as edges. We then traverse all edges to discover all reachable (channel, package name) tuples. The solver should be able to find a solution with only this subset. This subset is overgenerous since the user is unlikely to want to install very old packages and their dependencies. If this is too slow, we could deploy heuristics that automatically ignore older package versions. We could also allow the user to configure minimum versions of common packages and ignore older versions and their dependencies, falling back to a full solve if unsatisfiable. We treat both sharded and monolithic repodata as if they were made up of per-package shards, computing a subset of both. This is because it is possible for the monolithic repodata to mention packages that exist in the true sharded repodata but would not be found by only traversing the shards. We treat all repodata as sharded, even if no actual sharded repodata has been found. ## Example usage The following constructs several repodata (`noarch` and `linux-64`) from a single channel name and a list of root packages: ``` from conda.models.channel import Channel from conda_libmamba_solver.shards_subset import build_repodata_subset channel = Channel("conda-forge-sharded/linux-64") channel_data = build_repodata_subset(["python", "pandas"], [channel.url()]) repodata = {} for url in channel_data: repodata[url] = channel_data.build_repodata() # ... this is what's fed to the solver ``` ) annotationsN)deque)FutureThreadPoolExecutor)suppress) dataclass)Path) SimpleQueue) TYPE_CHECKING) shards_cache)AnnotatedRawShard)ZSTD_MAX_SHARD_SIZEShards_shards_connectionsbatch_retrieve_from_cachebatch_retrieve_from_networkfetch_channelsshard_mentioned_packages)IterableIteratorSequence)LiteralTypeVar)Channel) ShardCache) ShardDict) ShardBase T)ordercz\rSrSr%\R rS\S'SrS\S'Sr S\S'Sr S \S 'Sr S\S 'SS jr S r g)NodehintdistancestrpackagechannelFboolvisited shard_urlcX[URURUR5$N)NodeIdr)r*r-selfs [/data/cameron/miniconda/lib/python3.13/site-packages/conda_libmamba_solver/shards_subset.pyto_id Node.to_idpsdllDLL$..AAN)returnr0)__name__ __module__ __qualname____firstlineno__sysmaxsizer&__annotations__r)r*r,r-r4__static_attributes__r7r6r3r#r#hs>KKHcGSGSGTIsBr6r#)r!eqfrozenc>\rSrSr%S\S'S\S'SrS\S'SrSrg ) r0tr(r)r*r'r-cZ[URURUR45$r/)hashr)r*r-r1s r3__hash__NodeId.__hash__zs T\\4<<@AAr6r7N)r9r:r;r<r?r-rGr@r7r6r3r0r0ts L LIsBr6r0c ## UHLnUHCnX#;dM [SX#RURU5S9nUR5nXT4v ME MN g7f)zA Yield (NodeId, Node) for all root packages found in shardlikes. r)r-N)r#urlr-r4) root_packages shardlikesr) shardlikenodenode_ids r3_nodes_from_packagesrP~sR !#I#Aw ATATU\A]^**,m# $!s AAAc\rSrSr%S\S'S\S'SrSSjr\SSj5rSS jr SS jr S r S r S r SSjrSSjrSrg)RepodataSubsetzdict[NodeId, Node]nodeszSequence[ShardBase]rL pipelinedc20Ul[U5Ulgr/)rTlistrL)r2rLs r3__init__RepodataSubset.__init__s z*r6c [USU35$)zH Return True if this class provides the named shard traversal strategy. reachable_)hasattr)clsstrategys r3 has_strategyRepodataSubset.has_strategys sj 344r6c## [5nURHnURU;aMURUR5n[ U5Hn[ XSR 5nX`R;dM)[URS-XSR 5URU'URUv XR;dMqURU5 M M g7f)zd Retrieve all unvisited neighbors of a node Neighbors in the context are dependencies of a package rN) setrLr) fetch_shardrr0rJrTr#r&add)r2rN discoveredrMshardr)rOs r3 neighborsRepodataSubset.neighborss U I||9,)) E4E: --8**,*.t}}q/@'==*YDJJw'**W--0"w/;)sA7C=ACCc#H# URU5H nUS4v M g7f)z8 All nodes that can be reached by this node, plus cost. rN)rg)r2rNns r3outgoingRepodataSubset.outgoings#%AQ$J&s "c[[XR55Ul[ URR 55nURVs/sHn[ U[5(dMUPM nnU(aUVs1sH!oUR(aMURiM# nnU(a [U[U55n[U5 [U5n[U5Hkn UR5nUR(aM&SUlUR!U5H)upU R(aMUR#U 5 M+ Mm U(aMggs snfs snf)z Fetch all packages reachable from `root_packages`' by following dependencies using the "breadth-first search" algorithm. Update associated `self.shardlikes` to contain enough data to build a repodata subset. TN)dictrPrLrTrvalues isinstancerr,r)rsortedrlenrangepopleftrkappend) r2rK node_queuesshardedrN to_retrieve not_in_cache level_size_ next_nodes r3 reachable_bfsRepodataSubset.reachable_bfss .}ooNO 4::,,./ "ooGoAv1F1oG4>SJDll<4<TRTT5upU(a"TRU5 TRU5 U(a)TRSU55 TRU5 [U5[U5-$)zw Find shards we already have and those we need. Submit those need to cache_in_queue, those we have to shard_out_queue. c3*# UH upUv M g7fr/r7).0rOr|s r3 ERepodataSubset.pipelined_main_thread..pump..*s @4ZW4s) drain_pendingupdaterrr)haveneedr in_flightpendingr2rshardlikes_by_urls r3pump2RepodataSubset.pipelined_main_thread..pump sq ++G5FGJD  &""4(  @4 @@##D)t9s4y( (r6TrtimeoutNzShard timeout %s, pump_count=%dzpending: %s...c38# UHn[U5v M g7fr/r(rrOs r3r7RepodataSubset.pipelined_main_thread..;s2WwG3w< %%goou=6Nu6UV#-95G OP""4(%)"Yg7@H;; !V  ;XR *F2Ww2W,WX[Y[,\] ,f4[QZ4[.[\_]_.`a +s4::7 79N9N9PQ 9>;R;R;TU 79N9N9PQyIICDA >>&DXJO##&w<. S^0G0G0I/JL ) s%E.E3E33DT"X/UQ70UD6$![a,nURS5 URU5 SnAgSnAff=fr/)rr)r out_queuerkwargsefuncs r3wrapper#exception_to_queue..wrappersH =d=f= =  LL  MM!   s  A"AA) functoolswraps)rrs` r3exception_to_queuers' __T Nr6cUR5n[U5HnURUVs/sHoURPM sn5n/n/nUHJnUR UR5=n (aUR XY45 M9UR U5 ML U(aUR U5 U(dMUR U5 M UR S5 UR S5 gs snf)a Fetch batches of shards from cache until in_queue sees None. Enqueue found shards to shard_out_queue, and not found shards to network_out_queue. When we see None on in_queue, send None to both out queues and exit. Args: in_queue: NodeId (URLs) to fetch. shard_out_queue: fetched shards sent to queue. network_out_queue: cache misses forwarded to queue. Same queue is network_fetch_thread's in_queue. cache: used to retrieve shards. N)copyrretrieve_multipler-rrur) rrnetwork_out_queuerrrOcachedfound not_foundrfs r3rrs( JJLE.x8((8)T8*;*;8)TU13"$ G 7#4#455u5 g-.  )     ! !) , 5    &!9$$%*UsC7 c^^^^ ^ ^ ^TR5m[R"[S9m UVs0sHoDRU_M snmSSjm S U U U4SjjnS UU U4SjjnS U4Sjjn[ [ 5S9m [T5HDnUH;n [U [5(a U"U 5 M"U"U 5n U RU5 M= MF SSS5 gs snf!,(df  g=f) ar Fetch shards from the network that are received on in_queue, until we see None. Unhandled exceptions also go to shard_out_queue, and exit this thread. Args: in_queue: NodeId (URLs) to fetch. shard_out_queue: fetched shards sent to queue. cache: once shards are decoded they are stored in cache. shardlikes: list of (network-only) shard index objects. )max_window_sizecdURU5nUR5 URnXU4$r/)rraise_for_statuscontent)rwrJrOresponsedatas r3fetch#network_fetch_thread..fetchs155:!!#T!!r6c>TURn[U[5(d [S5eURnTURR UR 5nTRTX#U5$)Nz.network_fetch_thread got non-network shardlike)r*rpr TypeErrorsessionr-r)submit)rOrMrrJexecutorrrs r3r$network_fetch_thread..submit$se%goo6 )V,,LM M##0::7??KugG<UR5upn[RSU[U55 [R "TR U[S95nTR[XRU55 TRX$4/5 g)NzFetch thread got %s (%s bytes))max_output_size) resultrrrrmsgpackloads decompressrinsertr r)r)futurerJrOrrfrdctxrs r3 handle_result+network_fetch_thread..handle_result-sw#]]_d 2CTC#== OOD2EO F   &sOOTBCg-./r6c*>TRU/5 gr/)r)rrs r3result_to_in_queue0network_fetch_thread..result_to_in_queue;s  fXr6) max_workersN)rJr(rOr0)rOr0)rr) r zstandardZstdDecompressorrrJrrrrpradd_done_callback)rrrrLrwrrrnode_ids_and_resultsnode_id_or_resultrrrrrs``` @@@@r3rrs& JJLE  % %6I JD+56:a:6" == 0 0 (;(= >($>x$H %9!/88!"34#$56F,,-?@ &:%I ? >K7J ? >sC?AC!! C/)rKz list[str]rLrr8zIterator[tuple[NodeId, Node]])rKrrzdict[str, Channel]rzLiteral['bfs', 'pipelined']r8r)rzQueue[Sequence[_T] | None]r8zIterator[Sequence[_T]])rQueue[Sequence[NodeId] | None]rzr.s-^# 9! .@!<<*',== #%  BBB $t,BB-B $ $*= $" $ }(}( }(F.<-L-L  + 8 B(<  (,(Q(6(  ((V@A5@AM@A @A @A@Ar6