8j&ddlmZddlmZmZmZmZddlmZm Z m Z m Z er ddl m Z mZmZmZedZdZ GddeeZd d d d  dd Zd d d d  dd Zed d d d  ddZed d d d  ddZd d d d  ddZed d d d  ddZed d d d  ddZd d d d  ddZddZddZy ) ) annotations) TYPE_CHECKINGProtocolTypeVaroverload)KeyValueStoreKey ListOfKeysListOfRequestsRequest) AsyncIterator AwaitableCallableIteratorTiceZdZUdZded<y)HasItemsaStructural contract for a single page of results from a paginated API endpoint. Implementations must expose `items`. They may optionally expose `count` - the number of rows the API scanned to produce this page, which `len(items)` can land below (filters drop items) or above (`unwind` splits one row into several items). The iterator helpers consult `count` opportunistically via `getattr` for offset bookkeeping and fall back to `len(items)` when it is absent. zlist[T]itemsN)__name__ __module__ __qualname____doc____annotations__O/var/www/html/GAP/venv/lib/python3.12/site-packages/apify_client/_pagination.pyrrs NrrN)limitoffset chunk_sizec#K|xsd}|xsd}|xsd}d} t|||}||||z} | jEd{t| |} || z }| r|r||k\ryJ7 w)aYield individual items from offset-based paginated API responses. The `callback` is invoked lazily to fetch each page from the API. It must accept `limit` and `offset` keyword arguments and return an object whose `items` attribute is a list. If the object also exposes a `count` attribute, it is used for offset bookkeeping - `_page_scanned_rows` describes how the next offset is derived. Iteration stops when a page scans no rows or when the user-requested `limit` is reached. A page can scan rows while returning no items - filters like `clean` drop items from `items` but still count toward `count` - so terminating on scanned rather than returned rows keeps the iterator advancing across fully-filtered pages. The `total` field is intentionally not consulted, because it can change between calls. Args: callback: Function returning a single page of items. limit: Maximum total number of rows scanned across all pages. On the dataset items endpoint `unwind` can turn one row into several items, so more items than this can be yielded. `None` or `0` means no limit. offset: Starting offset for the first page. chunk_size: Per-page cap, sent to the API as its `limit`. `None` or `0` lets the API decide. rrrN_next_page_limitr_page_scanned_rows) callbackrrreffective_chunkinitial_offset initial_limit fetched_items page_limit current_page page_scanneds rget_items_iteratorr,s2!oAONJQMM %m]OT !M1  %%%%), C %  -=2P   &s?A$A"!A$cK|xsd}|xsd}|xsd}d} t|||}||||zd{} | jD]} |  t| |} || z }| r|r||k\ryT76w)z|Async variant of :func:`get_items_iterator`. The `callback` must be an awaitable returning a single page of items. rr Nr!) r$rrrr%r&r'r(r)r*itemr+s rget_items_iterator_asyncr/Ls!oAONJQMM %m]OT %!M1  !&& DJ *, C %  -=2P   s3A.A,7A.)cursorrrcyNrr$r0rrs rget_cursor_iteratorr4ls"%rcyr2rr3s rr4r4tsrc#K|xsd}|xsd}d} |t||||}|jEd{|t|jz }t|tr |j n |j }||r||k\ryt7Ow)aYield individual items from a cursor-paginated API response. This iterator supports the two API responses that use cursor pagination. `ListOfKeys` is used for key-value store keys, while `ListOfRequests` is used for request queue requests. Pagination continues until either: - the API returns no next cursor, or - the requested `limit` is reached. An empty page does not explicitly stop the iteration. In practice, both supported endpoints return a next cursor only when the current page contains items, so an empty page always has a `None` cursor and naturally ends the iteration. The endpoints determine the next cursor differently: - For key-value store keys, the cursor is the last key returned on the current page. - For request queue requests, a cursor is returned only when the current page is full. Args: callback: Function that returns one page of items and accepts `cursor` and `limit` keyword arguments. cursor: Cursor to use for the first request. If `None`, iteration starts from the beginning. limit: Maximum total number of items to yield across all pages. chunk_size: Maximum number of items to request in a single API call. rrr0Nr"rlen isinstancer next_exclusive_start_key next_cursor)r$r0rrr%r'r(r*s rr4r4|s@!oAOJQMM "=-Q  %%%%\//00 5? j5YL 1 1_k_w_w  >m 0N   &s4BBABcyr2rr3s rget_cursor_iterator_asyncr>s'*rcyr2rr3s rr>r>s!rc(K|xsd}|xsd}d} |t||||d{}|jD]}| |t|jz }t|tr |j n |j }||r||k\ry~7ew)z-Async variant of :func:`get_cursor_iterator`.rr7Nr8) r$r0rrr%r'r(r*r.s rr>r>s!oAOJQMM %"=-Q  !&& DJ  \//00 5? j5YL 1 1_k_w_w  >m 0N   s(BBA&Bc4|s|S||z }|s|St||S)a%Compute the `limit` value for the next API call. `0` means no limit on the wire (matches the Apify API contract). When both an overall `initial_limit` and a per-page `effective_chunk` are set, the call is clamped to whichever is smaller; if either is unset (`0`), the other wins. )min)r'r(r% remainings rr"r"s-  -I  y/ **rcvtt|ddt|j}|r t ||S|S)aCompute how far the offset advances past `page`, in dataset rows. Neither reported number is right on its own. `count` follows the rows the API scanned, but it is derived from a dataset's item count, which is incremented by a throttled write and so lags a fresh push. `len(items)` counts the items the API shaped out of those rows: filters (`clean`, `skip_empty`, `skip_hidden`) drop some, and `unwind` splits one row into several. The larger of the two absorbs a `count` that lags behind the items returned, and capping it at the rows the call asked for keeps an unwound page from advancing past rows the next call would then never read. The cap is a valid bound because the endpoint applies the `limit` it is sent verbatim; on a page covering fewer rows than that, the advance can still overshoot into rows a concurrent push appends afterwards. A `requested_limit` of `0` means the call sent no limit, leaving the advance unbounded. countr)maxgetattrr9rrB)pagerequested_limit scanned_rowss rr#r#s5wtWa0#djj/BL1@3|_ -RlRr) r$zCallable[..., HasItems[T]]r int | NonerrKrrKreturnz Iterator[T]) r$z%Callable[..., Awaitable[HasItems[T]]]rrKrrKrrKrLzAsyncIterator[T]) r$zCallable[..., ListOfKeys]r0 str | NonerrKrrKrLzIterator[KeyValueStoreKey]) r$zCallable[..., ListOfRequests]r0rMrrKrrKrLzIterator[Request]) r$z*Callable[..., ListOfKeys | ListOfRequests]r0rMrrKrrKrLz.Iterator[KeyValueStoreKey] | Iterator[Request]) r$z$Callable[..., Awaitable[ListOfKeys]]r0rMrrKrrKrLzAsyncIterator[KeyValueStoreKey]) r$z(Callable[..., Awaitable[ListOfRequests]]r0rMrrKrrKrLzAsyncIterator[Request]) r$z5Callable[..., Awaitable[ListOfKeys | ListOfRequests]]r0rMrrKrrKrLz8AsyncIterator[KeyValueStoreKey] | AsyncIterator[Request])r'intr(rNr%rNrLrN)rHz HasItems[T]rIrNrLrN) __future__rtypingrrrrapify_client._modelsrr r r collections.abcr r rrrDEFAULT_CHUNK_SIZErr,r/r4r>r"r#rrrrTs"==VVLL CL x{ ! *(* *  *  *  *`! 3       @ ! %'% %  %  %  % % ! +        ! 181 1  1  1 4 1h ! *2* *  *  * % * * ! !6! !  !  !  ! !! C      > : + Sr