Batching

[kelleyvanevert]

So yesterday I wrote up some thoughts on batching (see https://github.com/tannerlinsley/react-query/discussions/364), but this was too eager of me.

For proper support of batching, it would be really working against the grain of the library. Some examples of the weird situations I find myself in due to trying to "wrap" it around:

• Needing to refetch a part of what was previously fetched in a batch

• I define (+ type) API methods in definitional objects, and save some query config with them, e.g. stale time and refetch options. When I batch, I have to figure out how to combine these options somehow:

  const { data } = useWrappedApi({
    apiMethod: batch([A, B, C]), // magically has to "combine" these query configs somehow
    params: [pA, pB, pC]
  });

• In other places in the UI, fetching part of what was possibly elsewhere batch-fetched (e.g. with prefetching as in my previous discussion post).

This all brings me to the "obvious" conclusion that batching, if at all, should be implemented in react-query itself, not in my wrapper logic. I've seen this elsewhere as well, in the graphql world where people are trying to get data specifications + necessary queries to be a part of component design. Then, e.g., any number of queries that are requested within 50 ms of each other would be "batched" by default.

I think this would be generally doable, apart from effort (which I'm willing to put in), and it would amount to:

• Letting query request acculumate for a certain duration (config specified, and defaulting to 0)
• Somehow letting the developer's config decide if/whether to batch the acculumated requests
  • Make the config so that the conclusion entails a mapping between individual query keys and batched keys
• Let the cache properly manage partial batch results, saving cached results individually, etc. The cache should only every store individual results, not batched results: these should be stored and retrieved from individual results.

The one major design problem seems to be that react-query is totally agnostic of the query function. A developer might choose to use radically different query functions for different calls. As a result, it almost doesn't make sense at all to just install some global option like maybeBatch which would take multiple query requests and decide whether or not the batch them.

Another design could maybe be to supercharge the query function argument. E.g.:

• If the developer passes the exact same queryFn to multiple calls, and that queryFn happens to have a .batch method.

An example:

function Page() {
  const { invoiceId } = useParams();
  const { data: me } = useApi("me", myApiFetcher);
  const { data: invoice } = useApi(["invoice", invoiceId], myApiFetcher);
}

function myApiFetcher(method, possibleParam) {
  // implemented in some way
}

myApiFetcher.batch = (queries) => {
  // e.g. queries[0] might be { key: ["me"] } and queries[1] = { key: ["invoice", 2] }
  // implemented in some way
};

Any thoughts? Is this something that you would be interested in seeing implemented in the short or long term? What would be a good API design?

[wolverineks]

what are the benefits of batching?

[KATT]

Fewer round-trips, coalesce loading state, multiple components can use the same remote data without doing several trips

You can read it some here from the GraphQL-world

• https://www.apollographql.com/blog/batching-client-graphql-queries-a685f5bcd41b/
• https://github.com/graphql/dataloader

[kamranayub]

Looking into this too. In our case, we can avoid the queryFn issue because we use named queries (we keep a registry). This means all our queryFns are the same for each named query and we hide some of the internals of react-query. It looks something like this:

const { useQuery } = makeQuery({
  name: 'my-cool-name',
  queryFn: (vars, opts) => { },
  defaultOptions: {
    staleTime: Infinity
  }
});

I think your idea of looking for a .batch property or other props on the queryFn reference makes sense; for us we could build it into our abstraction:

const { useQuery } = makeQuery({
  name: 'my-cool-name',
  queryFn: (vars, opts) => { },
  batchFn: (queries) => { },
  defaultOptions: {
    staleTime: Infinity,
    batchTime: 50
  }
});

If RQ had this built-in, we'd be passing batchFn to the .batch prop on the function.

Thinking about how this would be designed, I came to the same conclusion that implementing this within RQ would make sense. It may be doable outside RQ but it still needs to orchestrate the query cache to track queries/batches. The way I was going to explore it was by creating a custom BatchedQueryCache implementation and playing around with some ideas.

I made a visualization of how to think about this. In our case, it's a little more sophisticated because we essentially want to "combine" variables to send in one backend request and then distribute that response back to each query (probably using a selector or something equivalent).

[kamranayub]

If you have multiple query hooks on the page, in different parts of the tree, you do not know what fields to retrieve from the API. This is not retrieving all fields from an API, only the ones all the queries in the batch need which could be 0..N fields. On one page, you end up fetching 2 fields worth in one request. On the next, 5 fields worth.

If you were selecting everything, every time, then I don't think you'd need batching enabled on that query, no.

[TkDodo]

isn't that what graphQL is doing? allowing the client to select with fields they want?

[kamranayub]

If you're using GraphQL, only to a certain extent. If you have two like-queries on the page even (a superset/subset), it will not combine them into a single request (though it will update the local cache when either one completes). There are some mechanisms I've seen before with Apollo batching that does something similar to what we're discussing for React Query that accumulates the queries on a page and batches them. Different kind of mechanism (better, IMHO). But RQ is not GraphQL and we can't use the same exact semantics like that, it has to be a bit more manual on the query-end.

In any case, we aren't using GraphQL so it's more of a place to borrow ideas from but not exactly applicable to this implementation. The diagram I made is a simplistic example for illustration. In our actual environment, we are talking about dozens of queries and they are shared across contexts (some live in libraries, others in each individual site). We can't dictate whether we can fetch all fields from an API at the platform level as each page and combination of components dictate that and we want to avoid overloading the backend APIs asking for data we don't need. So this kind of situation is not something you'd probably need on an individual site but it is something that a large-scale set of apps with shared packages/queries may need.

The easiest way to explain it is to imagine you created a site but your header came from another shared package. The header has some queries in it. You import the same queries and use them on your page. Suddenly, you can diverge from the queries in the header and now you have 2 similar network requests to the same API when you load the page. The constraint is that you cannot update the header package to fetch all the applicable fields from the API because your site might need them, another site may not, and you don't want to just fetch everything all the time.

Hopefully that helps explain why this is useful. 👍

[kamranayub]

Just following up, we implemented this and it works like we want. If I can find a spare moment, I can try to either extract it into a new package or at least provide the boilerplate for someone to build from.

[KATT]

I do this in trpc as well, parallel useQuery calls are automatically combined in the same http request

[jbeuckm]

I keep coming to this use case that I would love to do with react-query. I have thought about writing useDatabase('key', fetchBatch(id:[])=>[]) that would take a function that fetches a list of ids and calls react-query cache functions under the hood to fill a database of ['key', id] queries. Has there been progress on this since last spring?

[kamranayub]

My team has an implementation that's working like we want. I think it might be more accurate to call it "query aggregation" or "query reduction" since it consists of a reducer function that takes in query arguments and spits out new one that combines certain arguments together.

If I can find the time, I can try to share the actual code that handles it. I achieved it by overriding startFetch and replacing with a function that handles the batching logic and handles the query result to modify/transform it before inserting into the cache.

I think this approach could even be simplified to just wrapping queryFn with the same logic but we haven't tried that yet.

[cromosom]

@kamranayub would love to see your implementation

[kamranayub]

I was just thinking, it's still V2 as we haven't upgraded. But if it's still helpful I can try to find time to make a little isolated example.

[sseppola]

I was able to write a batching mechanism by combining react-query with dataloader, and it works remarkably well. I built a little demonstration CodeSandbox: https://codesandbox.io/s/react-query-dataloader-lgosp

Key points:

• Both react-query and dataloader have caches so we need to disable dataloader's cache and let react-query manage it all. That way we get de-duplication and caching from react-query, and batching from dataloader.
• You want to keep a single loader rather than multiple in order to batch correctly
• dataloader requires the fetch function to return data in the same order as requested, so make sure your fetch function sorts the data before returning it.
• You can just as easy load an array of ids as a single id (eg. characterLoader.load(id) characterLoader.loadMany(ids)), and you never have to write different functions for fetching one or multiple (eg. fetchCharacter(id) and fetchCharacters(ids)) as you always use fetch multiple.

What I like about this is that it lets react-query do what its great at, and achieves batching with just a thin layer on top.

[TkDodo]

This is pretty amazing - thanks for that. Does that mean that graphql users could combine queries into one gql query that fetches multiple different entities in one request, but they are cached separately, potentially with different stale times?

[sseppola]

In theory yes, people would need a single fetch function that can fetch the different entities.

Relay has the "GraphQL Global Object Identification Specification" that would allow exactly this. In principle it creates globally unique ids like character:123 (often obfuscated with base64 and opaque to the client) so that the server can parse it and identify which domain object to fetch. This has the lovely effect that you don't need to write individual root resolves per domain object, so it can save a lot of work. More on that here: https://stackoverflow.com/questions/33399901/in-relay-what-role-do-the-node-interface-and-the-global-id-spec-play

The quickest way to make this work would be to standardize the fields fetched per entity (eg. over-fetch), and make your loader accept an object like {id, type}.

However, if you wanted to build the query dynamically and avoid over-fetching I think you would face a few obstacles. I peaked down this rabbit hole, I don't think there's a clear cut solution to how you achieve this. Regardless of implementation I've tried to outline the cases you need to consider:

Given two request for the same entity, each containing at least one unique field each

1. you'll need to somehow stitch the requests together so you query all the fields requested.
2. if one component triggers a refetch of the entity, you'll want to refetch all the fields and update the result of each subscriber. Otherwise you face having stale results in one component and not the other.
3. now mount a third query that requests the same entity but with a unique field. We'll have a partial cache hit at best, but being strict this query should be in a "loading for the first time" state, and the two others "stale while refetching".

react-query as de-duplicator and cache:
Continuing the architecture from the demo we try to use the react-query cache as much as possible, so your cache keys will look like: [{ entity: 'character', id: 1 }].

1. you'll need to somehow stitch together the queries behind the scenes. Intuitively I'm thinking some active query register that the query function looks up.
2. refetching in this solution is elegant, nothing needs to change because all queries share the same query key.
3. Based on the query key we get a cache hit, but in reality it's only a partial hit. I consider this a failure, and I don't think there's an elegant way to resolve this conceptually with react-query.

I don't know if it makes sense, but a way to resolve this would be to give the query an option to reject the cache hit and force a round trip. It feels more like mismanaged query keys, so I think you're going to face problems.

react-query as cache, dataloader as de-duplicator
I intuitively don't like this approach because it involves two caches, however dataloader on the server often just caches results on a per-request basis, so perhaps we can find a way to keep it in line.

Because we want to involve the graphql query in the query key we can introduce the query hash and borrow concepts from persisted queries in Relay. So our query keys can now be [{ entity: 'character', id: 1, queryHash: 'YXNkYXNk...' }] and represent our entire query! The mapping from queryHash to queryString should happen on the server, but we could also do it on the client if the backend isn't ready and for testing purposes.

1. our two initial queries now have unique query keys so they'll both be passed to dataloader. We want dataloader to de-duplicate on entity and id, and use the queryHash to "compile" the query.
2. I'm not sure what react-query does in the background when you call refetch(), but we'd want it to ignore the queryHash so it applies to both of our current subscribers when one of them refetches.
3. mounting the third subscribing query will naturally pass through the react-query cache, and by this time the dataloader cache should have been cleared so the request misses that cache as well. Whether it should query the union of the three active queries or just the last one I don't know, both could work.

This solution makes more sense conceptually, but requires more plumbing to work. I think the main problem here is whether we can get the extra cache to behave as we want or not.

Considering the refetch case you could argue that it's not react-query's responsibility to handle re-fetching across keys, however react-query could facilitate making it simpler if this becomes common.

Considering the cache I think makes most sense that each query gets its own cache entry, and that a stitched query is filtered down to match each individual query component.

[kamilkisiela]

Regarding GraphQL batching but it also applies to batching in general, it’s usually a bad idea. It sounds good at first but once you have slow requests it degrades the performance of the app. The latency of the request equals the latency of its slowest query. It’s fine to implement something like that for simple queries but once you go into merging graphql queries into a single document it becomes super hard to debug and detect performance regressions (as you don’t know what runs when - it always depends what is rendered and which queries are called).

[fzaninotto]

I've implemented batching as well in react-admin, where multiple components querying one record by id on mount lead to a single query for an array of ids (see implementation). This is equivalent to using a DataLoader.

It is working fine, but I have a performance problem. When the aggregated query (something like WHERE id IN (123, 456)) resolves, the aggregator resolves the promises of each individual query in a single tick. Yet, each of these individual resolves triggers a render. The result is that the list renders in several passes, despite the fact that all queries are resolved in the same tick.

So my question is more about batched updates than batch queries: if 2 react-queries resolve in the same tick, how can I batch the updates so that React makes only 1 render?

I tried using unsafe_batchedUpdates without success.

[TkDodo]

I haven't looked through all the code - it's quite a handful 😅 . But I get the problem - every resolving promise will trigger a re-render. Maybe you can try out our react-18 draft PR? Between automatic batching in react-18, useSyncExternalStorage and some improvements we've made internally to scheduleMicrotask, it might just work like you'd expect? There is a codesandbox preview build on the PR that you can try out:

• Feature/react 18 #3064
• codesandbox preview
• you can also get it to try out on your project with:
  • "react-query": "https://pkg.csb.dev/tannerlinsley/react-query/commit/48d642aa/react-query",

[Shenlok]

I tested this batching approach using a DataLoader and verified that each individual query triggers a render in React 17, but in React 18 the automatic batching appears to fix the problem (even without using react-query v4). Codesandbox for those interested, adjust index.js and dep versions to test React 17 vs 18

To try and minimize the impact of the multiple renders if you can't easily upgrade to React 18, you could consider lifting the useQueries into a parent component, and pass the data to child components that have been memoized with React.memo. The parent will still render once for each query, but at least most of the heavy lifting will avoid being repeated if it's done inside a memoized child.

[jbeuckm]

I am working on a hook that batches the request for react-query cache misses on a list of ids. When I set queryFn to my batch handler but also use useQueries with a different queryFn on the same set of queryKeys, I don't understand which queryFn will be used.

If you call useQuery('abc', f1) and also useQuery('abc', f2), which function do you expect to run?

import { useEffect } from 'react'
import { useQueries, useQueryClient } from 'react-query'

export function useQuerySet<T>(
  baseKey: string,
  ids: string[],
  fetchBatch: (ids: string[]) => Promise<Array<T>>
) {
  const queryClient = useQueryClient()

  const getFetchSingle = (id: string, note: string) => () => {
    console.log('getFetchSingle', note)
    return fetchBatch([id]).then(responses => responses[0])
  }

  useEffect(() => {
    if (!ids) {
      return
    }

    const batchIds: string[] = []
    let resolveBatchPromise: Function
    const batchPromise: Promise<Record<string, T>> = new Promise(
      resolve => (resolveBatchPromise = resolve)
    )

    let batchComplete = false

    ids.forEach(id => {
      const queryKey = [baseKey, id]

      const queryState = queryClient.getQueryState(queryKey)

      // if this query hasn't been loaded,
      if (!queryState || (!queryState.data && queryState.status === 'idle')) {
        batchIds.push(id)

        // set up the query to either...
        queryClient.fetchQuery(queryKey, async () => {
          // wait for the batch if batch hasn't run
          if (!batchComplete) {
            const responses = await batchPromise
            // access item in batch result by id
            return responses[id]
          }

          // or fetch one at a time
          return await getFetchSingle(id, 'new batched')()
        })
      }
    })

    if (batchIds.length === 0) {
      return
    }

    fetchBatch(batchIds).then(responses => {
      const responseIndex: Record<string, T> = {}

      responses.forEach((response, index) => {
        const id = batchIds[index]
        // index items by id
        responseIndex[id] = response
      })

      // deliver batch results to new queries
      resolveBatchPromise(responseIndex)

      // tell them to only wait for the batch once
      batchComplete = true
    })
  }, [ids.sort().join(',')])

  return useQueries(
    ids.map(id => ({
      queryKey: [baseKey, id],
      queryFn: getFetchSingle(id, 'useQueries'),
    }))
  )
}

[TkDodo]

If you call useQuery('abc', f1) and also useQuery('abc', f2), which function do you expect to run?

I don't think that's a valid use-case. Refetches triggered e.g. onWindowFocus just lookup the query by the first observer. So I think it's fair to say we expect a single queryFn per queryKey.

[fhur]

[kobybum]

@fhur Great implementation! Thanks for sharing, very clean and makes a ton of sense
Would love to have it in react query built in, or an external library

[IronTony]

@fhur found this problem

[yornaath]

Just wrote a library to help with this exact issue using react query. Check it out at https://github.com/yornaath/batshit
Feel free to test and report any issues or improvements wanted for it!

Here is an example:

import { useQuery } from "react-query"
import { Batcher, windowScheduler, keyResolver } from "@yornaath/batshit"

type User = { id: number, name: string }

const users = Batcher<User, number>({
  fetcher: async (ids) => {
    return api.users.where({
      userId_in: ids
    })
  },
  resolver: keyResolver("id"),
  scheduler: windowScheduler(10)
})

const useUser = (id: number) => {
  return useQuery([ "users", id ], async () => {
    return users.fetch(id)
  })
}

const UserDetails = (props: {userId: number}) => {
  const {isFetching, data} = useUser(props.userId)
  return <>
    {
      isFetching ? 
        <div>Loading user {props.userId}</div> 
      : 
        <div>
          User: {data.name}
        </div>
    }
  </>
}

/**
 * Since all user details items are rendered within the window there will only be one request made.
 */
const UserList = () => {
  const userIds = [1,2,3,4]
  return <>
    {
      userIds.map(id => <UserDetails userId={id} />)
    }
  </>
}

[yornaath]

@TkDodo Think I put it in the right docs place 😄 Here is the pr: #5385

[Play-AV]

Thanks for writing this library. The code sandboxes aren't working (they don't seem to fetch data for me) but this example, and the docs weren't terribly hard to work off of and figure out how to implement it with a rest api. I'm going to keep testing this properly but I quite like it so far. The use case is perfectly suited to this and should greatly cut down on the amount of data duplication / extra fetching by allowing individual items to exist as their own query keys. It'll be interesting to see how it holds up once there's a few thousand items in memory.

[yornaath]

@Play-AV Thanks! I will see if I can get the codesanbox examples working, but yeah the api shouldnt be to hard to figure out from the repos readme(https://github.com/yornaath/batshit).

Im interested to hear how it goes for you if you are running it with large datasaets. If you run into any issues please report them on the issue page of the repo 🙏

[Play-AV]

Yeah! Only one major gripe but probably not related to your library.
So from playing around with it, React Query Devtools are un-usable and essentially kill the page once opening the Query DevTools is attempted unfortunately. Even with a few hundred items it's a no go.

Having said that, I was able to get like 500 of these components loaded into the DOM on a page without issue (and honestly, they all have framer motion + floating UI stuff attached to each so I'm relatively impressed).
I'm in a possibly weird edge case here as I'm pulling the data from a dedicated local server, and in this case these items are stored in a redis database so my latency / request times are exceedingly low even when asking for a large # of items (7-15kb per individual item maybe but I guess the server gzips the response?) and the entire duration tends to be under 20ms when dealing with hundreds.

This definitely is looking like the right paradigm for my use case though (I'm glad I didn't have to write some terrible implementation myself). It'll be interesting to see if it's superior to the more traditional approach (request loads a bunch of items those items get stored in a single Query Key, and then just get passed into individual mapped out elements)

[tho-graf]

Hi @yornaath, I already used batshit successful in one project (not react and TanStack/query). Thanks for that!
At the moment I am planning to use a batching mechnism with react and TanStack/query. I stumbled upon this thread and also read about using dataloaders (we are already using them in our kotlin backend).

Now I ask myself: What's actually the difference between using dataloader and batshit? Aren't they solving pretty much the same problem?

[dima-getselectstar]

I am a bit curious why there is no built-in support for so common feature as batching.

[tsirlucas]

Hey guys, if you're having problems trying to implement this with Suspense and dataloader, it may be due to an issue with concurrent react. Ive faced it recently and decided to write a small blog post: https://dev.to/tsirlucas/integrating-dataloader-with-concurrent-react-53h1

If you're using bashit, probably their default 10ms scheduler already fixes the issue.