Search in Conversations with Typesense

[mdmohsin7]

Closes #1433

• Cleanup old local search stuff
• Setup and backfill memories on dev env with typesense
• Add endpoint to search using typesense
• Paginate search results
• Add search functionality with debouncing in App
• Setup and backfill memories on prod env
  1. Create an account on Typesense Cloud with company creds
  2. Define schema (use the one provided in this PR) and create collection in Typesense Cloud
  3. Setup Typesense extension in Firebase (use this version)
     • Firestore Collection Path: users/{userId}/memories
     • Firestore Collection Fields: structured,transcript_segments,created_at,deleted,discarded,started_at,id,finished_at,geolocation,userId
  4. Create typesense_sync collection and add doc backfill with data {'trigger' : true}
  5. Update the creds in backend

The last step can be done once the PR is merged. It should mostly be done during the inactive (or non-peak) hours to avoid missing memories while backfilling

ScreenRecording_12-28-2024.21-38-42_1.1.mp4

[beastoin]

1/ if we need a search feature on the conversation, it must be implemented from the server not local. we can not release a incomplete feature where users might immediately bounce back with the obvious question: why cant i find my conversation from 3 months ago? again, let's clarify the search feature first. my initial questions are: why the user need that feature and what's their next action(after finding their conversation) ?
2/ separated action items into their own page is a good idea. but this implementation is far from the good. let's put at least 01 wow moment into the feature if you want the user use it. the question is: would you use this separate action item page? everyday ?
3/ 'Home' does not make sense to me. Conversations is more direct.

overall, man, the value of that ticket is small but the maintenance efforts is significant.

@mdmohsin7

[mdmohsin7]

1/ Yes I only restored the existing local search, I was waiting for your confirmation on whether I should use Typesense or not. I am not sure if I can answer this question correctly why the user need that feature and what's their next action(after finding their conversation) ? because Nik opened the issue #1433 requesting for search feature and hence I just decided to restore the existing search

2/ Yes this is the very initial version to test the hypothesis, we have certain percentage of users making use of the action items by opening each memory detail. With this we only want to test whether more people will start using action items or not because now the friction is a lot less for them (not having to open multiple memories to view action items). If more people actually start using it, then we will move forward with improving it further and adding wow features, if not then we will simply restore the old layout (this is why I did not delete the old layout code and kept it in old_conversation_page.dart file and did not introduce too much new logic and because of it the implementation isn't good)
would you use this separate action item page? everyday ? Yes, and once we will have the ability to add custom action items to a memory and edit existing action items, it will completely replace the other todo app which I use

3/ Yes I too wanted to use Conversations but in the current bottom tab bar there isn't enough space and the text overflows. If I try reducing the padding, font size and stuff then it looks ugly

Lmk what we should do with this? Close it?
@beastoin

[smian1]

@mdmohsin7 maybe think about replacing the bottom nav with icons and if you really need to? Add a tiny text. Most apps have icons and can definitely add more.

[beastoin]

oops! let me drop a super clear message here so that we can move forward

even if this ticket is requested by Nik. your high-standard product crafting skill know that the only way to(have a chance to) create a great product is fully understand what and why - the very first step.

fighting @mdmohsin7 💪

[beastoin]

4/ dont store the no-index or non searchable info in typesense.firebase is the master storage. typesense is the search db.
-> minimize the data pushed to typesense pls.
5/ can we manual push the data(memory) to typesense ?

@mdmohsin7

[mdmohsin7]

4/ we can certainly do that, while setting up typesense extension we can define what all fields we want to be indexed

5/ Yes we can do that as well
https://typesense.org/docs/27.1/api/documents.html#index-documents

The typesense extension automatically creates a firebase function that gets executed everytime a document is created/updated in the memories collection. And I am using this at the moment in dev environment

@beastoin

[beastoin]

4/ yes. have you done for this step? i guess we will have the updated schema for the future usage ? you know the config file is the key for automating things.

5/ ✅

@mdmohsin7

[mdmohsin7]

4/ I haven't done typesense setup for prod. Was waiting for your confirmation. Should I do it now?

@beastoin

[mdmohsin7]

Use this version of typesense extension (for backfill support):
https://console.firebase.google.com/project/_/extensions/install?ref=typesense/[email protected]

During the Firestore extension setup, use the following path:

users/{userId}/memories

If you use a value other than {userId} to denote the doc id, then it should be updated in the below schema as well.
Note: we need the docId/userId to filter search results

Fields that will be indexed in typesense (means these are the fields that will be searchable and are stored in RAM):
(use the below schema while creating a collection in typesense)

{
  "name": "memories",
  "fields": [
    {
      "name": "structured",
      "type": "object"
    },
    {
      "name": "structured.category",
      "type": "string",
      "facet": true
    },
    {
      "name": "transcript_segments",
      "type": "object[]"
    },
    {
      "name": "created_at",
      "type": "int64"
    },
    {
      "name": "deleted",
      "type": "bool"
    },
    {
      "name": "userId",
      "type": "string"
    }
  ],
  "default_sorting_field": "created_at",
  "enable_nested_fields": true
}

Fields that will be stored in typesense (means the fields which are not there in the above schema will be stored in disk and not RAM):

structured,transcript_segments,created_at,deleted,discarded,started_at,id,finished_at,geolocation,userId

When we search, it will only return the data that is stored in RAM and Disk, so we won't get the fields like plugin_results in the response of typesense. So I've added a call to memory detail endpoint to fetch that specific memory details.

[beastoin]

lgtm @mdmohsin7 🥳

pls process next steps to make it happen on prod