Handling Activities with Bulk Actions

A storage-efficient approach to handle bulk actions in the RSS3 UMS.

Background

On the RSS3 Network, each activity is equivalent to a blockchain transaction and each action is equivalent to an event log within the transaction. It is expected that we will end up with activities with multiple actions.

While this mapping design is not perfect, it is the optimal structure after numerous internal debates and tests, taking into consideration both performance and cost factors to ensure efficacy. Therefore, certain challenges inevitably need to be addressed.

Here we discuss the challenges and the solutions for handling activities with bulk actions, and explain why we believe the chosen solution is the best for the RSS3 Network, developers and users.


Challenges

The One on Cost

One challenge with this mapping is that when a transaction contains a large number of event logs, it creates an activity with the same number of actions. The size becomes huge, and storage and transmission of such activities quickly become expensive when you have billions of them in storage, and millions of requests per month.

A Polygon Transaction with 2501 Event Logs.

A transaction on Polygon with 2,501 event logs. Rendering such a large number of event logs often makes your browser unresponsive for a few seconds.

The One on Value

A transaction with over 100 event logs, likely but not always, represents spam.

No one likes spam it unless it's in a can.

No one likes spam unless canned (still not healthy though).

Devs are going to scratch their heads (we did 😩) to figure out a way to render a spam transaction: Should I grey it out? Should I truncate it? Should I use an accordion? Should I just ditch it? Should I just ignore it? What if it freezes my app?

No sane user is going to drill into a spam transaction, wanting to find out who the other victims are.

So let's be honest, having an activity with a huge number of actions, in 99.99% of situations, does not create any value for your apps and your users.


Simple Math

Internally, we had multiple rounds of discussions on the most appropriate way to address the above challenges. Since we index every single transaction on the blockchains we support, we need to take the cost into consideration. We ran the following calculations to determine the impact of these activities on our storage systems.

We use Polygon as an example, other blockchains are not too far off.

Criteria, Logs in 1 Tx
Number of TxsPercentage (Polygon had 212,663,866 Txs in 2023 Q3)
> 501,783,3290.84%
> 1001,265,2470.59%
> 200921,7500.43%
> 500309,2630.15%

Solutions

Similarly, we also engaged in multiple rounds of discussions to agree on the most appropriate solution.

Let's take this activity of 0x827431510a5D249cE4fdB7F00C83a3353F471848, mapped from this transaction on Polygon as an example.

{
  "owner": "0x827431510a5D249cE4fdB7F00C83a3353F471848",
  "id": "0xbe6f8469757c786b923b8fd9a35285fd3d02d3fc3799672ad3a34957d930d606",
  "network": "polygon",
  "from": "0x323693381602960c98cB4e0Cf13F098E3f6E2894",
  "to": "0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E",
  "tag": "collectible",
  "type": "mint",
  "status": "successful",
  "direction": "in",
  "feeValue": "1.0466209226",
  "actions": [
    {
      "tag": "collectible",
      "type": "mint",
      "from": "0x0000000000000000000000000000000000000000",
      // this is not even the address that's being requested
      "to": "0xc3B9E7c111028CeC42f8b2f5BECE0046F6228176",
      "metadata": {
        "contract_address": "0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E",
        "id": "0",
        "value": "1",
        "name": "! !$100,000 BONE #01",
        "symbol": "Shiba Inu",
        "standard": 1155,
        "uri": "https://shibnft.cc/80.json\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
      },
      "related_urls": [
        "https://opensea.io/assets/matic/0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E/0",
        "https://polygonscan.com/tx/0xbe6f8469757c786b923b8fd9a35285fd3d02d3fc3799672ad3a34957d930d606"
      ]
    }
    ,
    {
        // there are another 2498 Actions here...
    }
  ]
}

A Hot and Cold Storage Split

In this solution, we relocate actions of qualified activities from our hot storage (SSD) to a much more affordable cold storage (HDD). We then alter our logic to enable pagination of actions, when such activities are being served to developers.

The result will look like this:

{
  "owner": "0x827431510a5D249cE4fdB7F00C83a3353F471848",
  "id": "0xbe6f8469757c786b923b8fd9a35285fd3d02d3fc3799672ad3a34957d930d606",
  "network": "polygon",
  "from": "0x323693381602960c98cB4e0Cf13F098E3f6E2894",
  "to": "0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E",
  "tag": "collectible",
  "type": "mint",
  "status": "successful",
  "direction": "in",
  "feeValue": "1.0466209226",
  "actions": [
    {
      "tag": "collectible",
      "type": "mint",
      "from": "0x0000000000000000000000000000000000000000",
      "to": "0xc3B9E7c111028CeC42f8b2f5BECE0046F6228176",
      "metadata": {
        "contract_address": "0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E",
        "id": "0",
        "value": "1",
        "name": "! !$100,000 BONE #01",
        "symbol": "Shiba Inu",
        "standard": 1155,
        "uri": "https://shibnft.cc/80.json\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
      },
      "related_urls": [
        "https://opensea.io/assets/matic/0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E/0",
        "https://polygonscan.com/tx/0xbe6f8469757c786b923b8fd9a35285fd3d02d3fc3799672ad3a34957d930d606"
      ]
    }
    ,
    {
        // there are another 19 actions, as we set the page size to 20
    }
  ],
  "actions_count": 2500,  // a new field to represent the total
  "actions_page": 1  // a new field to represent the page
}

Pros

  • We estimate a saving of up to 10% in storage and data transfer.
  • We see no performance impact (actually a slight improvement since the response size is smaller).
  • The activity history and UMS are intact, nothing is missing.

Cons

  • We need to invest some engineering hours to implement it.
  • Additional requests are needed to fetch these actions.

⭐️ Drop All but Keep One (The One We Eventually Chose)

In this solution, we drop all actions when the number hits 100, we then keep a pointer to a single action (after removing ownership-related metadata) to maintain our UMS structure.

{
  "owner": "0x827431510a5D249cE4fdB7F00C83a3353F471848",
  "id": "0xbe6f8469757c786b923b8fd9a35285fd3d02d3fc3799672ad3a34957d930d606",
  "network": "polygon",
  "from": "0x323693381602960c98cB4e0Cf13F098E3f6E2894",
  "to": "0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E",
  "tag": "collectible",
  "type": "mint",
  "status": "successful",
  "direction": "in",
  "feeValue": "1.0466209226",
  "actions": [
    {
      "tag": "collectible",
      "type": "mint",
      "metadata": {
        "contract_address": "0x0757921E361B8B20162FBbC5Fe73Dc8E32fD418E",
        "id": "0",
        "value": "1",
        "name": "! !$100,000 BONE #01",
        "symbol": "Shiba Inu",
        "standard": 1155,
        "uri": "https://shibnft.cc/80.json\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000\u0000"
      },
      "related_urls": [
        "https://polygonscan.com/tx/0xbe6f8469757c786b923b8fd9a35285fd3d02d3fc3799672ad3a34957d930d606"
      ]
    } // no more actions
  ],
  "total_actions": 2501 // so that you know how many actions were dropped
}

Pros

  • We estimate a saving of up to 14% in storage and data transfer.
  • We see no performance impact (actually a slight improvement since the response size is smaller).
  • The activity history is intact, nothing is missing.

Cons

  • We need to invest some engineering hours to implement it.
  • Some valuable activities with more than 100 actions, albeit only a small fraction (<0.01%), are not served.
  • Some slight changes to the UMS, due to the removal of ownership-related fields (from and to).

Conclusion

By dropping the actions mapped from transactions with more than 100 logs, we expect a saving of up to 15% in storage and data transfer for every node operator, with minimal effort required from the developers who build on the RSS3 Network. The chosen solution ensures the integrity and accuracy of activity history.

In our opinion, these activities do not provide any value other than distracting developers and app users from what is truly worth paying attention to.