Story time. This has basically nothing to do with this post other than it involves a limit of 10,000 but hey, it's Christmas and I want to tell a story. I used to work for Facebook and many years ago people noticed you couldn't block certain people but the one that was most public was Mark Zuckerberg. It would just say it failed or something like that. And people would assign malice or just intent to it. But the truth was much funnier. Most data on Facebook is stored in a custom graph database that basically only has 2 tables that are sharded across thousands of MySQL instances but most almost always accessed via an in-memory write-through cache, also custom. It's not quite a cache because it has functionality built on top of the database that accessing directly wouldn't have. So a person is an object and following them is an edge. Importantly, many such edges were one-way so it was easy to query if person A followed B but much more difficult to query all the followers of B. This was by design to avoid hot shards. So I lied when I said there were 2 tables. There was a third that was an optimization that counted certain edges. So if you see "10.7M people follow X" or "136K people like this", it's reading a count, not doing a query. Now there was another optimization here: only the last 10,000 of (object ID,edge type) were in memory. You generally wanted to avoid dealing with anything older than that because you'd start hitting the database and that was generally a huge problem on a large, live query or update. As an example, it was easy to query the last 10,000 people or pages you've followed. You should be able to see where this is going. All that had happened was 10,000 people had blocked Mark Zuckerberg. Blocks were another kind of edge that was bidirectional (IIRC). The system just wasn't designed for a situation where more than 10,000 people wanted to block someone. This got fixed many years ago because somebody came along and build a separate system to handle blocking that didn't have the 10,000 limit. I don't know the implementation details but I can guess. There was a separate piece of reverse-indexing infrastructure for doing queries on one-way edges. I suspect that was used. Anyway, I love this story because it's funny how a series of technical decisions can lead to behavior and a perception nobody intended.
 Top