this post was submitted on 15 Dec 2023
-5 points (41.4% liked)

Programming

17670 readers
234 users here now

Welcome to the main community in programming.dev! Feel free to post anything relating to programming here!

Cross posting is strongly encouraged in the instance. If you feel your post or another person's post makes sense in another community cross post into it.

Hope you enjoy the instance!

Rules

Rules

  • Follow the programming.dev instance rules
  • Keep content related to programming in some way
  • If you're posting long videos try to add in some form of tldr for those who don't want to watch videos

Wormhole

Follow the wormhole through a path of communities [email protected]



founded 2 years ago
MODERATORS
 

If you are wondering why lemmy is moving away from offset pagination since 0.19, here is a nice article about it

all 17 comments
sorted by: hot top controversial new old
[–] SpaceNoodle 16 points 1 year ago (2 children)
[–] wccrawford 10 points 1 year ago (4 children)

https://ivopereira.net/efficient-pagination-dont-use-offset-limit

This seems to be the same article.

I have my doubts about the technique, but it could be useful in certain controlled situations.

[–] [email protected] 6 points 1 year ago

Lemmy just implemented it for 0.19 and it makes a big difference on heavier queries like Scaled homepage.

It also has the advantage of your pagination not getting screwy if new content has been added between page 2 and 3 queries.

[–] [email protected] 3 points 1 year ago (1 children)

I was going to recommend looking at https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results - but it looks like that method is now not advised- but if you read up above it it looks like there’s a search_after/PIT method described which sounds similar to the article.

This is all to say - I don’t think this is a one-off concept - it’s been around for a bit.

[–] SpaceNoodle 1 points 1 year ago

Thanks! Agreed, it's a very limited usecase.

[–] [email protected] 1 points 1 year ago

I have my doubts about the technique, but it could be useful in certain controlled situations

This is completely uncontroversial advice and has been for 30 years. What are your doubts exactly?

I’d go further: if you see a query that uses “offset” on a non-trivial production DB something is very, very wrong.

Of course, the trick is that you need to make sure you have indexes for all sort orders you need to display, but that’s obvious.

[–] [email protected] 1 points 1 year ago

Sorry, I updated the link

[–] [email protected] 4 points 1 year ago (2 children)

For the article-impaired,

Using OFFSET+LIMIT for pagination forces a full table scan, which in large databases is expensive.

The alternative proposed is a cursor+based navigation, which is ID+LIMIT and requires ID to be an orderable type with monotonically increasing value.

[–] Blackmist 2 points 1 year ago

Which it almost never is.

Any data as simple as that is unlikely to reach a number of rows likely to cause an issue with performance.

[–] [email protected] 1 points 1 year ago* (last edited 1 year ago)

Having said this, I'd say that OFFSET+LIMIT should never be used, not because of performance concerns, but because it is fundamentally broken.

If you have rows being posted frequently into a table and you try to go through them with OFFSET+LIMIT pagination, the output from a pagination will not correspond to the table's contents. Fo each row that is appended to the table, your next pagination will include a repeated element from the tail of the previous pagination request.

Things get even messier once you try to page back your history, as now both the tip and the tail of each page will be messed up.

Cursor+based navigation ensures these conflicts do not happen, and also has the nice trait of being easily cacheable.

[–] [email protected] 2 points 1 year ago (1 children)

Is there a way I can access this article without making an account?

I'm not going to make an account.

[–] [email protected] 4 points 1 year ago (1 children)

Sorry for inconvenience, I updated the link

[–] [email protected] 2 points 1 year ago

Oh thanks mate 👍

Interesting article but I kinda fail to see how you'd go if your paginating through sorted rows - you'd have to have an id in the sequence of your sort order?

[–] [email protected] 1 points 1 year ago (1 children)

Still wants me to make an account.

[–] [email protected] 2 points 1 year ago* (last edited 1 year ago)

That's really weird, because I have no account on medium and it works for me. Maybe try this link instead:

https://ivopereira.net/efficient-pagination-dont-use-offset-limit