Lemmy.world Support

3253 readers

24 users here now

Lemmy.world Support

Welcome to the official Lemmy.world Support community! Post your issues or questions about Lemmy.world here.

This community is for issues related to the Lemmy World instance only. For Lemmy software requests or bug reports, please go to the Lemmy github page.

This community is subject to the rules defined here for lemmy.world.

To open a support ticket

You can also DM https://lemmy.world/u/lwreport or email [email protected] (PGP Supported) if you need to reach our directly to the admin team.

Follow us for server news 🐘

Outages 🔥

https://status.lemmy.world

founded 2 years ago

MODERATORS

What is the rate limit for requests on the webrequests for search and API for lemmy.world? (self.support)

submitted 2 years ago by Beliriel to c/support

5 comments fedilink hide all child comments

I would like to try building a search index for this instance (maybe others) and as such would like to crawl the site with automated spiders. Now with the shutdown of the reddit API I expect the site to come under quite substantially load and also I would ofc try to not spam the site with too many requests as to not get banned or blocked, due to looking like a DOS attack. Could anyone provide some information on this?

top 5 comments

sorted by: hot top controversial new old

[–] marsara9 1 points 2 years ago (1 children)

I'm already working on it. If you able to help out I need developers who can contribute.

https://github.com/marsara9/lemmy-search

But otherwise you can access /api/v3/site to see the individual rate limits.

[–] Beliriel 1 points 2 years ago

Oh nice I'll check it out. I'll guess I'll have to learn it but shouldn't pose much of a problem. Btw I sent you a request on Discord (I assume it's the same username)

[–] ericjmorey 1 points 2 years ago (1 children)

Many existing Fediverse services are being operated by people who are opposed indexing the content on their instance(s). You may run into resistance from that angle.

[–] Beliriel 1 points 2 years ago (1 children)

I mean unless they make their instance private I don't see why you wouldn't index them? That's literally why google provided such a value in their early days.

[–] ericjmorey 1 points 2 years ago

Even Google doesn't index webpages that include "noindex" in a header. You are going to run into a lot of people who don't agree with what you are trying to do. If you start reaching out to the people running Fediverse services to let them know that you're trying to index the data on their services, you can learn what they think of the idea.