searchengines

cross-posted from: https://lemmy.world/post/17906460

DuckDuckGo, Bing, Mojeek, and other search engines are not returning full Reddit results any more.

This brings us to the current state of affairs: Google is no longer trying to index the entire web. In fact, it's become extremely selective, refusing to index most content. This isn't about content creators failing to meet some arbitrary standard of quality. Rather, it's a fundamental change in how Google approaches its role as a search engine.

From my experience, Google now seems to operate on a "default to not index" basis. It only includes content in its index when it perceives a genuine need. This decision appears to be based on various factors:

• Extreme content uniqueness: It's not enough to write about something that isn't extensively covered. Google seems to require content to be genuinely novel or fill a significant gap in its index.
• Perceived authority: Sites that Google considers highly authoritative in their niche may have more content indexed, but even then, it's not guaranteed.
• Brand recognition: Well-known brands often see most of their content indexed, while small or unknown bloggers face much stricter selectivity. Temporary indexing and de-indexing: In practice, Google often indexes new content quite quickly, likely to avoid missing out on breaking news or important updates. Soon after, Google may de-index the content, and it remains de-indexed thereafter. So getting initially indexed isn't necessarily a sign that Google considers your content valuable.

Google has finally addressed those unhinged web search results seemingly generated by its AI, screenshots of which circulated far and wide on social media this past week.

In short, the internet goliath argued it's not as bad as it looks, though vowed to eliminate the system's baffling responses.

For those who missed it, Google introduced these so-called AI Overviews this month, graduating the system from an optional experimental feature to putting it into worldwide production starting with US users.

...

That summary is supposed to be accurate and relevant. However, as some folks discovered, Google sometimes came back with absurd and nonsensical answers.

No doubt at least some of the replies screenshotted and shared on social networks were edited by humans to make it look as though the Big G had completely lost the plot. That said, in two especially high profile examples, if genuine, AI Overviews said people "should eat one small rock per day" and that cheese not sticking to pizza could be fixed by adding "non-toxic glue" to the sauce.

These idiotic replies appear to have stemmed from, we assume among other things, jokes and snark made on Reddit, which is a source of training data for various LLMs including Google's – the Chrome behemoth is paying Reddit about $60 million a year to ingest its users' posts and comments.

...

Google is taking the position that the screenshotted examples of dodgy advice are a fraction of the AI system's overall output. It defended its Gemini-based results, and said the system only needed a few tweaks, rather than a full reworking, to get it on track. Reid claimed "a very large number" of the bizarre results we've seen were faked, and denied AI Overviews actually recommended smoking while pregnant, leaving dogs in cars, or jumping off bridges to cure depression, as some on social media alleged.

Google says one key issue with AI Overviews is that it took "nonsensical queries" far too seriously, specifically pointing out that the recommendation to literally eat rocks was only spat out by the search engine because the question was: "How many rocks should I eat?

Google is also limiting its reliance on info written by everyday netizens, which is how the glue on pizza suggestion came up: From someone called Fucksmith joking around on Reddit.

AI Overviews are the culmination of a long line of products, going back almost two decades to the launch of its customized homepage, that have turned Google.com into its own self-contained online ecosystem. One of its first major advances in the amount of information Google would display on its search page came in 2012, with the debut of Knowledge Panels – boxes of information, usually taken from Wikipedia, that display basic information, photos and biographical details about a person or subject.

Knowledge Panels expanded to the point that Google chief executive Sundar Pichai boasted in 2016 that they contained 70bn facts. Next came other services like stock prices and weather reports that would have previously required users to direct their attention to websites, causing alarm among outlets built around providing such information. When Google began featuring sports schedules on its page in 2013, TechCrunch ran an article titled “Google Embeds March Madness Bracket In Search, Because Screw Sports Sites”.

...

Google has also expanded so much over the years that at times it’s hard to see the barriers of when its platform ends and another site begins. In 2015 Google launched accelerated mobile pages, or AMP, that loaded articles faster on Google’s platform. Major news outlets quickly began publishing AMP articles, only to find that AMP pages generated far less advertising revenue than their own mobile sites.

Publishers have long been wary of what Google’s strong gravitational pull has done to their reliance on the platform. The growing dependency on its traffic has resulted in over a decade of media companies seeking revenue through search-engine-optimized articles that range from HuffPost’s 2011 classic of the form “What Time is the Super Bowl?” to Bon Appetit’s recent “What is That White Stuff on Your Food?”. Those types of article, created in response to the chase for Google referrals, now seem the most likely to become fodder for AI Overviews.

...

The potential threat from AI Overviews is especially acute because other major platforms have become dwindling and unreliable sources of traffic.

Facebook’s changes to its news feed have sent reverberations throughout the media industry for years, drastically decreasing traffic to digital outlets and leading them to make major structural changes like the mid-2010s infamous “pivot to video”. Facebook has decreased its algorithmic emphasis on news content to the point that politics magazine Mother Jones experienced a 99% decline in referrals since its peak year, while Meta announced in February it would kill its Facebook news tab for US users. Entire countries such as Canada see no links to news on Facebook.

Other platforms haven’t offered any respite from Facebook’s turn away from news. Twitter, never a large source of traffic or ad revenue compared with Facebook or Google, has become even more irrelevant for publishers since billionaire Elon Musk took over the platform, spurned news content and embraced on-platform viral videos. Apple News has alternatively driven an immense amount of traffic to news sites that work with its app, but publishers have struggled to gain revenue from these partnerships, as most users stay within Apple’s platform.

What’s left for publishers is largely direct visits to their own home pages and Google referrals. If AI Overviews take away a significant portion of the latter, it could mean less original reporting, fewer creators publishing cooking blogs or how-to guides, and a less diverse range of information sources. It would also increase Google’s dominance over what we see when we look at the internet, an issue that is already the subject of antitrust lawsuits from the US Department of Justice that allege the company has illegally monopolized the search and advertising industries.

“Two decades ago, Google became the darling of Silicon Valley as a scrappy startup with an innovative way to search the emerging internet,” the justice department stated in its 2020 complaint. “That Google is long gone. The Google of today is a monopoly gatekeeper for the internet, and one of the wealthiest companies on the planet.”

The article that inspired the previous post:

When it comes to the company’s core search engine, however, the image of progress looks far muddier. Like its much-smaller rivals, Google’s idea for the future of search is to deliver ever more answers within its walled garden, collapsing projects that would once have required a host of visits to individual web pages into a single answer delivered within Google itself.

The company’s AI-powered search results, which it calls Search Generative Experience, are coming to all users in the United States soon, Google announced today. By the end of 2024, they will appear at the top of results for 1 billion users.

As I noted when I wrote about the new, more extractive search engines from companies like Arc and Brave, Google’s move to answer more questions on the search engine results page is simply a continuation of a long-standing practice. But where the company once limited itself to gathering low-hanging fruit along the lines of “what time is the super bowl,” on Tuesday executives showcased generative AI tools that will someday plan an entire anniversary dinner, or cross-country-move, or trip abroad. A quarter-century into its existence, a company that once proudly served as an entry point to a web that it nourished with traffic and advertising revenue has begun to abstract that all away into an input for its large language models.

This new approach is captured elegantly in a slogan that appeared several times during Tuesday’s keynote: let Google do the Googling for you. It’s a phrase that identifies browsing the web — a task once considered entertaining enough that it was given the nickname “surfing” — as a chore, something better left to a bot.

...

When I started covering Google a decade ago, the company talked often about evolving search until it resembled the computer from Star Trek: an omnipresent, omniscient digital assistant. The Star Trek computer is notable for its ability to answer questions more or less instantly, and if the information it provides is supplied by something akin to the world wide web, it is not something that its users ever see. Whatever labor funded the production of knowledge that it refers to goes unmentioned, and whatever sources it relies on go uncredited.

It is this vision of the future that Tuesday’s announcements moves us ever closer to. And it is one that is understandably of concern to the many people who have come to rely on Google answering questions imperfectly, or partially, and funneling traffic to them since 1998.

“Web publishers brace for carnage as Google adds AI answers,” read an accurate headline in the Washington Post on Monday. Until now, publishers have been able to rely on significant volumes of traffic coming from the blue links that appear under many queries. But what the company is now calling AI overviews often obscure these links, requiring users to click to see them, or simply abstracting them away in an automatically generated summary. Analysts who have studied the company’s early experiments with SGE say a bloodbath is coming.

...

Still, as the first day of I/O wound down, it was hard to escape the feeling that the web as we know it is entering a kind of managed decline. Over the past two and a half decades, Google extended itself into so many different parts of the web that it became synonymous with it. And now that LLMs promise to let users understand all that the web contains in real time, Google at last has what it needs to finish the job: replacing the web, in so many of the ways that matter, with itself.

cross-posted from: https://lemmy.ml/post/15652001

cross-posted from: https://hachyderm.io/users/maegul/statuses/112442514504667645

Google's play on Search, Ads and AI feels obvious to me.

* They know search is broken.
* And that people use AI in part because it takes the ads and SEO crap out.
* IE, AI is now what Google was in 2000. A simple window onto the internet.
* Ads/SEO profits will fall with AI.
* But Google will then just insert shit into AI "answers" for money.
* Ads managed + up-to-date AI will be their new mote and golden goose.

@technology

See @caseynewton 's blog post: https://mastodon.social/@caseynewton/112442253435702607

Cntd (Edit):

That search/SEO is broken seems to be part of the game plan here.

It’s probably like Russia burning Moscow against Napoleon and a hell of a privilege Google enjoy with their monopoly.

I’ve seen people opt for chatGPT/AI precisely because it’s clean, simple and spam free, because it isn’t Google Search.

And as @caseynewton said … the web is now in managed decline.

For those of us who like it, it’s up to us to build what we need for ourselves. Big tech has moved on

A good search engine is difficult to find these days, with people constantly trying to hide things from you. I created this website to give people a bit of information which makes the decision of where to search a bit easier. It is mainly based on information from The Search Engine Map but includes functionality which allows you to dive into options easily.

So I use the Brave Search Engine instead of Google because of, well not because of Privacy, but because of a new change, and of which I find the Brave Search to have a new look and feel. Google just feels outdated IMO.

Anyway, I noticed a feature with this search engine, as I use Firefox with Brave. Whenever I type any query, and in the end, I put me! (the "!" is important) this redirects to a website, which leads to, you guessed it, Fandom, Mass Effect. I find it to be a fun and quirky feature by the Brave Devs.

I'm not sure if brave is the only one, as I only use Brave. Likewise, I tried; Duck, Bing, Yahoo and Mojeek, of which none of them redirected. I was wondering if it's easy to implement this redirect and how it actually works.

Try:

I Paid 5 Developers to create an ENTIRE App for me!

Edit 2: I made a list of the URLs that could possibly make Brave Redirect.

If you think you've been seeing an awful lot more Reddit results lately when you search on Google, you're not imagining things.

The internet is in upheaval, and for website owners the rules of "winning" Google Search have never been murkier.

Google's generative AI search engine is coming from one direction. It's creeping closer to mainstream deployment and bringing an existential crisis for SEOs and website makers everywhere.

Coming from the other direction is an influx of posts from Reddit, Quora, and other internet forums that have climbed up through the traditional set of Google links.

Data analysis from Semrush, which predicts traffic based on search ranking, shows that traffic to Reddit has climbed at an impressive clip since August. Semrush estimated that Reddit had over 132 million visitors in August 2023. At the time of publishing, it was projected to have over 346 million visitors in April 2024.

None of this is accidental. For years, Google has been watching users tack on "Reddit" to the end of search queries and finally decided to do something about it.

Google started dropping hints in 2022 when it promised to do a better job of promoting sites that weren't just chasing the top of search but were more helpful and human.

Last August, Google rolled out a big update to Search that seemed to kick this into action. Reddit, Quora, and other forum sites started getting more visibility in Google, both within the traditional links and within a new "discussions and forums" section, which you may have spotted if you're US-based.

The timing of this Reddit bump has led to some conspiracy theories. In February, Google and Reddit announced a blockbuster deal that would let Google train its AI models on Reddit content. Google said the deal, reportedly worth $60 million, would "facilitate more content-forward displays of Reddit information," leading to some speculation that Google promised Reddit better visibility in exchange for the valuable training data. A few weeks later, Reddit also went public.

A Google spokesperson told BI: "Our agreement with Reddit absolutely did not include ranking its content higher on Search."

...

YouTube already has heaps of videos offering advice on hijacking Reddit to rise to the top of Google Search. One of the most popular videos right now teaches viewers to purchase abandoned Reddit accounts, seek out popular Reddit posts to post comments filled with affiliate links, and then artificially inflate their popularity using "upvotes" purchased from third-party websites. This way, they can feasibly rocket to the top of Google.

Lily Ray, vice president of SEO strategy and research for marketing agency Amsive, told BI she is concerned that Google is "shifting the burden" of monitoring abuse onto Reddit moderators. Neglected subreddits risk getting spammed with unhelpful or even false content.

Ray wrote on LinkedIn last week that she found an example of Reddit ranking prominently at the top of Google when she searched "How to lose 10 pounds in a week".

"A redditor suggested 'cutting your arm off,'" she wrote.

cross-posted from: https://lemmy.smeargle.fans/post/152387

HN Discussion

I should note that I’ve previously — and erroneously — referred to the “code yellow” as something that Gomes raised as a means of calling attention to the proximity of Google’s ads side getting too close to search. The truth is much grimmer — the Code Yellow was the rumble of the Rot Economy, with Google’s revenue arm sounding the alarm that its golden goose wasn’t laying enough eggs. Gomes, a Googler of 19 years that built the foundation of modern search engines, should go down as one of the few people in tech that actually fought for a real principle, destroyed by and replaced with Prabhakar Raghavan, a computer scientist class traitor that sided with the management consultancy sect. More confusingly, one of the problems was that there was insufficient growth in “queries,” as in the amount of things people were asking Google. It’s a bit like if Ford decided that things were going poorly because drivers weren’t putting enough miles on their trucks.

...

Five months later, a little over a year after the Code Yellow debacle, Google would make Prabhakar Raghavan the head of Google Search, with Jerry Dischler taking his place as head of ads. After nearly 20 years of building Google Search, Gomes would be relegated to SVP of Education at Google. Gomes, who was a critical part of the original team that made Google Search work, who has been credited with establishing the culture of the world’s largest and most important search engine, was chased out by a growth-hungry managerial types led by Prabhakar Raghavan, a management consultant wearing an engineer costume.

cross-posted from: https://lemmy.ca/post/19828008

Raphael Kabo's personal website and writing on programming, poetry, and academia.

"What's clear right now is that there's no one spamming Google [that's] not doing it with AI," Gillham told The Register. "Not all AI content is spam, but I think right now all spam is AI content."

Gillham's team has been producing monthly reports to track the degree to which AI-generated content is showing up in Google web search results. As of last month, about 10 percent of Google results point to AI content, it's claimed, and that's after Google vowed to take down a whole host of websites that were pushing such trash.

"Google did these manual actions to try and win a battle, but then seem to still be sort of struggling with their algorithm being overrun by AI content," Gillham told us.

One of the great joys of working on a search engine is that you get to reverse engineer SEO spam, and overall study how it evolves over time.

I’ve been noticing the search engine spam strategy of adding ‘reddit’ to page titles for a few years now, but it feels like it’s been growing a lot recently. I don’t think it’s actually working, but it’s so cute that they are trying.

The “white noise baby reddit” item has an interesting word salad quality to it. Reminds me of the A Japanese Toaster You Can Fuck listicle I discovered a while back, which to this date is my all time favorite SEO trainwreck.

So, let's here what people think of the state of search engines and see what recommendations people have, preferably for non-corporate ones. Popular on here are:

• SearXNG
• kagi

There's also:

• Startpage

What else?

I'll update here with suggestions.

Do you know about a private customizable search engine? I would like to hint the search engine about what search results I like without specifying them on every single query.

For instance when need to search up some programming documentation almost always there are some blogs in the search results meanwhile I'm always looking for the official documentation. I want to go to settings and specify something like

+ official documentation
- blog

and every single time I search something up I get less noise. Also being able to block some domains like

geekforgeeks.org
learn.microsoft.com

would also be nice. In case I get annoyed by some domain that have a lot of ads and information at the very bottom.

see how it goes ..