A big problem with large language machine learning models right now is the fact that they rely on massive troves of data freely available on the web to become as powerful as they are. In the early days, a lot of this content was and partially still is scraped from websites without their explicit consent, though things are better with new opt-out mechanisms available from Open AI and otherAIcompanies. Now, Google is joining the pack with a way to let website owners opt out ofGoogle Bard— without opting out of Google Search as a whole.
The news comes as part ofan update for web publisher controls, with Google launching a standalone generative AI web crawler called “Google-Extended.” With the standalone product available, it’s possible for web publishers to specifically opt out of Google running the crawler on their websites. Like other crawlers, it uses the robots.txt on a website’s to determine which pages it’s allowed to visit, so web publishers can use the tool to stop it from visiting and using their sites. Previously, publishers could only opt out of Google’s crawler as a whole, which isn’t ideal if you think about Google Search’s market share.
In addition to excluding Bard from crawling websites, the tool also makes it to stop Google’s Vertex AI, with the company saying that this will also stop future generations of models.
Other AI companies are also using web crawlers to scout the web for content to use for their generative AI products, and many publishers have already opted out of those. However, given how intertwined Google Search and Bard were for a long time, questions were raised about how to stop Google from using content for its AI without also blocking out Google Search’s content crawlers, which could have adverse effects on a website’s performance on Google Search.
Google Bard has come a long way since launch. It now connects to other Google services via extensions, allowing you to tap into data from Google Flights, Hotels, Maps, and YouTube. You can even connect Google Bard to your Google Workspace, allowing it to access documents from your Google Drive. This can allow the tool to give you quick summaries or create content based on your own content without any complicated workarounds for importing data.
Google has also added a way to double-check information from Bard, allowing you to easily validate via an automated Google search if what the bot tells you is actually true — after all, generative AI products have big issues with confidently giving out false answers. In addition to all this, it’s also possible to upload images to the bot via Google Lens to have the bot work with them. It can also show you images itself as part of Google Images.
Meanwhile, other AI models are making big strides, too. OpenAI’s ChatGPT is now capable of searching the web, a feature that it lacked when it was first released as it only had access to data up to some point in 2021.
