Basic Search Refinements

Google Tag Cloud Maker is a Greasemonkey script that aggregates all the words from the titles and snippets of Google search results and creates a box that lists the most frequent words. If you hover over one of the words, you'll see the search results that contain that word in the title or snippet and if you click on the word, it will be added to the current query.

This is a primitive way to refine a query using some information from the search results, but it's also a way to see the big picture, to understand more about your query without visiting each search result.

The idea doesn't work well for all the queries, the popular words are not always placed at the top, and the box also clutters the interface. But it may be useful at times.

To install the script you need Greasemonkey for Firefox. Find more Greasemonkey scripts for Google services at this site.

{ via Lifehacker }

A Campaign for Alternative Search Engines

Ask.com has a campaign in the UK called "Information Revolution". Their purpose is to convince the people who use Google to search online that they should use other search engines too. While the message is noble, Ask's arguments are a little out of place. They suggest that Google could control the information and become the Ministry of Truth from Orwell's 1984.

Did you know that more than 75% of people in the UK use just one search engine to find information? The same search engine. The biggest search engine. The most popular search engine. Why? It's not that there aren't choices, it just seems that most people don't use them.

You can imagine we were rather curious about this, so we did some research and found that 62% of Internet users don't think about which search engine they use. We call that sleep searching, you probably call it 'getting stuff online'.

But statistics aside, if this keeps up, who knows what could happen? One company could eventually be your only source of access to information on the Web! Controlling your mind would only be a step away! Then they'll have you. All too easy...

But this is 2007, not 1984. So we're speaking up before things get out of hand. Raging against the machine kind of thing. The machine of conventional wisdom if you like!

It's always a good idea to use more than one source of information, but scaring people away with Orwellian allusions is not a good strategy. Especially when the search engine you advertise has a long way to go before becoming a reliable source of information.

The campaign is also a call to rebellion against the establishment and the layouts and the texts seem to be inspired by Che Guevara, the leader of the Communist Revolution in Cuba.

While many people suggest that Google is actually a monopoly and has become synonymous with search, the cost of switching to other search engines is minimal. Especially when Yahoo and Microsoft ask you if you want to switch to their search engines everytime you visit them. You don't have to download new software, learn how to use it and become comfortable with it (you have to do this if you switch from Windows - Microsoft's monopoly). To change your default search engine is as easy as changing your homepage or the default engine from the search box. But to do that you need to have a reason, to see a search engine that shows much better results than Google for most queries. As long as the alternatives are worse, it's no wonder that people use Google.

Update:
Ask UK shows this "smart answer" when you search for [search]. According to Matt Cutts, the same "smart answer" was displayed when you searched for [Google].


{ The first image is licensed as Creative Commons by montini. }

Google Acquires Data Visualization Technology

Google collects a lot of data, but to make sense of it, they need proper tools to mine the data and to visualize the results.

Gapminder, a non-profit organization from Sweeden, wants to "make the data more accessible and easier to use for instant visual analysis." They developed a tool called Trendalyzer "that turns boring (...) statistics into attractive moving graphics" and used it to visualize the United Nations Common Database, that has a lot of information about almost all the countries in the world. The results is Gapminder World.

Googlers found about the organization last year, when Gapminder gave a very interesting presentation at Googleplex. After hosting their visualization tool on Google's servers for almost a year, Google decided to acquire the technology from Gapminder. "Gathering data and creating useful statistics is an arduous job that often goes unrecognized. We hope to provide the resources necessary to bring such work to its deserved wider audience by improving and expanding Trendalyzer and making it freely available to any and all users capable of thinking outside the X and Y axes," says Marissa Mayer.

Gapminder is also excited about this event: "Gapminder and Google share an enthusiasm for technology that makes data easily accessible and understandable to the world. Gapminder's Trendalyzer software unveils the beauty of statistics by converting boring numbers into enjoyable interactive animations. We believe that Google's acquisition of Trendalyzer will speed up the achievement of this noble goal. Trendalyzer's developers have left Gapminder to join Google in Mountain View, where Google intends to improve and scale up Trendalyzer, and make it freely available to those who seek access to statistics."

A better Google Trends, enhanced statistics for all Google services, or maybe a cool integration with Google Earth. How do you think they'll use Trendalyzer?

XML Output for Google Search Results

Note: This tool may conflict with Google's TOS, but I thought it was interesting enough to tell you about it.

Many people would like to have feeds for Google search results. They could use them to monitor some keywords or to develop their web applications. But Google didn't show any interest in providing this feature; moreover, they cut the support for the SOAP API.

In an interesting twist, someone realized that Google actually has a way to return results in an XML file, but you need to do some work to actually retrieve them. So your URL will look the same as the standard URL for a Google search, except that you'll have to add some new parameters:

* ch=[value of a checksum]
* client=navclient-auto

Basically, you'll pretend you're Google Toolbar (that's the explanation for the client parameter) and add a checksum for the query that uses a similar algorithm to the checksum used to find the PageRank value. Unlike the API, you won't have any limitation (although Google might realize you're not Google Toolbar).

The code and some demos are available here.

Homework:
1. How does this code breaches Google's TOS more than screen scraping?
2. Do you know where is this feature used in Google Toolbar?

The Most Popular Anchor Phrases for Your Site

Google Webmaster Central shows the most popular 100 phrases used by other sites to link to your site. If you go to Statistics / Page analysis, you'll find a list of anchor phrases, obtained by removing punctuations.

Google's algorithms use those keywords to understand a page better. Sometimes a page ranks well for some keywords that are not even contained in that page, but they're used in links from other pages. For example, Yahoo.com is the fourth result for [under 18] mostly because of backlinks that use this anchor text.


"This information is useful, because it helps you know what others think your site is about. How sites link to you has an impact on your traffic from those links, because it describes your site to potential visitors," notes Vanessa Fox.

Google's Presentation Tool Does Exist

I was mentioning last month about a presentation tool that might have been developed by Google. Apparently, the application has already been created and now is being tested.

I've just found a blog (screenshot, just in case), used by one of Google's trusted testers*, that describes one of the tests.

1. Log in to writely-testing
2. Open a new presentation
3. Insert text above the slide title
4. Observe the text above the slide title is displayed below in view presentation

The post is very recent (March 13) and gives another clue that Presently will be a part of Writely (now known as Google Docs).

* Here are some snippets from his/her posts. Most of the posts have been removed.

Google Phone for Developing Countries

The Spanish Noticias.com quotes Isabel Aguilera, director of Google Spain & Portugal, who says they work on a mobile device that will facilitate access to information in developing countries. Noticias.com doesn't mention other details about the phone, but we can imagine something similar to the One Laptop per Child project.

To make it really easy to find information, the phone should have some speech recognition capabilities, like those used in Google Voice Search, so you wouldn't have to type queries. Google's search interface should also be different and should shift the focus from browsing to retrieving information. Google Maps should be aware about your location. But how to build a cheap phone optimized for Internet applications?

{ via Engadget. }

Google Updates the Corner Links

Google seems to be undecided about the purpose of the navigation links (also known as "corner bookmarks") from Gmail, Google Calendar and other applications. While the only change I see is the replacement of Groups with Google Checkout, other people saw links to Google Reader, Google Maps, Blogger and other services. Maybe users should select their favorite services and the navigation links should be displayed consistently and extensively. It's weird that Google tested a page that let you customize those links but decided not to use it.

Google Talk Gadget

Google has just launched a gadget for Google Talk. It works similarly to the version embedded into Gmail, but it's created in Flash, so it has a richer interface. Each conversation has its own tab, so the gadget doesn't occupy too much space. It's also easier to search your contacts and to invite other people.

Some extra features:

* paste a link to a video from YouTube or Google Video and you'll see the title of the video and a thumbnail. If you click on "Play" an embedded player will show up and you'll be able to see the video inside the gadget.

* paste a link to an album or photo from Picasa Web and the gadget will display a thumbnail and some details. If you click on "View", the photos will be displayed in a small slideshow.

* you can also add links to videos and albums in your status. If your contacts use the gadget, they'll be able to see the content without opening a new page.

* the gadget can be added to your web page, although it won't be very useful there.


Here's a video from Google that explains more about the gadget:



{ Thanks, Zbyszek Żółkiewski. }

Gmail's Updated Quotas

Gmail's quota will continue to increase in the following years. Google updated the code responsible for the storage counter from Gmail's counter and revealed what to expect in the future:

2835 MB on April 1st 2007
2980 MB on April 1st 2008
3125 MB on April 1st 2009
3270 MB on April 1st 2010
3415 MB on April 1st 2011
3560 MB on April 1st 2012

As you can see, Gmail's storage will increase with 145 MB a year (that's almost 0.4 MB a day). Of course, if Google doesn't change its mind.

On April 1st 2005, when Google announced the "Infinity+1" plan, the storage increased from 1 GB to 2 GB and, from that point, the growth rate was 3.456 MB a day. But from October 2005, the growth rate dropped to 0.33 MB a day.

{ via Niraj Sanghvi }

Google's Custom Built Servers

Jeff Atwood visited the Computer History Museum from Mountain View, California. Among other impressive things, he saw one of Google's first production servers.

"Instead of buying whatever pre-built rack-mount servers Dell, Compaq, and IBM were selling at the time, Google opted to hand-build their server infrastructure themselves. The sagging motherboards and hard drives are literally propped in place on handmade plywood platforms. The power switches are crudely mounted in front, the network cables draped along each side. The poorly routed power connectors snake their way back to generic PC power supplies in the rear. (...)

Even today, Google is serious about exerting total control over the servers in their now-massive server farms. They build their own high-efficiency power supplies, and conduct fascinating, public research on disk failure. Current estimates put Google's server farm at around 450,000 machines - and they're still custom built, commodity-class x86 PCs, just like they were in 1999."

Jeff thinks it's a good idea to follow Google's model and to build your own PC, by using your favorite components. He even offers some suggestions.

Sort Blogger's Feeds by Published Date

I was complaining last month about the way Blogger sorts feeds (by last update, instead of published date). Phydeaux3 found the solution: you just have to add a parameter to the feed address. Instead of:

http://BLOGNAME.blogspot.com/feeds/posts/default

you'll use:

http://BLOGNAME.blogspot.com/feeds/posts/default?orderby=published

To actually change the address of your blog's feed, you'll have to edit the template or to modify the address of the original feed, if you use FeedBurner.

In the old version of Blogger, the feed was sorted by published date, but the new Blogger changed that. The new approach is good for important updates (you made a big mistake in a post published last month) and bad if you change the wording of an old post or fix a broken link.

So it's up to you to decide if you want to have all the updated posts in your feed or just the recently published posts. I chose the latter option.

Google Navigator, Launched in China

Google launched a new service in China: Google Navigator (Dao Hang), a directory of popular sites.

"We discovered Google [search] has two substantially different kinds of user. One is the experienced search user – the basic Google interface is very suitable for them. But we also have large numbers of another kind of user with extremely limited needs: 'mouse clickers'. They just want to type key words to go quickly to their favorite website, but they are often stuck in Google for a long time without finding what they want," explains a Google China PR (focus on the meaning of the text, not on its form).


As you can see in the screenshot above (that shows an English translation of the page), Google links to popular Chinese sites, including Google's competitors. There are also categories and subcategories that let you discover the site you want to visit using only the mouse.

"The topmost categories include blogs, novels and literature, desktop wallpapers, photos, pinups and humor. (...) Different category sections can be repositioned with basic Ajax or Web 2.0 techniques. But even here, simplicity reigns. There's no potentially confusing drag and drop interface, just simple arrows that move a section up or down."

Google tries really hard to adapt to a Chinese market dominated by Baidu, the local Google*. And if some of the services designed for China are successful, they might become a part of Google's international offering.

{ Thank you, Akay. }

* Baidu has a lot in common with Google: the name (Baidu means "hundreds of times"), the sparse homepage, the advertising model and the similar list of services.

Viacom Sues YouTube for 1 Billion Dollars

Reuters reports that Viacom sued the Google-owned YouTube for "more than $1 billion over unauthorized use of its programming online". Viacom says that YouTube hosted more than 150,000 videos from its programs and these videos were watched 1.5 billion times.

Viacom doesn't think that YouTube videos increased the awareness of its shows. "YouTube's strategy has been to avoid taking proactive steps to curtail the infringement on its site. Their business model, which is based on building traffic and selling advertising off of unlicensed content, is clearly illegal and is in obvious conflict with copyright laws," stated a Viacom representative.

YouTube removed 100,000 videos last month, at Viacom's request. But Viacom wants financial compensations and a content filtering programs that prevents future copyright infringements.

Google paid $1.65 billion in stock for YouTube in October 2006, to have a good platform for its video ads. Unfortunately, Google failed to make deals with significant content owners, so many videos hosted by YouTube have a fuzzy legal status.

Kartik Hosanagar, professor of operations and information management, gives an interesting reason for the failure of the negotiations. "If Google is the one controlling all the matching of ads and consumers, [traditional media companies will be] really worried about Google's market power. That worry is reflected in Viacom's deal with Joost."

An YouTube fan explains the whole thing in an animation that imitates Stephen Colbert's show on Comedy Central.


Update: Google responds to Viacom, by saying that "YouTube has respected the legal rights of copyright holders. We will certainly not let this suit become a distraction to the continuing growth and strong performance of YouTube and its ability to attract more users, more traffic and build a stronger community."

Google Trends Shows New Data

This is for the sad people that couldn't understand why Google Trends stopped to show new data in November last year. Google Trends has new data, this time until February 21.

The Creator of the First Web Search Engine Works at Google

"The Web is BIG". A small piece of MIT's homepage from August 15, 1994.

Matthew Gray is the creator of "Wanderer, the first autonomous agent on the web, used to track web growth since 1993". The data is available online and shows that in June 1993 there were 130 web sites (only two on the .com domain), but most Internet activity was on FTP.

Based on this crawler, Matthew created Wandex, the first web search engine. "It was not very good, but again, the web was small. Within a year, numerous better web search engines appeared. But, I am proud to have had the opportunity to be first," says Matthew.

Now he works at Google. "In the years since, I started a web analytics company, earned two degrees, joined a hardware startup, and started a wireless location company. Now, I have come full circle. In February, I started work at Google in Boston. The first month has been great and it's a remarkable company."

The last he did was to map the world, by looking at the frequency of locations mentioned in the books from Google's index. "I wanted to show the Earth viewed from books, where individual mentions of locations in books combine to yield another interpretation of the globe. The intensity of each pixel is proportional to the number of times the location at a given set of coordinates is mentioned across all of the books in Google Books Search."

Make YouTube Look Like Google Video

I've always liked Google Video's interface, even though is not as community-focused as YouTube's interface. Now that most video content will go to YouTube and Google Video will become a search engine, those who prefer its layout have a way to make YouTube more like Google Video.

This Greasemonkey script expands the video to occupy more space and puts the comments next to the related videos and the video description in a small sidebar. To install the script, you need to have Firefox and the Greasemonkey extension.


For a YouTube look in Google Video, you don't need a script.

P.S.: Did you know that Aaron Boodman, who created the user script manager Greasemonkey, works at Google on client-side scripting? "I would often encounter a Web page that didn't work the way I wanted. And I'd think to myself, I could easily fix it if I could just run my own JavaScript in the page," explains Aaron why he built Greasemonkey. To learn how to write your own scripts, there's a free book called Dive into Greasemonkey.

Gmail Contact Picker

I told you about this in December last year. Google's applications have a new way to pick the Gmail contacts you want to share a video with. Instead of typing the first letters of the name or email address and let Google autocomplete, the new interface (Gmail Contact Picker) shows at least two lists: the most contacted people, all the contacts and the contact groups. There's also a search box that actually shows all the contacts that match what you typed there.

If you send mail to a constant list of people, it's a good idea to create group and there's a little button that lets you save the current list of email addresses into a group. Next time, you can just select the group from the list and click on "Add all".

Many people didn't understand how autocomplete works in Gmail and asked questions like:

I'm new to Gmail. I had an account in another webmail provider but I always used Microsoft Outlook and now I'm missing, a lot, something I had in Outlook: When I compose a mail, there is no way to access my contact list to select multiple addresses from for the To field. Should I have to have memorized all of my Contacts entries in order to access them via the type-ahead functionality one at a time?

The new interface is richer, more intuitive and looks more like an "address book". I expect to see a better a contact manager soon and to have all these enhancements in Gmail.


To see this, go to a Google Video (like this one), click on the big blue button that says "Email - Blog - Post to MySpace" and then click on the "to" link. Gmail Contact Picker is also included in Google Spreadsheets, when you invite people to collaborate.

Add Gmail Features to Thunderbird

If you use Mozilla's email client and you miss Gmail's features, there's an extension called GmailUI that may help you.

GmailUI lets you archive your mail to a folder to keep your inbox uncluttered, you can use a small part of Gmail's keyboard shortcuts and search operators like from: or subject:.

A good reason to use a desktop mail client is to manage more email accounts from a single interface. To make webmail services like Yahoo Mail or Hotmail work with Thunderbird, you'll have to download another extension and follow the instructions from this site. Unlike Yahoo Mail or Hotmail, that don't support POP3*, Gmail does and it's easy to configure in Thunderbird.

Gmail recently added a mail fetcher feature that works only with standard POP accounts.


* Yahoo Mail supports POP3 for the non-US accounts. To get POP3 for the @yahoo.com accounts, you need to pay $19.99 a year for Yahoo Mail Plus.

New Web OS to Become a Platform for Developers

I've just read the press release of a Swedish company called Xcerion, that will soon launch a "web operating system". The OS seems to incorporate every idea people expected to see it coming from Google:

The company has since its inception 2001 been in "stealth mode" and developed a revolutionary OS that changes how software is distributed, sold and developed. CEO, Daniel Arthursson says that the OS will enable the long tail business for software and provide an alternative for consumers and small businesses world-wide that are seeking a simpler, cheaper and more effective way of everyday computing. (...) Xcerion especially hopes to give people that cannot afford to buy expensive, licensed software a viable alternative. New low-cost and thin computers may also be possible to produce running Xcerion Internet OS (XIOS) on top of a stripped down Linux. (...)

During the next three years Xcerion expects to deliver hundreds of applications running on the OS – most of them for free. Mr. Arthursson says that the goal is not to reproduce all the functionality of current software – our aim is software for the masses and they often only need 40-50% of the functionality of today's software. The massive amount of applications that will be delivered is possible due to Xcerions unique technology platform and its underlying operating system (XIOS – Xcerion Internet OS) - a research and development project that has been going on for five years. The OS enables Xcerion and its partners to develop applications in XML that are visually orchestrated without any programming. (...)

The secret is data centers that scales to millions of users and an OS with zero-footprint, zero installation that also runs on top of today's popular operating systems. The OS is delivered as a dynamic service over the Internet. This approach makes the cost for supplying software extremely low per user and also solves many problems like virus threats, backups and having to install upgrades and patches.

The company, that has two former Microsoft employees as investors, promises to launch the "operating system" in the third quarter of 2007 and seems very confident about the future:

"Xcerion believes in the future of Internet self-service applications. Applications will move from the traditional desktop into the cloud and become available for free with added benefits of the dynamics of the Internet, like collaboration and social networking. We have the technology platform to create Web 2.0 applications faster, cheaper, more scalable and richer than anyone else."

Unlike Google, the company wants to open their "operating system" to developers, who will receive a share of the revenue obtained from the applications built by them. As reported by eWeek, "Xcerion is creating an online marketplace for finding, buying and selling software to enable a 'long tail' business for software by creating a new economy cluster with an online community, entrepreneurs and corporations. The service is targeted at consumers and small and midsize businesses."

There are already many "Internet operating systems" that run from the browser and replace some of the functionality of desktop software, but their performance fails to impress. Google also builds an "operating system", but most of its pieces are yet to be integrated.

Restricting Search Results to a Date Range

Sometimes you want to find a web page written in a certain interval, but Google is not very helpful and it doesn't let you restrict a search to pages written, let's say, between May 2001 and March 2002. A good reason is that Google only records the date when the crawler visited a page last time, so it's not an easy task to find the date when a page was published.

But there are some areas where Google and other search engines allow you to specify an interval for your search:

1. Blog Search. Blogs are a recent phenomenon, and the oldest blog posts indexed by Google Blog Search date back to October 1999.

Restrict your search to blogs if you want to see what ordinary people think about an event, a product or a web page. After you perform a search, click on "Choose dates" in the sidebar.

2. News Archive is an index of historical content, including including major newspapers, magazines, news archives.

To restrict the date, go to advanced search and enter an interval. Note that most of the content from Google News Archives requires subscriptions. A good way to get free access to some of the archives is to install Congoo toolbar (Windows only).

You can also find good lists of news archives, like this one.

3. Google Books, Google Scholar and Google Patents are specialized search engines for books, scholar papers and patents. The advanced search options offer different ways to choose a temporal interval.

4. Google Groups indexes Usenet archives and is an excellent place to search for old discussions that date back to 1981.

You can find, for example, the first mention of AIDS:

"Last I heard, the cause or means of transmission of AIDS was not known. I would appreciate any pointers to well-documented claims to the contrary."

"The disease sounds very frightening. I had heard about it about two weeks ago. Seems like the public should be more aware of it."

5. Google offers some options regarding the date when a page was updated, but they are pretty fuzzy: you can restrict the results to pages updated in the past 3, 6, or 12 months.

There's also a daterange operator, with the syntax daterange:startdate-enddate. Unfortunately, the dates must be entered as Julian dates (the Julian date is calculated as the number of days since January 1, 4713 BC). Based on a simple conversion algorithm, you can create a form that lets you restrict a search to a date range:



6. Alltheweb, now owned by Yahoo, has a similar option in the advanced search and it works pretty well.

7. Some queries that may help you find pages published in a certain year. You can add these to your query.

"last updated * * 2003"
"last modified * * 2003"
inurl:2003
"march * 2003"

8. Once you found a page, you might be interested to see how it looked before. The Wayback Machine has been capturing pages since 1996, but it doesn't have a way to search through those copies.

9. Google Desktop and Search History build personal archives ordered by the date you entered a query or visited a web page. Google Desktop even caches each new version of a web page, so it builds your own Wayback Machine.

But, as you can see, except for some specialized search engines, Google doesn't have memory and it's impossible to search the Internet from 2002, or to determine the precise date when a document was written.

As this article from First Monday concludes, "search engines are unreliable tools for data collection for research that aims to reconstruct the historical record or for research that aims to analyze the structure of information at a particular moment in history."

Special Google Page for Cricket World Cup

Cricket World Cup is an important sporting event held every four years since 1975. This year, the event will be hosted by the West Indies from March 13 to April 28.

Like it did last year with FIFA World Cup, Google uses this event to showcase some of its services. The most important incentives from the special page dedicated to this event are:

* two gadgets for the Personalized Homepage that display the scores and the latest news about the event

* five gadgets for Google Desktop, including one that lets you view the latest YouTube videos related to cricket matches

* a blogging contest for the best blog that covers the event (the blog should be hosted at Blog*Spot)

* a special blog created in association with Google

* an orkut community

Google tries to show the usefulness of some of its tools by connecting them to an important event. Probably the most successful promotion of a Google product happened last year and was a contest that advertised a movie called The Da Vinci Code.