Google added a small button that lets you collapse the group of links, but the setting wasn't preserved when you reloaded the page because of a bug. Now you can permanently hide the links to "contacts" and "tasks".
If you ignore the blue bar at the top of the page and you don't click on "Mail", you'll no longer see the two links. Gmail could add an option to move the horizontal navigation bar below the list of labels, even as a Gmail Labs experimental feature. I partially solved this problem by enabling "Quick Links" in Gmail Labs and adding links to the contact manager and to a mail view (is:unread).
Another option is to use this userscript that moves the links below the labels. The script should work in Firefox (Greasemonkey is required), Chrome and Opera.
Apparently, Gmail's links bar is quite important and it's likely that Gmail will add links to other services. A Google employee detailed its purpose:
"Gmail is home to a few different apps, including Mail, Contacts, and Tasks. The links on the left below the app navigation change depending which app you're using, while the app navigation links remain consistent so that you can go between apps. If you're in the Contacts app, the big action button becomes the way to add a new contact. We use this button as the way to create things in other products like Google Docs and Google Calendar, too."