What Are Web Standards?
Web Standards are the guidelines and recommendations of best practices for the World Wide Web. Web Standards cover everything internet related, including the development of websites. Recommendations and specifications for Web Standards come from a variety of Boards and Councils, the most notable being World Wide Web Consortium (W3C). The W3C is an international community comprised of hundreds of companies including Microsoft, Apple, Sony, IBM and even Walt Disney and the Mayo Clinic. These companies, along with full time W3C staff and the public, all work together to develop Web Standards.
Even though your website is not required to follow Web Standards, there are several reasons why it should.
Following Web Standards takes less code. Plain and simple. Many claim a website built on Web Standards takes less than half the code it would take if you didn't follow Web Standards. Obviously that means far less programming work to initially build a site as well as to maintain it.
A major factor in load time is how much code has to download to someone's computer before they can see the website. Clearly if Web Standards result in half the code, websites built on Web Standards will load much more quickly.
If you have a very high usage site, bandwidth costs can be a considerable expense. Since a site built on Web Standards employs less code, the file sizes are smaller, and less bandwidth is required.
A website developed using Web Standards will continue to work in future browsers. This is because all browser manufacturers are themselves committed to Web Standards. Gone are the days of website upgrades every time a new version of any browser is released. That's a considerable savings right there.
Many of the specifications within Web Standards cover accessibility for anyone with any type of physical challenge. But really, these Standards help a lot more people than just those physically challenged. They enable the average user to better navigate a website. By not following these Standards, your site will frustrate a significant percentage of your visitors.
Search Engine Optimization
Search Engine Optimization (SEO) is the practice of increasing the traffic your site receives from Search Engines. Search Engines are machines (computer programs) that read every line of code on your website. Following Web Standards makes it much easier for Search Engines to read your code, and index your content. This then makes it easier for people using Search Engines to find your site.
Balanced with Common Sense and Experience
There always is an exception, or several. Web Standards should be followed, but they should be followed in the spirit of what they are - recommendations. And not all well thought out recommendations are true Best Practices in all situations. Common sense (or experience) and Web Standards must be balanced. For example, the Standard for a link when using XHTML Strict is that you should never open it in a new window. This is impractical in the real world and there are some situations where opening in a new window IS the Best Practice.
Web Standards have improved the web immeasurably. Adopting Web Standards on your site will absolutely provide all the benefits I listed. At the same time, Web Standards should not be given carte blanche, but rather mitigated with common sense and experience.