What others are trying to say here, and what I think you are asking, fall into two things.
There are markup tags, the code. Then there is the text and other things you're marking up.
The reason <, >, &, ', and " are special is because these are special characters for HTML and SGML themselves. You use those for the markup (code) and the HTML character entities < > & ' " in the text (the regular, readable text you're marking up). Note that semi-colon (";") is part of the character entity. You have to include it too.
In the regular text you're marking up, Those same special characters from the paragraph above need to be character entities < > etc. or they will confuse the HTML display. Because your web pages may be viewed on Windows, Macintosh, Linux, iOS, Android, or whatever else in the future, any special symbols like curly quotes ‘ ’ “ ” and dashes — and accented letters é and so on, need to be those HTML entities too. Or else your meta charset tag *and* the file's Save As format need to be set properly to the UTF-8 or other character set you need, in order to display properly. Hint: If you see two strange symbols in a row, instead of the proper/intended character, it's a tip the encoding is off. This happens to anyone, big sites or little. For foreign language text, especially non-European or non-Latin alphabets, you'll really need the charset to be right, because wading through a bunch of ' (and &#dddd; whatever) is really not going to make anyone happy.