In software engineering, we often encounter the terms, UNICODE, UTF-8, Base64, ISO-8859–1, and ASCII. Somehow its tough to find out a clear explanition about them.
In this 4 posts series, I’ll try to provide a “simplified” explanation of what these terms are and an over-simplified history of, how did we get where we are today.
Background
Humans, reply on graphical characters to represent data & operations. For example, language letters (A, a, B, b, C, c, etc.), numbers (1,2,3, etc), operations (+, -, ÷, etc), actions (carriage return, etc), and let's not forget the emojis (😀, 🥸, etc) and much more.
However, computers can’t understand these characters. Computer processors are made up of logic gates having only two states, ON and OFF.
Coincidentally (not really), the binary number system is comprised of only two symbols 1 and 0, those can be interpreted as ON and OFF state of logic gates in computers. Therefore, its easy to fed data and instructions to computers in the binary numbers.
But the problem is, that humans are more comfortable working with decimal than with binary. (Because our biology allowed us to easily manage numbers with 10 bases). Thanks to maths, all binary numbers can be converted to Decimal, Octal, Hexadecimal, or any other base.
With all this in mind, early computer scientists created the concept of the Character Set. They mapped some human understandable graphical characters (and some actions) to predefined decimal values, which intern converted to binary for computers.
Earlier Encodings and limited Characters
America & computers
In the early-mid 20th century, computers were used in very few nations. Initial advancements in computers happened in Europe, mostly in Germany & UK. But in the early 20th century, during the distressful times of many wars, America was far from the warzone and technologically not far behind. So, many great scientists & scholars moved there. As a result, the majority of the foundation work for computers happened only in America.
Latin-script-alphabets
English is the official language of America. It uses Latin-script-alphabets and so do many if not all other western European languages. Therefore, The earliest Character Sets were developed only for Latin script alphabets. Because until until the last quarter of the 20th century, the non-European or non-American world was irrelevant.
- Let’s skip the earlier history. We do not want hundred of pages in this post.
- EBCDIC was one of the earliest relevant 8-bit binary-code data-encoding-system, developed by IBM and mainly used in IBM mainframes.
- In the early days, different manufacturers were accustomed to using their character-set and encoding. It was resulting in trouble communicating among the different devices.
- To avoid this, in 1963 American-Standards-Association formalized American Standard Code for Information Interchange (ASCII). They used 7 bits, to map from numbers-0 to numbers-127 with 95 printable charactersand 33 control codes, all shown in the below picture.
Fun Fact —
ASCII was formulized in 1963, But it was not used until IBM incorporated it in 1981 with its release of the world’s first personal computer.
Note from Forgotten History —
Soviet engineer Vladimir Lukyanov built the world's first computer that could solve partial differential equations, as early as 1936. But due to Stalin's hostile nature toward computers, there were no inventions in the Stalin era. And the Soviet was left behind in the computer race. Still, GOST-10859 was a soviet language character set used in the late-early days.
Confusion in terminology
Encoding vs CharacterSet
The confusion in understanding various terms comes because ASCII, which was used for a long time, was both a character set and an encoding scheme. As it has a one-to-one mapping from binary value to character/control code.
When the character-set increased, these two terms character-set & encoding (or decoding which is also referred to as encoding, go figure) used two different technologies, more on it in a later post in this series.
This was all about the motivation behind the character set, early charsets, encoding, and ASCII. The next post will see what happened when the computers reached a bigger audience.
Comments