Understanding encoding in details and various encoding mechanisms.

In very simple terms encoding is the process of converting a information in a format which makes it efficient for storage or transmission without loosing the information contained in it.Lets take a deeper look into it and try to understand the mystery around encoding and decoding .

Lets take a human perspective on language,

• Sound have been encoded and decoded to represent certain information.
• Same sound if taken out of context or in a different language will mean something else.
• The schema and semantics of Sound encoding and decoding is with the humans and humans have consistently maintained the schema and symantics from one generation to other.

To be more specific .

KAPUT in GERMAN means “broken” while in Hindi it means a “evil son”. Its the same word, same pronunciation still the meaning changes, as GERMAN and HINDI are different languages and schema to extract out the information from the same sound is different.

Lets put the same analogy in computers, in a computer system everything is represented at sequence of 1 and 0s. How does one represent alphabets , numbers and special character as the underlying storage format is just 1 and 0s .

Lets take a Byte and understand what in the raw forms it mean.

• In raw form Byte is a sequence of 8 bits with each bit can take value of 0 or 1.
• The same byte can be treated as a char or short.
• byte implicitly has no information about how its going to be treated.
• Same byte(binary sequence) if treated as a char will reflect something and something else if treated as int.
• It is based on the interpretation of the byte of what information its going to convey.

ASCII (American Standard code for Information Interchange) is 7 bit character encoding mechanism thus providing mapping between 2 power 7 = 128 unique sequence (1 and 0) to represent symbols  (a-z, A-Z, 0-9 and all other character).

As 128 unique sequences are not sufficient to include characters of all world languages UTF-8, UTF-16 and UTF-32 encoding schemes were also adopted.

[addToAppearHere]

Key Take Aways.

• Encoding holds the information of how to represent a sequence of 1 and 0s to which specific character.
• There is no direct relation between the represented characters and decimal numbers.
• mapping was from sequence of 1 and 0 (00110000 => 0) and decimal representation of  00110000 is 48 .
• Do understand humans have been trained to think in terms of decimal numbers hence we convert 00110000 to 48 and say 48 is ASCII integer for 0.
• This is associative relationship and not direct, this has more to do with how we think.
• Do remember decimal number, octal number, hex number are themselves a kind of encoding over the binary sequences.
• On the same Logic the whole idea of File System also exist where the schema of how to interpret the binary data from hard disk stays with the OS and hard disk being completely agnostic of any kind of file, character word, lines sentences.

Code

package com.big.data.java.samples;

import java.nio.ByteBuffer;
import java.util.stream.IntStream;


public class EncodeDecode {

    public static void main(String[] args) {

        // In Java character is of two bytes but not the chars in the String 
        // Depending on the encoding settings of the JVM which is UTF-8 String occupies one Byte
        // "01010101" String length is 8 bytes and not 16 bytes
     String characterRepresentation = "01010101";

        // Raw bytes , the schema is no more relvent
        // Raw bytes are sequence of 1 and 0
     ByteBuffer rawBytes = ByteBuffer.wrap(characterRepresentation.getBytes());

        // Char is 2 bytes 
        // hence on each call its consuming 2 bytes  hence only 4 iteration required
     IntStream.range(0, 4)
         .forEach(value -> 
             System.out.println("Interprete as CHAR value is " + rawBytes.getChar()));

        // rewind the ByteBuffer internal pointer
     rawBytes.rewind();

     IntStream.range(0, 8)
         .forEach(value -> 
             System.out.println("Interprete as BYTE value is " + rawBytes.get()));

        // rewind the ByteBuffer internal pointer
     rawBytes.rewind();

        // Short is 2 bytes 
        // hence on each call its consuming 2 bytes hence only 4 iteration required
     IntStream.range(0, 4)
       .forEach(value -> 
           System.out.println("Interprete as SHORT value is " + rawBytes.getShort()));

        // rewind the ByteBuffer internal pointer
     rawBytes.rewind();

        // rawBytes is of length 4 hence we can interprete it as Int
        // In java Int is 4 bytes
     IntStream.range(0, 2)
       .forEach(value -> 
           System.out.println("Interprete as SHORT value is " + rawBytes.getInt()));

 

Key Take Aways

 ByteBuffer rawBytes = 
ByteBuffer.wrap(characterRepresentation.getBytes());

rawBytes.get()

rawBytes.getShort()

[addToAppearHere]

OUTPUT

Interprete as CHAR value is 〱
Interprete as CHAR value is 〱
Interprete as CHAR value is 〱
Interprete as CHAR value is 〱

Interprete as BYTE value is 48
Interprete as BYTE value is 49
Interprete as BYTE value is 48
Interprete as BYTE value is 49
Interprete as BYTE value is 48
Interprete as BYTE value is 49
Interprete as BYTE value is 48
Interprete as BYTE value is 49

Interprete as SHORT value is 12337
Interprete as SHORT value is 12337
Interprete as SHORT value is 12337
Interprete as SHORT value is 12337

Interprete as INT value is 808529969
Interprete as INT value is 808529969