|
|
@@ -25,17 +25,27 @@
|
|
|
|
|
|
|
|
|
\section{Compression aproaches}
|
|
|
-The process of compressing data serves the goal to generate an output that is smaller than its input data. In many cases, like in gene compressing, the compression is idealy lossless. This means it is possible for every compressed data, to receive the full information that were available in the origin data, by decompressing it. Lossy compression on the other hand, might excludes parts of data in the compression process, in order to increase the compression rate. The excluded parts are typicaly not necessary to transmit the origin information. This works with certain audio and pictures files or network protocols that are used to transmit video/audio streams live.
|
|
|
-For \acs{DNA} a lossless compression is needed. To be preceice a lossy compression is not possible, because there is no unnecessary data. Every nucleotide and its position is needed for the sequenced \acs{DNA} to be complete. For lossless compression two mayor approaches are known: the dictionary coding and the entropy coding. Both are described in detail below.\\
|
|
|
+The process of compressing data serves the goal to generate an output that is smaller than its input data.\\
|
|
|
+In many cases, like in gene compressing, the compression is idealy lossless. This means it is possible for every compressed data, to receive the whole information, which were available in the origin data, by decompressing it.\\
|
|
|
+Before going on, the difference between information and data should be emphasized.\\
|
|
|
+% excurs data vs information
|
|
|
+Data contians information. In digital data clear, physical limitations delimit what and how much of something can be stored. A bit can only store 0 or 1, eleven bits can store up to $2^11$ combinations of bits and a 1\acs{GB} drive can store no more than 1\acs{GB} data. Information on the other hand, is limited by the way how it is stored. In some cases the knowledge received in a earlier point in time must be considered too, but this can be neglected for reasons described in the subsection \ref{k4:dict}.\\
|
|
|
+% excurs information vs data
|
|
|
+The boundaries of information, when it comes to storing capabilities, can be illustrated by using the example mentioned above. A drive with the capacity of 1\acs{GB} could contain a book in form of images, where the content of each page is stored in a single image. Another, more ressourcefull way would be storing just the text of every page in \acs{UTF-16}. The information, the text would provide to a potential reader would not differ. Changing the text encoding to \acs{ASCII} and/or using compression techniques would reduce the required space even more, without loosing any information.\\
|
|
|
+% excurs end
|
|
|
+In contrast to lossless compression, lossy compression might excludes parts of data in the compression process, in order to increase the compression rate. The excluded parts are typicaly not necessary to persist the origin information. This works with certain audio and pictures formats, and in network protocols \cite{cnet13}.
|
|
|
+For \acs{DNA} a lossless compression is needed. To be preceice a lossy compression is not possible, because there is no unnecessary data. Every nucleotide and its position is needed for the sequenced \acs{DNA} to be complete. For lossless compression two mayor approaches are known: the dictionary coding and the entropy coding. Both are described in detail below \cite{cc14}.\\
|
|
|
|
|
|
\subsection{Dictionary coding}
|
|
|
-Dictionary coding, as the name suggest, uses a dictionary to eliminate redundand occurences of strings. Strings are a chain of characters representing a full word or just a part of it. This is explained shortly for a better understanding of dictionary coding but is of no great relevance to the focus of this work:
|
|
|
+\label{k4:dict}
|
|
|
+Dictionary coding, as the name suggest, uses a dictionary to eliminate redundand occurences of strings. Strings are a chain of characters representing a full word or just a part of it. For a better understanding this should be illustrated by a short example:
|
|
|
% exkurs
|
|
|
Looking at the string 'stationary' it might be smart to store 'station' and 'ary' as seperate dictionary enties. Which way is more efficient depents on the text that should get compressed.
|
|
|
% end exkurs
|
|
|
The dictionary should only store strings that occour in the input data. Also storing a dictionary in addition to the (compressed) input data, would be a waste of resources. Therefore the dicitonary is made out of the input data. Each first occourence is left uncompressed. Every occurence of a string, after the first one, points to its first occurence. Since this 'pointer' needs less space than the string it points to, a decrease in the size is created.\\
|
|
|
|
|
|
-Unfortunally, known implementations like the ones out of LZ Family, do not use probabilities to compress and are therefore out of scope for this work. Since finding repetations and their location might also be improved, this chapter will remain.
|
|
|
+Unfortunally, known implementations like the ones out of LZ Family, do not use probabilities to compress and are therefore out of scope for this work. Since finding repeting sections and their location might also be improved, this chapter will remain.
|
|
|
+
|
|
|
|
|
|
% unuseable due to the lack of probability
|
|
|
\mycomment{
|
|
|
@@ -46,14 +56,19 @@ The computer scientist Abraham Lempel and the electrical engineere Jacob Ziv cre
|
|
|
% example
|
|
|
}
|
|
|
|
|
|
+ % (genomic squeeze <- official | inofficial -> GDC, GRS). Further \ac{ANS} or rANS ... TBD.
|
|
|
+\subsection{\ac{LZ77}}
|
|
|
+ \ac{LZ77} basically works, by removing all repetition of a string or substring and replacing them with information where to find the first occurence and how long it is. Typically it is stored in two bytes, whereby more than one one byte can be used to point to the first occurence because usually less than one byte is required to store the length.
|
|
|
+
|
|
|
+
|
|
|
\subsection{Shannons Entropy}
|
|
|
-The founder of information theory Claude Elwood Shannon described entropy and published it in 1948 \autocite{Shannon_1948}. In this work he focused on transmitting information. His theorem is applicable to almost any form of communication signal. His findings are not only usefull for forms of information transmition.
|
|
|
+The founder of information theory Claude Elwood Shannon described entropy and published it in 1948 \cite{Shannon_1948}. In this work he focused on transmitting information. His theorem is applicable to almost any form of communication signal. His findings are not only usefull for forms of information transmition.
|
|
|
|
|
|
% todo insert Fig. 1 shannon_1948
|
|
|
|
|
|
Altering this figure shows how it can be used for other technology like compression.\\
|
|
|
The Information source and destination are left unchanged, one has to keep in mind, that it is possible that both are represented by the same phyiscal actor.
|
|
|
-transmitter and receiver are changed to compression/encoding and decompression/decoding and inbetween ther is no signal but any period of time \autocite{Shannon_1948}.\\
|
|
|
+transmitter and receiver are changed to compression/encoding and decompression/decoding and inbetween ther is no signal but any period of time \cite{Shannon_1948}.\\
|
|
|
|
|
|
Shannons Entropy provides a formular to determine the 'uncertainty of a probability distribution' in a finite field.
|
|
|
|
|
|
@@ -65,7 +80,7 @@ Shannons Entropy provides a formular to determine the 'uncertainty of a probabil
|
|
|
\label{k4:entropy}
|
|
|
\end{figure}
|
|
|
|
|
|
-He defined entropy as shown in figure \ref{k4:entropy}. Let X be a finite probability space. Then x in X are possible final states of an probability experimen over X. Every state that actually occours, while executing the experiment generates infromation which is meassured in \textit{Bits} with the part of the formular displayed in \ref{k4:info-in-bits}\autocite{delfs_knebl,Shannon_1948}:
|
|
|
+He defined entropy as shown in figure \ref{f4:entropy}. Let X be a finite probability space. Then x in X are possible final states of an probability experimen over X. Every state that actually occours, while executing the experiment generates infromation which is meassured in \textit{Bits} with the part of the formular displayed in \ref{k4:info-in-bits}\cite{delfs_knebl,Shannon_1948}:
|
|
|
|
|
|
%\bein{math}
|
|
|
% log_2(frac{1}{prob(x)}) \equiv - log_2(prob(x)).
|
|
|
@@ -74,7 +89,7 @@ He defined entropy as shown in figure \ref{k4:entropy}. Let X be a finite probab
|
|
|
\centering
|
|
|
\includegraphics[width=8cm]{k4/information_bits.png}
|
|
|
\caption{The amount of information measured in bits, in case x is the end state of a probability experiment.}
|
|
|
- \label{k4:info-in-bits}
|
|
|
+ \label{f4:info-in-bits}
|
|
|
\end{figure}
|
|
|
|
|
|
%todo explain 2.2 second bulletpoint of delfs_knebl. Maybe read gumbl book
|
|
|
@@ -92,10 +107,6 @@ This means the intervall start of the character is noted, its intervall is split
|
|
|
To encode in binary, the binary floating point representation of a number inside the intervall, for the last character is calculated, by using a similar process, described above, called subdividing.
|
|
|
% its finite subdividing because processors bottleneck floatingpoints
|
|
|
|
|
|
- % (genomic squeeze <- official | inofficial -> GDC, GRS). Further \ac{ANS} or rANS ... TBD.
|
|
|
-\subsection{\ac{LZ77}}
|
|
|
- \ac{LZ77} basically works, by removing all repetition of a string or substring and replacing them with information where to find the first occurence and how long it is. Typically it is stored in two bytes, whereby more than one one byte can be used to point to the first occurence because usually less than one byte is required to store the length.
|
|
|
-
|
|
|
\subsection{Huffman encoding}
|
|
|
% list of algos and the tools that use them
|
|
|
The well known Huffman coding, is used in several Tools for genome compression. This subsection should give the reader a general impression how this algorithm works, without going into detail. To use Huffman coding one must first define an alphabet, in our case a four letter alphabet, containing \texttt{A, C, G and T} is sufficient. The basic structure is symbolized as a tree. With that, a few simple rules apply to the structure:
|
|
|
@@ -109,7 +120,8 @@ The well known Huffman coding, is used in several Tools for genome compression.
|
|
|
\item the less weight a node has, the higher the probability is, that this node is read next in the symbol sequence
|
|
|
\end{itemize}
|
|
|
The process of compressing starts with the nodes with the lowest weight and buids up to the hightest. Each step adds nodes to a tree where the most left branch should be the shortest and the most right the longest. The most left branch ends with the symbol with the highest weight, therefore occours the most in the input data.
|
|
|
-Following one path results in the binary representation for one symbol. For an alphabet like the one described above, the binary representation encoded in ASCI is shown here \texttt{A -> 01000001, C -> 01000011, G -> 01010100, T -> 00001010}. An imaginary sequence, that has this distribution of characters \texttt{A -> 10, C -> 8, G -> 4, T -> 2}. From this information a weighting would be calculated for each character by dividing one by the characters occurence. With a corresponding tree, created from with the weights, the binary data for each symbol would change to this \texttt{A -> 0, C -> 11, T -> 100, G -> 101}. Besides the compressed data, the information contained in the tree msut be saved for the decompression process.
|
|
|
+Following one path results in the binary representation for one symbol. For an alphabet like the one described above, the binary representation encoded in ASCI is shown here \texttt{A -> 01000001, C -> 01000011, G -> 01010100, T -> 00001010}. An imaginary sequence, that has this distribution of characters \texttt{A -> 10, C -> 8, G -> 4, T -> 2}. From this information a weighting would be calculated for each character by dividing one by the characters occurence. With a corresponding tree, created from with the weights, the binary data for each symbol would change to this \texttt{A -> 0, C -> 11, T -> 100, G -> 101}. Besides the compressed data, the information contained in the tree msut be saved for the decompression process.\\
|
|
|
+% todo shannon fano mention. SF might be older than huffman and inspired it?
|
|
|
|
|
|
\section{DEFLATE}
|
|
|
% mix of huffman and lz77
|
