%SUMMARY
%- ABSTRACT
%- INTRODUCTION
%# BASICS
%- \acs{DNA} STRUCTURE
%- DATA TYPES
% - BAM/\ac{FASTQ}
% - NON STANDARD
%- COMPRESSION APPROACHES
% - SAVING DIFFERENCES WITH GIVEN BASE \acs{DNA}
% - HUFFMAN ENCODING
% - PROBABILITY APPROACHES (WITH BASE?)
%
%# COMPARING TOOLS
%- 
%# POSSIBLE IMPROVEMENT
%- \acs{DNA}S STOCHASTICAL ATTRIBUTES 
%- IMPACT ON COMPRESSION

% todo: use this https://www.reddit.com/r/bioinformatics/comments/7wfdra/eli5_what_are_the_differences_between_fastq_and/
% bigger picture - structure chapters like this:
% what is it/how does it work
% where are limits (e.g. BAM)
% what is our focus (and maybe 'why')

\chapter{Datatypes}
\label{chap:datatypes}
As described in previous chapters \ac{DNA} can be represented by a String with the buildingblocks A,T,G and C. Using a common fileformat for saving text would be impractical because the ammount of characters or symbols in the used alphabet, defines how many bits are used to store each single symbol.\\
Storing a single \textit{A} with \ac{ascii} encoding requires 8 bit (\,excluding magic bytes and the bytes used to mark \ac{EOF})\, since there are at least $2^8$ or 128 displayable symbols. Since the \ac{DNA} buildingblocks only require a minimum of four letters, two bits are needed e.g.: \texttt{00 -> A, 01 -> T, 10 -> G, 11 -> C}. Depending on the sequencing method, more than four letters are used. The complex process of sequencing \ac{DNA} is not 100\% preceice, so additional Letters are used to mark nucelotides that could not or could only partly get determined.\\
More common everyday-usage text encodings like unicode require 16 bits per letter. So settling with \ac{ascii} has improvement capabilitie but is, on the other side, more efficient than using bulkier alternatives like unicode.\\

Several people and groups have developed different fileformats to store genomes. Unfortunally for this work, there is no defined standard filetype or set of filetypes therefor one has to gather information on which types exist and how they function by themself. In order to not go beyond scope, this work will focus only on fileformats that fullfill following factors:\\
\begin{itemize}
  \item{the format has reputation, either through a scientific paper, that prooved its superiority to other relevant tools or through a broad ussage of the format.}
  \item{the format does not include \ac{IUPAC} codes besides A, C, G and T \autocite{iupac}.}
  \item{the format is open source.}
  \item{the compression methode used in the format is based on probabilities.}
\end{itemize}

Some common fileformats would be:
\begin{itemize}
% which is relevant? 
  \item{FASTA}
  \item{\ac{FASTQ}}
  \item{twoBit}
  \item{SAM/BAM}
  \item{VCF}
  \item{BED}
\end{itemize}
% src: http://help.oncokdm.com/en/articles/1195700-what-is-a-bam-fastq-vcf-and-bed-file

Since methods to store this kind of Data are still in development, there are many more filetypes. The few, mentioned above are used by different organisations and researchers and are backed by a scientific publication. % todo find sources to both points in last sentence
%rewrite:
In order to not go beyond the scope, this paper will only focuse on compression tools which are using standard formats.

\section{\ac{FASTQ}}
Is a text base format for storing sequenced data. It saves nucleotides as letters and in extend to that, the quality values are saved.
\ac{FASTQ} files are split into multiples of four, each four lines contain the informations for one sequence. The exact structure of \ac{FASTQ} format is as follows:
\texttt{
Line 1: Sequence identifier aka. Title, starting with an @ and an optional description.\\
Line 2: The seuqence consisting of nucleoids, symbolized by A, T, G and C.\\
Line 3: A '+' that functions as a seperator. Optionally followed by content of Line 1.\\
Line 4: quality line(s). consisting of letters and special characters in the \ac{ascii} scope.}\\

The quality values have no fixed type, to name a few there is the sanger format, the solexa format introduced by Solexa Inc., the Illumina and the QUAL format which is generated by the PHRED software. 
The quality value shows the estimated probability of error in the sequencing process.
[...]

\section{Sequence Alignment Map}
% src https://github.com/samtools/samtools
\ac{SAM} often seen in its compressed, binary representation \ac{BAM} with the fileextension \texttt{.bam}, is part of the SAMtools package, a uitlity tool for processing SAM/BAM and CRAM files. The SAM/BAM file is a text based format delimited by TABs. It uses 7-bit US-ASCII, to be precise Charset ANSI X3.4-1968 as defined in RFC1345. The structure is more complex than the one in \ac{FASTQ} and described best, accompanied by an example:

\begin{figure}[ht]
  \centering
  \includegraphics[width=15cm]{k3/sam-structure.png}
  \caption{SAM/BAM file structure example}
  \label{k_datatypes:bam-struct}
\end{figure}

\subsection{Valid Symbols}
%- char set restrictions: \ac{ascii} in range ! to ~ apart from '@'
%- ... as regex (todo: describe in words): 
% This RegEx -> \[0-9A-Za-z!#$%&+./:;?@^_|~-][0-9A-Za-z!#$%&*+./:;=?@^_|~-]*
The regulare expression, shown above, filters touple of characters from a to z in lower and uppercase, numbers and several special characters. There are minor differences between the special chars first and second element of the touple, in the latter '*' and '=' are allowed which are not allowed in the first.

%\subsection{Index File}
 
% notes
%- may contian metadata: magic string, ref seq, bins, offsets, unmapped reads
%- allows viewing BAM data (localy and remote via ftp/http)
%- file extention: <filename>.bam.bai

%- stores more data than \ac{FASTQ}
 
% src: https://support.illumina.com/help/BS_App_RNASeq_Alignment_OLH_1000000006112/Content/Source/Informatics/BAM-Format.htm
%- allignment section includes
% - RG Read Group
% - BC Barcode Tag
% - SM Single-end alignment quality
% - AS Paired-end alignment quality
% - NM Edit distance tag
% - XN amplicon name tag

%- BAM index files nameschema: <filename>.bam.bai 

\section{Compressed Reference-oriented Ailgnment Map}
\ac{CRAM} was developed as an alternative to the \ac{SAM} and \ac{BAM} Format. It specification is maintained by \ac{GA4GH}. It features both lossy and lossless compression mode. Since it is not relevant to this work, the lossy compression is ignored from here on. Even though it is part of \ac{GA4GH} suite, the file format can be used independently.\\
The format saves data in containers which consist out of slices. Each slice is represented by a line in the file. Container and slices each store metadata in a header. Data is stored as blocks in slices, in a compressed form.
