Showing posts with label Data Structure. Show all posts
Showing posts with label Data Structure. Show all posts

Sunday, June 2, 2019

Merging heaps in O(log N)?

Can you merge two heaps in O(log N)? Turns out if they are leftist heaps, you can!
But what is a leftist heap, and how is it different from binary heap?

Recall that binary heap is both complete and balanced. Hence it looks something like this:
               1
             /   \
            2     3
           / \   /
          5   6 7
             Fig 1
It's clear that it supports O(log N) operations.

In contrast, this is what leftist heap looks like:
               1
              / \
             3   2
            /   /
           5   7
          /
         6
             Fig 2
It's neither complete nor balanced. Yet, it supports O(log N) operations too!

How does it work?

Monday, June 6, 2016

Data Structure: Fenwick Tree and Range Update

Something led me to read up on Fenwick Tree (a.k.a Binary Indexed Tree). There are lots of great tutorials that explain what this data structure is, and how it is useful. A google search is all you need. I would like to write up some short discussion on the tree, and also about a technique that is used to allow this tree to support range updates.

Sunday, May 8, 2016

How is deque implemented?

It sounds really simple, and I have never really given it a thought before. However just now I realised that it is not as straight-forward as I might have led to believe!

A refresher: deque is a data structure that supports O(1) append operations on both ends (head and tail) and O(1) random look-ups.

Hence deque has a similar characteristics as a vector, only better. Wait a minute, how could that be? If deque is simply a doubly linked-list, then random look-ups will be O(N). That's when I realise that I have been so ignorant and oblivious.

Tuesday, November 25, 2014

Conveyor Belts (Codeforces Round 278 Div. 1 Problem D)

Problem Statement:
Codeforces 278 Div. 1
D. Conveyor Belts

Solution:
What a challenging problem, I could never imagine solving this kind of problem during a real contest. Cool people, cool stuff, cool problem. Let's keep learning!

The problem is actually a range minimum query (RMQ) problem and hence solvable using any data structure that can efficiently compute such queries (\(O(N\lg{N})\) segment tree, \(O(\sqrt{N})\)  square root decomposition, and the like). How can we model this as a RMQ problem?

Sunday, November 23, 2014

Data Structure: Sliding Window Minimum / Monotonic Queue

Given an array of elements \( a_0, a_1, a_2, \ldots, a_n \), and queries Q(i,i+L) which means "find the minimum element in \(a_i, a_{i+1}, \ldots, a_{i+L}\) ". How can we answer such queries efficiently?

We can have an \(O(n \lg{n})\) complexity by using a minimum priority queue, RB tree, or a binary tree representation of multiset, but there in this setting we can implement a data structure called monotonic queue which only requires \(O(n)\) in construction. The implementation of this data structure requires a dequeue.

Let D be the dequeue which maintain a pair of information (i, \(a_i\)). An important property of D that we will maintain is that element in D will always be in sorted order (invariant). We will first start with an empty D, and will insert \(a_i\) and remove elements from D accordingly as we iterate from the left to the right of array \(a\) (which is from i = 0, 1, 2, ..., n).

Suppose that we are now at index \(i\) and considering to add \(a_i\). Notice that when \(a_i\) is added, all elements in (j, \(a_j\)) in D such that \(a_j\) is bigger than \(a_i\) can never be a minimum value as we go forward, hence they can be removed from D. Furthermore, if the element (i-L-1, \(a_{i-L-1}\)) is in D (which will be located at the top of D if it exists), we remove that element as well. Lastly, we append \(a_i\) at the back of D. Then we will have Q(i-L, i) = the top element in D when we reach index i. Since each element will enter and leave D only once, we have a total of \(O(N)\) operations. :D

Thursday, November 13, 2014

Codeforces Round #277 (Div. 2) Problem E. LIS of Sequence

Follow Up to this post.

Problem Statement:
E. LIS of Sequence

Solution:
This problem is quite nice to think about :D. Before attempting this problem, it might be useful to have a knowledge beforehand on how to solve LIS problem in \(O(N\lg{N})\) complexity using binary search.

This problem is more or less an extension to that idea. First we have an array of stacks stack[i] which is initialized with the following conditions:
1. the size of the array is initially 1.
2. push \(a_1\) into stack[0], so stack[0] contains \(a_1\).
Now we iterate from \(a_2,\ldots\) onwards till the end of the elements:
1. find the position of the maximum element amongst all elements on top of the stacks such that \(a_i\) is equal or larger.
2. if such position exist, push our current \(a_i\) into that stack. Otherwise, we add a new stack (hence incrementing the size of the array by 1) and push \(a_i\) to that newly created stack.

Monday, October 27, 2014

Data Structure for Frequent Range Updates

Given an update operation as following: from index i to index j, we add p to each array[i]. What kind of data structure that can support this operation efficiently?

Of course segment tree and fenwick tree does this well in \(O(N\lg{N})\) complexity for building the tree, \(O(\lg{N})\) for updates (insertion), and \(O(\lg{N})\) for processing search queries. However, what if we have a use case where we do updates very very frequently, like 99% of the time?

Two days ago I've discovered this data structure while reading on some people's code, and I'm not really sure whether there is a standard way to call it. The data structure I am about to present here does the updates in O(1) (yeap) but to do a search query we have at worst O(N) at each time. But the simplicity is really striking.

We use an array to keep track on the updates information, call this mark[]. For each updates in the form of (p, i, j), we do:
1. mark[j] += p
2. mark[i-1] -= p
And this information suffices to show that we intent to add p to each and every cell in i to j.

How do we process the information when we need to do a retrieval of data? Once the need arise, we do a one sweep operation from i = N to 1: mark[i-1] += mark[i]
Then each mark[i] represent how much values we already added to element i throughout the updating process!

Sunday, October 19, 2014

a bit of ds: QuadTree (and Implementation)

A few days ago I made a little implementation on QuadTree on my (still very green) github io page, here is a link: QuadTree Demo

What is cool about this data structure is that like binary tree, it is recursive in nature, so you can write very little code and achieve a highly complex work. I think it is widely use in game programming for collision detections between objects, especially when the number of objects are pretty huge where \(O(N^2)\) check won't cut it.

QuadTree is conceptually very simple. Suppose we have a "quadrant". We will further split this quadrant into four equal parts if and only if there is at least two elements inside this current quadrant. Then we recursively do the same on the four resulting parts. That's it!

For collision detection application, you can determine beforehand how deep is the recursion or how big is the area for which a collision can occur.

Tuesday, July 15, 2014

a bit of data structure: Segment Tree

This data structure is used to efficiently support queries which request for the index of the minimum element in a given range.

The idea is using a divide and conquer: given a range \((i, \ldots, j)\), and then we find the index to the minimum element for \((i, \ldots, \frac{i+j}{2})\) and \((\frac{i+j}{2}+1, \ldots, j)\) seperately, and finally combine the result by choosing the lower element of the two indexes as the minimum index for \(i, \ldots, j\).

The tree is built on a one dimensional array just like Binary Heap and it requires \(O(4N)\) space. To process a query, we traverse down the tree to find the sub-interval which lies totally inside the query range.

Using Segment Tree we can maintain a dynamically changing information on a static data structure as update operations can be implemented efficiently. Furthermore, to increase its performance, we can opt to have a lazy propagation of updates, by maintaining an extra array that mark whether a segment have to be updated or not.

Here is a quick implementation in C/C++ :



vector<int> stree(4*N + 1,0);
vector<int> arr(N + 1,0);

int build(int i, int j, int k)
{
 if(i == j)
 {
  stree[k] = i;
  return stree[k];
 }
 int mid = (i+j)/2;
 int left=-1,right=-1;
 left = build(i,mid,2*k);
 right = build(mid+1,j,2*k+1);
 if(arr[left] > arr[right])
  stree[k] = right;
 else 
  stree[k] = left;
 return stree[k];
}

int rmq(int p, int i, int j, int L, int R)
{
 //invalid query
 if(i > R || j < L) return -1;
 if(i <= L && R <= j) return stree[p];
 int mid = (L+R)/2;
 int left = rmq(2*p, i, j, L, mid);
 int right = rmq(2*p + 1, i, j, mid+1, R);
 if (left == -1) return right;
 if (right == -1) return left;
 return (arr[left] < arr[right]) ? left : right;
}

Monday, July 14, 2014

a bit of data structure: Binary Heap

In my opinion, Binary Heap is one of the most ingenious data structure in the sense that its implementation can be done on a static 1D array, albeit its nature of being a dynamic non-linear data structure.

Binary Heap maintains the property that the parent node has a key less its children (if it has any). Each node has 2 child at most, a left child and a right child. The relationship between parent node and child node can be laid down on an array of size N, where index \( 1 \leq i \leq N\) represent a node in the Binary Heap. Furthermore, left child of \(i\) is placed at index \(2i\) while right child is placed at \(2i + 1\). Under this relationship, an array that maintain these properties will form a full binary tree representing the heap.

Insertion is done by appending the new element at the ending entry of the array, and a procedure called Up-Heapify is called. It recursively check whether the parent of the new element is greater than itself, and if it is, they swap places. If the root is reached, or the parent is already less than itself, the procedure terminates.

Erasing the top element is done by swapping the top element with the last entry of the array, and a procedure called Down-Heapify is called. It recursively check whether the children of the new element is less than itself, and if so, it swaps place with the lowest of the two.

Binary Heap is used for the implementation of Priority Queue, and other algorithms that require keeping track of the lowest/highest element in a collection.

Here is a simple implementation of Min-Heap which places the minimum element on top of the heap throughout.


//Implementation of Min-Binary Heap
//Support: Insert, Pop, Top
#include <iostream>
#include <cstdio>
#include <string>
using namespace std;

class MinHeap
{
 private:
  typedef int pos;
  static const pos NIL = 0;
  int max_size; //max size of array
  pos last;  //position of last element
  int *minheap; //pointer to an array
  //MinHeap Construction Algorithm
  void upheap(pos x);
  void downheap(pos x);
  pos left_child(pos x);
  pos right_child(pos x);
  pos parent(pos x);
  void heapswap(pos x, pos y);
 public:
  MinHeap(int sz): max_size(sz), minheap(new int[sz+1]), last(NIL){}
  
  ~MinHeap()
  {
   delete[] minheap;
  }

  void push(int m);
  void pop();
  int top();
  bool empty();
  void to_string();
};

MinHeap::pos MinHeap::left_child(pos x)
{
 if(2*x > last)
  return NIL;

 return 2*x;
}

MinHeap::pos MinHeap::right_child(pos x)
{
 if(2*x + 1 > last)
  return NIL;

 return 2*x + 1;
}

MinHeap::pos MinHeap::parent(pos x)
{
 if(x == 1)
  return NIL;

 return x/2;
}

void MinHeap::heapswap(pos x, pos y)
{
 int temp = minheap[x];
 minheap[x] = minheap[y];
 minheap[y] = temp;
}

void MinHeap::upheap(pos x)
{
 pos par = parent(x);
 while (par != NIL && minheap[par] > minheap[x])
 {
  heapswap(x,par);
  x = parent(x);
 }
}

void MinHeap::downheap(pos x)
{
 pos next = x;
 pos left = left_child(x);
 pos right = right_child(x);
 if(left != NIL && minheap[left] < minheap[next])
  next = left;
 if(right != NIL && minheap[right] < minheap[next])
  next = right;
 if(next == x) return;
 heapswap(x, next);
 downheap(next);
}

//Function definitions
void MinHeap::push(int m)
{
 if(last+1 > max_size)
 {
  printf("MinHeap is already full, insertion is aborted\n");
  return;
 }
 ++last;
 minheap[last] = m;
 upheap(last);
}

int MinHeap::top()
{
 if(last < 1)
 {
  printf("MinHeap is empty\n");
  return NIL;
 }
 return minheap[1];
}

void MinHeap::pop()
{
 if(empty())
  return;
 minheap[1] = minheap[last];
 --last;
 downheap(1);
}

void MinHeap::to_string()
{
 for(pos it = 1; it <= last; ++it)
 {
  printf("%d ", minheap[it]);
 }
 printf("\n");
}

bool MinHeap::empty()
{
 if(last == NIL)
  return true;

 return false;
}

int main()
{
 /**
  * TEST BODY
  * 3 Commands: Push, Pop, and Top
  */

 printf(":::::::::::MinHeap:::::::::::\n 2014 - Prajogo Tio \n Simple Implementation of MinHeap\n\n");
 printf("Enter the intended size of the MinHeap: ");
 int max_sz;
 cin >> max_sz;
 MinHeap myHeap(max_sz);
 printf("Command:\n");
 printf("1. push <number>\n2. top\n3. pop\n4. quit\n\n");
 string cmd;
 int num;
 while(printf("Key in your command: \n"), cin >> cmd)
 {
  if( cmd == "quit")
  {
   printf("Program is terminating. See you again.\n\n");
   break;
  }

  if( cmd == "push")
  {
   cin >> num;
   myHeap.push(num);
   cout << num << " has been pushed to the heap\n\n";
  }
  else if( cmd == "pop")
  {
   myHeap.pop();
   cout << "Top element has been removed\n\n";
  }
  else if( cmd == "top")
  {
   if(myHeap.empty()) 
   {
    cout << "MinHeap is empty\n\n";
   } else {
    cout << "Top element: " << myHeap.top() << endl << endl;
   }
  }

 }

 return 0;
}

Thursday, July 10, 2014

a bit of data structure: Fenwick Tree (Binary Indexed Tree)

Problem Statement:
Given a set of values \(i \in \{ 1, \ldots, n \} \) and set of frequencies \(f_i\):
1. find the cumulative frequency from i to j
2. support updating frequencies

Data Structure: Fenwick Tree (also known as Binary Indexed Tree)

Construction:
Every number can be represented the sum of powers of 2, which is the basis of the binary representation of numbers. Define LSB[i] as the least significant bit of i (the last bit 1 in the binary representation of i). For example:
1. LSB[3] = LSB[11] = 0
2. LSB[14] = LSB[1110] = 1
3. LSB[52] = LSB[110100] = 2

We define FTree[i] as the sum of frequencies from \( i - 2^{\text{LSB[i]}} + 1, \ldots, i\). This allows an iterative representation of Cumulative Frequency from 1 to i CF[i] in terms of FTree, for example:

1. CF[10001101] = FTree[10001100] + FTree[10001000] + FTree[10000000]
2. CF[1101] = FTree[1100] + FTree[1000]

As such, CF[i] can be found in \(O(\log{N})\) time, where \(N\) is the length of the binary representation of i.

In case we update the entry in FTree[i] by a value val, we iteratively add val to FTree[i + \(2^\text{LSB[i]}\)], hence updating all responsibility chain in FTree in \(O(\log{N})\) time as well.

The pseudocode for construction of Fenwick Tree:

LSB(i):
1. Return (i & ~i)

Build-FTree(array of frequency F):
1. let FTree[] be an array, FTree[0] = 0.
2. For each i from 1 to n = F.length
    2.1 Set FTree[i] = sum of F[k] for k from (i - LSB(i) + 1) to i
3. Return FTree

Find-CF(FTree, i):
1. Set cumfreq = 0
2. While i != 0:
    2.1 Set cumfreq = cumfreqFTree[i]
    2.2 Set i = i - LSB(i)
3. Retrun cumfreq

Update-FTree(FTree, i, val):
1. While i not greater than n
    1.1 Set FTree[i] = FTree[i] + val
    1.2 Set i = i + LSB(i)

The implementation is very sweet and simple :D



vector<int> ftree(11,0);
int maxval;

int lsb(int i) 
{
 return i & (-i);
}

void update(int i, int val)
{
 while(i <= maxval)
 {
  ftree[i] += val;
  i = i + lsb(i);
 }
}

int cumfreq(int i)
{
 int tot = 0;
 while( i > 0)
 {
  tot += ftree[i];
  i = i - lsb(i);
 }
 return tot;
}

Wednesday, July 9, 2014

a bit of string matching: Suffix Automaton

I will not have the time to expand the derivation of this data structure yet, but here I will present the algorithm as written in a paper by Blumer et. al. on smallest suffix automaton that recognizes all subwords.

Problem Statement:
Given a text (fixed), build a deterministic finite automaton (DFA) which accepting states are substrings in the given text.

Algorithm: Construction of the DFA (called Suffix Automaton / Directed acyclic word graphs (DAWG) )

Build-DAWG (string s):
1. Create a node u representing the empty state
2. Set root as u, sink points to u
3. For i from 0 to length of s:
    3.1 Create a new node called newsink
    3.2 Create a primary edge from sink to newsink with label s[i]
    3.3 Set currentstate points to sink, and suffixstate as null
    3.4 While currentstate is not root and suffixstate is null:
           3.4.1 Set currentstate = currentstate.suff
           3.4.2 If currentstate has a primary edge labeled s[i]:
                    3.4.2.1 Set suffixstate to the state that edge points to
           3.4.3 Else if currentstate has a secondary edge labeled s[i]:
                    3.4.3.1 Let child be the state that edge points to
                    3.4.3.2 Set newstate = Split( currentstate, child )
                    3.4.3.3 Set suffixstate = newstate
           3.4.4 Else:
                    3.4.4.1 Create a secondary edge from currentstate to newsink with label s[i]
    3.5 If suffixstate is still null:
           3.5.1 Set suffixstate = root
    3.6 Set newsink.suff = suffixstate
    3.7 Set sink = newsink

Split (parent, child):
1. Create a new node newstate
2. Set that secondary edge to a primary edge from parent to newstate
3. For all outgoing primary and secondary edge from child:
    3.1 Create a similar outgoing secondary edge from newstate pointing to the same state
4. Set newstate.suff = child.suff
5. Reset child.suff = newstate
6. Set currentstate = parent
7. While currentstate is not root:
    7.1 Set currentstate = currentstate.suff
    7.2 If currentstate has a secondary edge to child:
          7.2.1 Reset the secondary edge to point to newstate
    7.3 Else:
          7.3.1 Break from the loop
8. Return newstate


Here is an implementation (a very inefficient one :D haha) in C++ :



#include <cstdio>
#include <iostream>
#include <vector>
#include <string>
#include <algorithm>
using namespace std;
//Implementation of Suffix Automaton
//First Try, Must be very INEFFICIENT
//Using pointers maybe

struct state;
struct edge;
state* split(state* parent, state* child, vector<edge*>::iterator it);

struct edge
{
 state* child;
 char label;
};

struct state
{
 //information regarding the state
 int index;
 int pos;
 //suffix link
 state* suff;

 //prim edge
 vector<edge*> prim_edge;

 //sec edge
 vector<edge*> sec_edge;
};

state *root, *sink;
int idx = 1;

void init(string str)
{
 //create the root state
 root = new state;
 root->suff = NULL;
 root->prim_edge.clear();
 root->sec_edge.clear();
 root->index = 0;
 root->pos = -1;

 //initialize sink
 sink = root;

 int len = str.size();

 for(int i=0;i<len;++i)
 {
  //edge label
  char a = str[i];
  
  state* newsink = new state;
  newsink->index = idx++;
  newsink->pos = i;
  newsink->suff = NULL;

  //initialize new edge to the new sink
  edge* to_newsink = new edge;
  to_newsink->child = newsink;
  to_newsink->label = a;

  sink->prim_edge.push_back(to_newsink);

  state* currstate = sink;

  while( currstate->suff != NULL && newsink->suff == NULL) 
  {
   currstate = currstate->suff;

   bool finished = false;

   //if exist prim_edge of currstate with label == a
   for(vector<edge*>::iterator it = currstate->prim_edge.begin(); it != currstate->prim_edge.end(); ++it)
   {
    if( (*it)->label == a )
    {
     //update suff link of newsink to currstate
     newsink->suff = (*it)->child;
     finished = true;
     break;
    }
   }

   if( finished ) break;

   //if exist secondary edge
   for(vector<edge*>::iterator it = currstate->sec_edge.begin(); it != currstate->sec_edge.end(); ++it )
   {
    if( (*it)->label == a )
    { 
     //split the state
     state* child = (*it)->child;

     state* clone = split(currstate, child, it);

     //update suffix link of newsink to clone
     newsink->suff = clone;
     finished = true;
     break;
    }
   }

   if( finished ) break;

   //Else, add secondary edge to newsink and continue
   currstate->sec_edge.push_back(to_newsink);
  }

  //if newsink still does not have suffix link
  if (newsink->suff == NULL)
  {
   //point to source
   newsink->suff = root;
  }

  //update sink
  sink = newsink;
 }

}


state* split(state* parent, state* child, vector<edge*>::iterator iter)
{
 state* clone = new state;
 clone->index = idx++;
 clone->pos = child->pos;

 char label = (*iter)->label;
 //copy all outgoing edge

 for(vector<edge*>::iterator it = child->prim_edge.begin(); it != child->prim_edge.end();++it)
 {
  edge* e = new edge;
  e->label = (*it)->label;
  e->child = (*it)->child;
  clone->sec_edge.push_back(e);
 }

 for(vector<edge*>::iterator it = child->sec_edge.begin(); it != child->sec_edge.end();++it)
 {
  edge* e = new edge;
  e->label = (*it)->label;
  e->child = (*it)->child;
  clone->sec_edge.push_back(e);
 }

 //update suffix link of clone
 clone->suff = child->suff;

 //update suffix link of child
 child->suff = clone;

 //update parent's sec edge to child to prim edge to clone
 parent->sec_edge.erase(iter);

 edge* to_clone = new edge;
 to_clone->label = label;
 to_clone->child = clone;

 parent->prim_edge.push_back(to_clone);

 //update all edge to child to point to clone
 state* currstate = parent;

 while( currstate->suff != NULL )
 {
  currstate = currstate->suff;

  bool update = false;
  for(vector<edge*>::iterator it = currstate->sec_edge.begin(); it != currstate->sec_edge.end(); ++it)
  {
   if( (*it)->child == child )
   {
    (*it)->child = clone;
    update = true;
    break;
   }
  }

  if( !update ) break;
 }

 return clone;
}

void print(state* s)
{
 printf("state with index: %d\n", s->index);

 vector<edge*>::iterator it;

 for(it = s->prim_edge.begin(); it != s->prim_edge.end(); ++it)
 {
  printf( "==%c (%d)" , (*it)->label, (*it)->child->index);
  printf("\n");
 }

 for(it = s->sec_edge.begin(); it != s->sec_edge.end(); ++it)
 {
  printf( "--%c (%d)" , (*it)->label, (*it)->child->index);
  printf("\n");
 }

 for(it = s->prim_edge.begin(); it != s->prim_edge.end(); ++it)
 {
  print((*it)->child);
 }

 for(it = s->sec_edge.begin(); it != s->sec_edge.end(); ++it)
 {
  print((*it)->child);
 }
}

int search(string str)
{
 int len = str.size();
 state* currstate = root;

 for(int i=0;i<len;++i)
 {
  vector<edge*>::iterator it;
  bool found = false;
  for(it = currstate->prim_edge.begin(); it!= currstate->prim_edge.end(); ++it)
  {
   if( (*it)->label == str[i] )
   {
    currstate = (*it)->child;
    found = true;
    break;
   }
  }

  if( found ) continue;

  for(it = currstate->sec_edge.begin(); it!= currstate->sec_edge.end(); ++it)
  {
   if( (*it)->label == str[i] )
   {
    currstate = (*it)->child;
    found = true;
    break;
   }
  }

  if (!found)
   return -1;
 }

 return currstate->pos;
}

int main()
{
 printf("Welcome to SUFFIX AUTOMATON SEARCH\n");
 printf("Enter your string/dictionary\n");

 string str;
 getline(cin,str);

 init(str);

 string src;
 while (1)
 {
  printf("Enter your search term:\n");

  getline(cin,src);

  int pos = search(src);

  if(pos < 0) 
  {
   printf("The term is not found\n");
  }
  else
  {
   printf("The term is found at end-pos: %d\n", pos);
  }
 }
 return 0;
}

Sunday, June 22, 2014

a bit of data structure: Disjoint Set Find-Union

Problem Statement:
How do we represent disjoint sets in a manner such that it supports these two functions efficiently:
1. union(A,B): given disjoint sets \(A\) and \(B\), return \(C = A\cup B\)
2. same(u,v): given 2 element \(u\) and \(v\), check whether they are in the same set.

To efficiently support a data structure of disjoint sets, we need a clever construct to represent the disjoint sets. How? Here we'll discuss the standard way of doing it.

Let's say we have \(N\) elements indexed as \(1,2,3,\ldots , N\). Initially, all elements are disjoint to each other. We have a notion of parent, expressed as an array parent[\(1 \ldots N] such that parent[i] points to the parent of element i. At first, every element is a parent to itself since they are disjoint. We call a procedure to initialize this condition as init():

init():
input: none
output: initialize the disjoint set data structure


for i in 1 to N:
   parent[i] = i

The union of 2 elements will result in parent of the first element pointing to the second element. As this procedure proceeds, a forest of trees will result, where each tree is a representation of a set. Now imagine if we want to check if 2 element is indeed inside the same tree, that means we need to traverse up the parent chain of each element until we find an element which parent points to itself (hence the root of the tree) and finally compare whether the two roots are the same. This traversal may take \(O(\log{N})\) each time (the height of the tree). Hence we need to compress this tree such that subsequent check of the root of a tree will result in \(O(1)\) average. Call this procedure find(u) as described below:

find(u):
input: an element
output: the root of the disjoint set it is in


if u != parent[u]:
    parent[u] = find(parent[u])
return parent[u]

Here we check whether \(u\) is root, if so then we return \(u\). Otherwise, we need to update the parent of u to point to the root of the tree, and return it. Hence it updates the parent-child relationship in the tree such that the tree is compressed to a flatter tree.

Finally we can describe the two main functions above in an efficient manner:

union(u,v):
input: 2 elements u,v
output: union of the set containing u and the set containing v

parent[find(u)] = find(v)




same(u,v):
input: 2 elements u,v
output: is u and v in the same set?


return find(u) == find(v)


Here is a minimalist implementation of the data structure:
vector<int> p(100);
vector<int> r(100);
int init() { for (int i=0;i<100;++i) p[i]=i, r[i] = 0; }
int find_set(int i) { return (p[i] == i) ? i : (p[i] = find_set(p[i])); }
bool same_set(int i, int j) { return find_set(i) == find_set(j); }
void union_set(int i, int j) {
 if ( same_set(i,j) ) return;
 int x = find_set(i);
 int y = find_set(j);
 if(r[x] > r[y]) {
  p[y] = x;
 } else {
  p[x] = y;
  if (r[x] == r[y]) ++r[y];
 }
}

Thursday, May 15, 2014

a bit of linked list

As we have discussed, arrays are fixed in size as memories are allocated at compile time, which leads to possible wastage of memories due to excessive allocation. Hence, there is a desire to create a structure that can be flexible in terms of memory and yet provide the same functionality as arrays (which gives sequential representation of data). Such structures indeed exist, and one of the simplest implementation is called linked list.

As the name implies, linked list is a set of element linked to each other by a special relationship. Each element in a linked list has a value (called key) and a pointer to the element next to itself (called next). Hence, a linked list is constructed by artificially chaining an element to the next element until all elements are linked up with a parent-child relationship.


For example, if we want to represent {3,40,22,1} as a linked list, we will have 4 elements:

A {key=3,next=B}, B {key=40,next=C}, C {key=22,next=D} and D {key=1,next=NONE}. Hence we have a linked list of A->B->C->D. D is the end of the linked list, hence it does not point to any element, so we set its next to NONE.

The benefit of doing this is that when we want to insert element to the list, e.g. inserting E with key=90 after the last entry D, all we need to do is updating the pointer next of the last entry D to E, e.g. D {key=1,next=E}. Hence the new linked list is A->B->C->D->E.


On the other hand, if we are to add E in between B and C, we only need 2 changes:

1. instead of pointing to C, B will point to E
2. E should point to C.
e.g., set B to {key=40,next=E}, and set E {key=90,next=C}, hence we have a linked list of A->B->E->C->D.

Linked list also support deletion of an element, also by manipulating the pointers as we did in insertion. A simple implementation of deletion and insertion will be demonstrated in the codes provided below.

/* Linked List Simple Implementation */
struct node{
  int key;
  node* next;
};

class llist{
public:
  node* head; //starting point
  node* tail; //ending signal

  llist(){
    head=new node();tail=new node();
    head->next=tail; //initially, head points to tail(empty linked list)
    tail->next=tail; //tail points to itself
  }

  void del_next(node* n){
    node* t=n->next;
    n->next=t->next;
    delete t;  //freeing up memory
  }

  node* ins_next(node* n,int value){
    node* t=new node();
    t->key=value;
    t->next=n->next;
    return n->next=t; // returning "iterator" to last entry
  }
};

Notice that in its implementation, we need to provide 2 dummy elements, called head and tail, which signals the start and the end of a linked list. head will point to the first element of the linked list, while tail will be pointed by the last element of the linked list, with tail pointing to itself. Hence it will look like this: head->A->B->C->D->E->tail->(self-pointing).

You can extend this data structure to support more functionalities such as retrieving a pointer to a particular key, accessing an element through "index", sorting, and many more if you are interested.

Another interesting implementation of linked list is through array manipulation. While it may defeat the purpose of providing a structure with dynamic memory allocation, it allows us to understand the inner processes of linking elements. The implementation is very simple without the use of pointers (explicitly). We maintain 2 sets of arrays, one called key and another called next. Here is an example of the implementation:


#define MS 1000
int key[MS];
int next[MS];
int size,head,tail;

void link_init(){
  head=0;tail=1;size=2;
  next[head]=tail;
  next[tail]=tail;
}

void del_next(int node){
  next[node]=next[next[node]];
}

int ins_next(int node, int value){
  key[size]=value;
  next[size]=next[node];
  next[node]=size;
  return size++;

}

In this implementation, when we delete an element, it is not the element that is deleted, but rather its access that is discarded. This results in a very fast deletion and insertion operation on linked list. However, poor implementation of linked list may result in patches of memories becoming inaccessible and unavailable for future uses.

Wednesday, May 14, 2014

a bit of array

Algorithm attempts to solve problems through manipulation of data structures, hence data structure and algorithm go hand in hand. They are like Batman and Robin, only less gay (just saying).

One of the most fundamental data structure in any programming platform is array. When we initialize an array, especially in case of static array, we are allocating a fixed amount of blocks of memory to a certain data type, which are accessible by indicating the specific index of the array. The strength of an array structure is that by knowing the index, access operation on arrays takes O(1) time. However, since the size of arrays have to be known before use, there are chances of excessive unused memory allocated which results in lower performance.

Here is one interesting manipulation of array. Consider a problem of printing all prime numbers less than 50,000. A prime number is defined as a number which can only be divided by itself and 1. (Interestingly, 1 is not a prime ladies and gentlemen!)

A simple approach to this problem can be described as follows. For any number between 0 to 50,000, test whether it is prime, then print it. If a number is divisible by any number less than itself (excluding 1), than it is not a prime. Otherwise, we have a prime number.

Here is an interesting implementation of this idea, leveraging on the fact that if a number is a product of two other numbers (excluding 1), then it is not a prime.

#include <stdio.h>
#define N 10000

int main(){
  bool a[N]; //for number 1 to 49,999,a[i]=1 if prime, 0 otherwise
  for(int i=2;i<N;++i)a[i]=1; //initialize a[i]
  for(int i=2;i<=N/2;++i){
    for(int j=2;j<=N/i;++j){ //leveraging on symmetry of i*j
      a[i*j]=0; //not prime!
    }
  printf("Primes less than 10,000 are: ");
  for(int i=2;i<=N;++i){
    if(a[i])printf("%d ",a[i]);
  }
  printf("\n");
  return 0;
}

Leveraging on symmetry of i*j allows us to only check the value of i from 2 to N/2. j must be less than N/i since we are only interested in values of  i*j that are less than N. 

Saturday, May 10, 2014

What is algorithm? Sounds so cheem!

Indeed, the study of algorithm can get even more complex and deep than what the already cheem word 'algorithm' suggests. Personally, it is the study of the art of solving problems by manipulating data and structures to come up with a fast and reliable solution.

Before we dwell too deeply on defining the precise meaning of algorithm and data structures, let's consider the following daily "problem".

Suppose that we have to search for an item from a pile of random stuff. How would we find the item as quickly as possible? Of course one of the most straight forward way is to go through the items in the pile one by one, from the top-most item to the bottom, until we find what we are looking for. If there are 10 items, we can find our item real quick. If we have 20 items, maybe it gets a bit longer. But what if we have 10000 items? Or a million? Our method of scanning the items one by one will take a long time before we get to find the item. Can we do better?

This is the kind of problem that you will often face when dealings with problems and trying to come up with the correct and fast enough algorithm. In this example, there is really not much we can do, given that the stuff are randomly distributed in the pile. The best solution (for now) is really scanning the whole items, and it takes linear time to find our item. Linear time means that the time taken for the algorithm to terminate is proportional to the size of the problem.

However, if we are told that the items are sorted by their weight, e.g. from the lightest to the heaviest, there is a faster solution that beats our linear time method! Let's see the following demonstration.

Suppose the weight of our item is 500kg (wonder what it is huh) and it is inside a pile of items which is sorted by weight:
1 5 40 45 80 90 120 500 670 700 800

If we try to search for the 500kg item linearly from the first item (1kg), we will find our item after 8 steps:
1 5 40 45 80 90 120 500 670 700 800 // nah..
1 5 40 45 80 90 120 500 670 700 800 // not this one..
1 5 40 45 80 90 120 500 670 700 800 // still no..
1 5 40 45 80 90 120 500 670 700 800 // uhh..
1 5 40 45 80 90 120 500 670 700 800 // persevere..
1 5 40 45 80 90 120 500 670 700 800 // cmon..
1 5 40 45 80 90 120 500 670 700 800 // oh god.. why..
1 5 40 45 80 90 120 500 670 700 800 // FOUND IT!
Total: 8 steps

Now let's revise our method by utilising the information that we have. Consider this: If we choose an item from the pile somewhere in the middle, and it turns out to be lighter than 500kg, we can ignore all items that come before it (because they will be lighter that 500kg too!) and start searching on the other half. We effectively eliminate half of our search space in one step! We can repeat the process of choosing the middle item and eliminate one half of the piles for every steps: (the blue colored numbers are the current search space at each step)
1 5 40 45 80 90 120 500 670 700 800 //90 < 500, so 500 must be right
1 5 40 45 80 90 120 500 670 700 800 //670 > 500, so 500 must be on the left
1 5 40 45 80 90 120 500 670 700 800 //120 < 500, 500 must be on the right
1 5 40 45 80 90 120 500 670 700 800 //FOUND IT!
Total: 4 steps

This is called binary search, and this algorithm takes logarithmic time, since the expected number of steps needed is proportionate to the logarithmic curve, which grows much slower than a linear curve. Binary search is an example of the application of a problem solving paradigm called "Divide and Conquer", which divides up its problem space into smaller pieces to work on, one of the most powerful recurring ideas in algorithm study.

Here we can see a huge improvement from 8 steps to 4 steps. I must admit that if we are searching for items of different weight, for example 1kg, the linear method will return faster for this case. However, when the size of the problem becomes really big, e.g 1 million, and the item that we need to find is randomized, the number of steps needed for the linear method may reach 1 million, while for binary search you need at most 20 steps to find the item!

We can see that studying algorithm can give us huge performance boost and impact on the effectiveness of our solutions. Hence, its importance goes without saying, in which its applications can range from remotely useful (such as... almost nothing) to banking, scientific research, DNA sequencing, security, airplanes scheduling, to anything that you can think of!

This simple illustration demonstrates a few common ideas on algorithm and data structure, and there are many more to look forward to.