Category: Analytics

Posted on

CAP Theorem and Distributed System

CAP Theorem is also known as Brewer’s Theorem.

CAP refers to Consistency, Availability and Partition tolerance.

  • Consistency – All nodes in a cluster have same data at any particular time.
  • Availability – It is the end result of request i.e. success or failure.
  • Partition tolerance – As per this feature, the system continues to operate despite any hardware or network failures.

According to CAP Theorem any distributed system can exhibit only 2 out of the above 3 features.

Most distributed systems experience network failures, hence partition tolerance has to be met by distributed systems.

For the remaining one condition to be met:

Databases designed based on ACID properties choose consistency over availability.

Note: Consistency here not to be confused with the one in ACID.

On the other hand, databases designed based on BASE properties choose availability over consistency.

HBase

In case of HBase, consistency and partition tolerance are met. But HBase does not offer availability. HBase is a NoSQL database.

Posted on

Google Analytics for Jan-Feb-March 2018(Q4 )

The statistics for MOHANMA.COM for Jan-Feb-March(starting Jan 26th to 31st March) are as shown above.

There have been a total of 737 sessions with 1673 pageviews.

The Bounce Rate of the website is 66.76%, which is pretty good as lower the bounce rate better the performance of website.

New visitors account for 76.5% of total visitors and returning visitors account for 23.5%.

India accounts for the highest traffic, with Karnataka being the leading region to in-flowing traffic with 576 sessions so far.

Among devices, mobile phones lead the traffic generated followed by desktop and lastly by tablets.

Apple iphones are the leading mobile phones used to access the website followed by Xiaomi and Motorola phones.

Thanks to everyone who have been visiting the website.

Love,

Mohan M A

Posted on

Understanding Apache Spark

In my last blog post I had discussed about data, now let us understand a modern tool to process huge datasets(BigData) so as to extract insights from data.

Apache Spark – A fast and general engine for large-scale data processing. Spark is a more sophisticated version of data processing engine compared to engines using MapReduce model.

One of the key feature with Apache Spark is Resilient distributed datasets(RDD’s). These are data structures available in Spark.

Spark can run on Hadoop YARN cluster. The biggest advantage to keep large datasets in memory adds to the capability of Spark over MapReduce.

Applications types that find Spark’s processing model helpful are:

  1. Iterative algorithms
  2. Interactive analysis

Other areas which make Spark more adoptable are:

Spark DAG(Directed Acyclic Graph) – This component of the engine helps to convert variable number of operations into a single job.

User Experience – Spark makes user experience smooth by having a plethora of API’s to perform data processing tasks.

Spark has API’s in these languages: Scala, Java, Python and R.

Spark programming comprising of the Spark shell(also known as Spark CLI or Spark REPL) makes it simple to work on datasets. REPL stands for read-eval-print loop.

Spark on the other hand provides modules for:

  1. Machine learning(MLib) – Provides a framework for distributed machine learning.
  2. Graph processing(Graphx) – Provides a framework for distributed graph processing.
  3. Stream processing(Spark Streaming) – Helpful for streaming(real-time) analytics. Data ingestion takes place in mini-batches and RDD transformations are performed upon these mini-batches.
  4. SQL(Spark SQL) – Provides data abstraction known as SchemaRDD which supports structured and semi-structured data.

These components operate on Spark core. Spark core provides platform for in memory computing and referencing datasets in external storage systems.

Companies that are using Apache Spark – Google, Facebook, Twitter, Amazon, Oracle, et al.

Spark services are provided on notable cloud platforms such as Google Cloud Platform(GCP), Amazon Web Services(AWS) and Microsoft Azure.

Source: Apache Spark

 

Posted on

Data: The new “OIL” of Information Era

Dataoil

Anything and everything that can be produced, which we can quantify is referred as data. In simple words granular information is data. Automobiles and machinery have been running on oil extracted from earth. In the Internet age devices, machines and all mundane activities shall be driven by data.

All data that exists can be classified into three forms:

  1. Structured Data
  2. Semi-structured Data
  3. Unstructured Data

Structured Data – Structured data is a standardized format for providing information. Examples: Sensor data, machine generated data, etc.

Semi-structured Data – Semi structured Data does not exist in standard form but can be used to derive structured form with little effort. Examples: JSON, XML, etc.

Unstructured Data – Unstructured Data is any data that is not organized but may contain data which can be extracted. Examples: Social media data, Human languages, etc.

Most global tech giants operate from data generated by their users. Google, Amazon, Facebook, Uber, et al. come under the same umbrella. The insights derived from structured and semi structured data can help us in decision making. The magnitude and scale at which these companies generate data is astounding.

Databases play a very important role in storing data. But traditional databases are no longer a choice to store data in today’s fast moving world. New age file systems and infrastructure have started operating to cater the demands of ever expanding Internet space.

In the human world, the voice, speech, text, walking speed, everything can be classified as unstructured data, since we can derive a lot of insights from them. A mobile device per individual is pretty much sufficient to analyse the behavior of a sizable population in a region.

Data collected from a population for a relatively considerable time can be used to derive patterns about the population. Hence, data is the driving force which will fuel innovation and economy from here.

Posted on

Google Analytics for MOHANMA.COM : Launch(26th Jan) to 2nd March

Firstly, thanks to everyone who have visited mohanma.com. The above image shows the global footprint of mohanma.com.

The total number of sessions so far has been 563. India leads in web traffic with 493 sessions. Followed by United States of America, Australia and United Kingdom. Traffic has been flowing in from different parts of Europe, Canada, Peru, China, Japan, etc as well.

To be frank, I have been surprised with the kind of response that mohanma.com has been receiving from day 1. Thanks to all the readers, well-wishers who have been pouring in their feedback via WhatsApp, Facebook and the comments section on website.

Love,

Mohan M A