NoSQL Opensource Database: Architecture, Tools, Algorithm, Design, Interchange and Distributed architecture

Created on 2020-08-21 11:15

Published on 2020-08-22 14:05

Cloud Platforms (AWS/Azure/Google/Amazon) offer a range of non-hierarchical, non-RDBMS databases services (not necessarily No-SQL): Wide-Column, Document, Time Series, Graph, and Analytics. Powered by opensource, and launched on cloud platforms, these have seen a growth of 52% in 2019, with projected revenues exceeding 25% by 2022.

The next-gen cloud-native application architectures and the hunger for data variety for Analytics/AI will drive adoption of these data management services rapidly if one understands the underlying algorithms, architecture and design principles. Here is a compendium of the same:

Data Interchanges - Added 7th Sept 2020

Traditional and bulky interchange - Services/WS/RPC/streams/messages/events/Actor-based for data pipelines: JSON, XML, CSV. May not be a best choice for data intensive apps working over n/w. Fast-forward and consider these options:

1. Protobuf: If you are thinking RPC as interchange mechanisms and potentially gRPC
2. Thrift: Social streaming with facebook
3. Apache Avro: Opensource for Hadoop and large data, order agnostice (writer vs reader schema)

The challenges in all the interchange formats is to take care of implementing the forward and backward incompatibilities.

Architecture: Replication, Partitions, Transaction - Added 9th Sept 2020

Distributed data bases will be increasingly common as we head to cloud. Its important to understand and account for the challenges while devising applications/algorithms:

1. Replication challenges and solution:

2. Partitioning/Sharding: Horizontal scaling enabled by Key based, hash-based, primary/secondary index based, etc. The challenges are primary around need for rebalancing and routing, they are solved well by many large databases efficiently.

3. Transaction: Applications have to account for inconsistent implementation of Atomicity, Consistency, Isolation and Durability properties. This leads to Dirty reads, Dirty writes, lost updates, read/write skews (timing anomaly). The hard problem are solved by offering solutions that applications need to implement: Snapshot isolation with "BEGIN TRANSACTION", materializing conflicts with "SELECT FOR UPDATE", and Serialized isolation methods: poor performing Serial execution (REDIS), poor scaling 2-Phase locking (MYSQL using predicate, range-index, shared & exclusive locks), the latest Serializable snapshot isolation (PostGres) where conflicting writes are isolated.

IT Outsourcing Service Provider: Understanding the stakeholder dynamics: Order-Giver Vs Order Taker

In the earlier blog we saw different stakeholders involved in IT services organization. We broadly classifed them as Order takers: Mostly the IT service providers (offshore) with any contractors/partners at offshore and Order-Givers: Client Onshore, Client Captive centers offshore, and the IT service provider (Onshore). We saw the complexity continum in engagement depending on number of stakeholders in play. In this blog we look at the interaction dynamics between the different stakeholders:

The dynamics can best be understood along the following dimensions:
Teams: This details the team members involved
Interactions: This detailes the Software lifecycle phases when the interactions is pronunced
Projects: Details the projects where the interaction plays out.

It is important to understand not all stakehoders will be present in all engagements and also not all interactions may play out. One can see different combinations of stakeholders and interaction depending on how the organization and information flows are setup. In this blog we will look at the complete picture to understand the dynamics well.

The diagram below provides a pictorial representation of the stakeholders and the interactions dynamics:

We will look at each interaction pairs and the dynamics that are not so evident in the above diagram:

Client Onshore + IT Service Provider (Onshore): The cosy relationships whereby the client onshore stakeholder interacts with Onshore vendors. This interaction is often relationship based. Most often they have shared interest (retaining control) and drivers (read saving their jobs). They often pair up to protect their joint interests. If things go wrong it is always easy to blame the Offshore entity first and then followed by Order takers.
Client Captive centers Offshore + IT Service Provider (Offshore): The relationship is driven primariy in terms of deliverables, metrics and reports. The order giver and order taker relationship is noticebale and pronounced.  They often are at logger heads and built on mistrust. The mistrust stems from fact that the captive center leads despise the fact that the IT service provider at offshore get most of instructions and details from their Onshore counterparts. Often they captive center clients control information and draw their power and position on additional information they have. In times of trouble they easily blame the all woes to the IT Service provider offshore team. They like to pair themselves with the Client Onshore but often client onshore dont see them adding value and believe them to be necessary evil to deal with.
Client Captive centers Offshore + IT Service Provider (Onshore): Mostly the interactions are very limited, but whenever this interaction is strong the IT service provider often suffers due to miscommunication and vested interests. Mostly the client captive center  team hates them but engages them to down-play or comlain against IT service provider offshore team. Sometimes we see the IT Service provider onshore engage the captive center team just to draw a wedge and feed their own interests and drivers (jobs).
Client Onshore + IT Service Provider (Offshore): This relationship is seldom seen, but wherever this exits the interactions underplay the captice venter and IT Service provide onshore. Usually we see this interaction well established when the onshore component is non-existant and client prefers 100% offshore execution. This is best and most cost optimal solution in most cases and often not preferred by large Tier-1 players: IBM, Cap, Accenture, etc.
Client Captive centers Offshore + Client Onshore (Onshore): This interaction exists whenever there is a captive center. The interactions are mostly driven due to business reasons and exists for business purposes. The captive center role becomes relevant mostly when there are multi-vendor/partners on the deal. The relationship most often is driven to get work done from multiple vendors. This is not well detailed here, but they share common interest of saving their jobs and roles under the guise of project management and control.
IT Service Provider (Onshore) + IT Service Provider (Offshore): Normal relationship that exists and they pair up to protect the interests of the overall contract with IT Service provider. The relationship becomes complex in light of other interactions in the game.

But is there a right interaction and right stakeholder model that one can choose depending on the requirements? What is the optimal stakeholder and interaction model? We will discuss and cover this in the upcoming blogs.

Working with IT Services clients in offshore-onshore-captive centres: The Master: order-givers and Slave: order-takers

In the IT Services world the new manifestation of slavery seems to be client-service provider relationships. But who plays the master: order-giver is often a contention, but never a doubt on who is a slave:order-taker…

It seems that generally the offshore service provider team is the slave: order taker, but the masters: order-giver can be many. Why is the IT offshore service provider takes that place is due to fact that this is where the real work gets done. Then there are different masters: order-givers as they mistakenly believe they have got Gods gift of mankind of intellect, budgets, and/or I-Know-all: ego that drives them to always give orders. Let us look at the Order giver: players/teams in the engagement to understand this better:

Client at Onshore: They have the budgets to pay for the services and the budget themselves are provided by business. This often gives them the benefit of always playing the masters: Order giver role.

Client at offshore: This entity is relevant usually when client has captive centre near offshore service provider. They usually don’t have budgets and/or arguably the intellect/skills too, but get the deemed merit to give orders without knowing what it means.. Examples of such captive centres in high-tech: Cisco, EMC, Vmware, etc.

Service Provider at Onshore: They have proximity to budget provider and often work with the clients business stakeholders and get a unique position to know all, but arguably the intellect to know how to get things done. Sometimes this onshore service providers are either body shopped by pure play vendors in India: TCS, Wipro, Infosys, etc. and/or hired in local markets to serve their political/government constituencies. Usually, these players are ment to be order-takers but often not doers and therefore love to become order-givers.

The order-takers does not need so much description as these are the doers and make things happen. They have the intellect/skills that make sure the value for the budgets gets delivered. Often there is a misconception of their intellect: lack of intellect to understand business requirements, do not have discipline and structure to execute the orders and/or ability to deliver value for the provisioned budget. These are the Service Provider at Offshore.

The number and location of order givers may vary, but the order taker is mostly service provider at offshore. The more the number of order givers the more complex the IT Services engagement. Lets look at this continuum to understand the engagement complexity based on who the players are:




How will the challenges in different stakeholders present in the team? What are the dynamics of the engagement? What are the characteristics that make the engagement complex? In subsequent blogs we can see further details.

Have we seen climax of cloud hype?

Have we really seen the climax of Cloud hype already? Why do I think so?

maybe they really underestimated data privacy, control and ownership issue on one hand and the passing of contractual risks through liablity and idemnity clauses for data protection...that will hinder growth of cloud...


so what about the high decibel and hig frequency campaigners of cloud have to say now....

1. IAAS: Data center consolidation....maybe we are likely to see the MIS glass house again...maybe now they just call MIS glass house with some dynamic provisioning of servers :-)..
2. PAAS: are we seeing the ISVs attempt to get back in business by trying to repackage themselves as provisioning a platform fizzled out...sybase now repositions itself as mobility ISV...oracle Forms/repoorts moving into open platforms with eclipse and java-based solutions...maybe force.com doesnt have all that horsepower as it claimed it did...nobody knows whats up here another 1-2 years down the line...
3. SAAS: we just have SFDC here....others are all our classical hosting solution providers...maybe oracle ondemand, SAp are all trying to reinvent themselves...maybe they are failing in ERP III :-)....for those who saw MRP I, MRP II, MRP III really renamed themselves to SCM....no idea where this is headed....I'm not sure if anyone is really ready for a multi-tenant, pay-per-use kind of offerings for enterprise systems yet....

What is all the fuss about utility computing through cloud? we are not going to see IT becoming like a power utility company split based on generation (H/w: IBM, HP, SUN,; S/W: Oracle, Salesforce.com, SAP, N/W: Cisco!!), transmission (ISPs and telecom service providers) and distribution (???)....and businesses out there dying to hire them on rental service for what they use....it seems to me most businesses are not even sure the value they are getting from such IT systems anymore....

What do you think?

Whats up with margins of network equiment solution providers (Ciscos and sycamores)

The margins of network companies (Cisco, Juniper, HP, Sycamore) post 2008 has dropped significantly to 15-16% from 20% just about a year ago...

1. is it due to lack of any differentiations in their core business of routers and switches leading them on a race to the bottom for their margins?
2. are we seeing the climax of hype for cloud computing whereby the growth for more businesses buying new routers and switches will not materialize....
3. Imagine if cisco has to focus only on Brics for growth...we in brics know fully well there arent enough potential to grow beyond ther 10% theatre growth cisco estimates in their analyst briefing...what about their strategy of focusing on public sector investments....we have heard enough of fiscal deficits and govt. debt burden already...
4. We see Huawei and ZTE up and coming....should we be seeing chinese domination for the next decade in this core switches and routers business....

The only way for them to get back on the margin game it seems is through software and services (cisco services growing steadily to 20%+ of their revenue for FY'11). What would it make these players something like IBM? Maybe we need another lou Gestner to make this elephant dance...Lou made the mammoth dance with his services strategy...it seems we may see cisco going the IBM way....

What do you think?

Raodblocks/impediments to digitizing indian TV broadcasting

Indian broadcasting industry faces serious roadblocks to digitizing though KPMG survey estimates this to increase from 33% in 2010 to 64% by 2015. Nowadays, it is common to blame everything on corruption & politics...so it is not surprisingly true even here...the key impediments/roadblocks to quickly and profitably roll out digitisation:

1. Demand side effects: Subscriber costs will move up limiting slower uptake of services through Set-top-boxes/DTH. Subscribers have become too happy in paying Rs 200-250 for a boquet of popular channels through analog cables running. Probably the analog cable operators and franchisees can afford to provide these at such low cost due to underreporting of subscriber base by their franchisees :-)
2. Supply side: Increasing content development/delivery/marketing cost coupled with decreasing revenues in media (80% revenue comes from advertising in this industry but this would get strained probably due to other channels of communication) strains broadcasters margins & cashflows. This will make broadcasters wary of investing in digitization unless they are profitable. A sort of chicken-egg scenario....
3. Extraneous factors: Cables are run by politicians in some states...In others they are controlled by local goons who control which channel can be transmitted...the other interesting thing is politicians fear that digitization will bring accountability, and transparency removing any discretionary powers of state to control information reaching mass....so we will see lot of irrelevant non-issues being posed as reasons such as employment opportunities denied, poverty, last mile-connectivity, rural population affordability, minority issues, etc.

MIB has published a deadline to phases out analog transmission by 31st Dec 2013 and later created a phased roll out for cities and it seems it is delayed further... we can possible see a festering can of worms bred by unholy nexus here....

Why are IT services firm so confused and mess up when they sell their home-grown products and IT service?

IT services companies agents do miserably in pricing, selling and delivering their home-grown IT products and associated services. Why does this happen?

1. The sales and marketing team often sell IT software product like services instead of selling the products and then add-on services
2. The solution architects simply dont get how to solution and postion the services and IT Software product separately
3. The marketing folks are confused on how to price the licenses and support for the IT software products effectively and competetively in the market.
4. The delivery team often dont get the fact that you need to run the IT software products business like a product company and not like services. For example they need a Engineering team, development team, dedicated resources who can work on products, processes and methodologies for product development in line with what is used in the industry, professional services arm which focuses on services, which is seperate from the core products team, incentives and motivation for the team in line with industry standards, nurturing and influencing the ecosystem of the IT products through investments and participation, having a support team which focusses on providing ongoing support, a release and program management and product management team that works with cross-functional team and vendors out there to drive features,/functionalities/releasesetc.

The above are only a few, but the biggest challenge often is how do IT services companies solution and price these products?


So what is the right model to sell Home grown IT software products and associated service. To answer this lets look at what SAP and Oracles of the world do and learn from them:

SAP and Oracle charge a Software update and Product support yearly fee of ~25% of their license fee. Ofcourse, this can be negotiated. The software Update and Product support comes with free maintenance packs/patches, ongoing vendor support for product issues. If you keep applying the regular patches and maintenance packs and have done no bespoke RICEF customizations it is as good as upgrades. But this is seldom the case as any enterprise application will require you to build RICEF to work with other IT systems in organization that’s where IT service providers like us fit in.



Given the above background, if you are selling home-grown Software product, client is expecting you to provide this software update and product support. But this doesn’t come free; you have to charge client with a yearly fee a.k.a. 25% of software license fees. I don’t know what is the standard that is charged for the home-grown software product update and software support . You should provide ongoing Software updates/fixes to the product as this is home-grown software and you would have roadmap of future releases that enhances the product based on some charge. Over and above this you need to also clearly separate AM work for client like the way IT services firms do for SAP/Oracle and estimate this work and provide this as a separate service.



So your overall pricing = Home-grown software product license fees + Implementation Fees + Ongoing product and Software support fees + Application maintenance fees.



The first 2 components will be SI and does not concern your ongoing IT services. The last 2 is where you need to separate this our clearly and charge client as you are both the software vendor and application maintenance/management vendor. IT services companies often muddle up the last two components as they don’t have an existing shared services based support team which people like Oracle and SAP does so they often get confused and muddle up the scope of ongoing product and software support with application maintenance. My submission would be to scope and separate this out clearly and position this to client team so they can pitch it to client.