An Open Source-Based Cloud Data Storage and Processing Solution

Applications are increasingly being made available over the Internet. Several applications have a large user base that produces a huge volume of data, for instance, content in a community portal, emails in a web-based email system, and call log files generated at call centers. Due to a large amount of data being added every minute and the need to keep historical data for various requirements just as legal, reference, data warehousing, and analytics, the systems' data size keeps growing exponentially. This requires a huge storage and processing infrastructure, incurring a high cost of procuring and maintaining it for companies. Other typical challenges with such large data sets are how to store the data reliably and economically. How do you process the data efficiently? How do you provide search?

On-Demand Data Storage and Processing SolutionCloud computing offers the on-demand scalability of resources that can be leveraged for data storage to provide scalable storage. To efficiently and effectively manage the resources and data stored in the cloud, the cloud data storage and processing solution is presented here. Our solution uses Eucalyptus, the open source cloud platform, to manage the underlying storage infrastructure. There are some specialized open source cloud computing solutions just as Hadoop and Lucene that offer low-cost scalable alternatives for applications that need to process huge amounts of data.

Summary of the limitations of traditional solutions

Traditional vs Cloud Computing Based on On-Demand Storage SolutionsTable 1 provides a summary of the limitations of traditional solutions and how these new solutions address them.

Specialized Cloud InfrastructureThe foundation layer of the solution consists of the cloud infrastructure to virtualize the underlying hardware and provide elements on-demand. The solution leverages Eucalyptus, an open source cloud computing framework to provide the base cloud infrastructure [7]. Eucalyptus uses the Xen virtualization platform to virtualize the physical hardware. It provides on-demand scalability by enabling the addition, instantiation and management of the nodes in the cluster. These nodes not only can contain a virtual machine with the operating system now they can as well contain a complete software stack, in this way enabling the creation of virtual appliances that can be instantiated and shut down on demand. To boot, a cluster management module is included to automate and ease the management of these instances.

Zetta Enterprise Cloud Storage solutions support all unstructured data types and are backed by industry-leading data integrity and security. [5] EMC Atmos onLine is an Internet-delivered cloud storage service that provides Cloud Optimized Storage capabilities to clients with reliable SLAs and secure access. It enables clients to move data from on-premise to off-premise using policies. [6] The ParaScale Cloud Storage software does not require custom or dedicated hardware and can leverage existing IP networking interconnections. It aggregates disk storage on multiple standard Linux servers to present one or more logical namespaces, and enables file access via standard file-access protocols. Applications and customers don't have to be modified or recompiled to use PCS.

As clients traditionally store data in-house, they find it difficult to put their business at risk by moving their data out of their premises. As well they fear to risk of result of hardware failure or someone accidentally erasing or corrupting their high-value data outside their control. Along these lines private clouds are much in demand. Most of the existing solutions require the data to be moved out of the organization's premises. For having the on-demand scalable, distributed and fast-processing storage solution in the private cloud, very few options just as ParaScale Software are available. Nevertheless the open source-based solution proposed here provides a cost advantage over using commercial software. As well the customization can be done, as per client-specific requirements, with minimal effort and cost.

Shyam Kumar Doddavula works as a Principal Innovation Architect at the Cloud Computing Center of Excellence Group at Infosys Technologies Ltd. He has a MS in computer science from Texas Tech University and over 13 years experience in enterprise application architecture and development.

Senior Technical Architect with SETLabs

Nidhi Tiwari is a Senior Technical Architect with SETLabs, Infosys Technologies. She has over 10 years of experience in varied software technologies. She has been working in the field of performance engineering and cloud computing for 6 years. Her technology interests include adoption of cloud computing and cloud databases along with performance modeling. She has authored papers for international conferences, journals and has a granted patent.