2015-05-08 21:59:16 UTC
Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .
The proposal is available on the wiki here:
The proposal text is also attached to the end of this email.
Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.
Thank you in advance for giving the proposal a read.
Trafodion Apache Incubator Proposal
Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.
Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
features of Apache Trafodion include:
* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community
Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.
The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HPâs NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.
Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase co-processor.
Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.
Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.
Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.
Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.
Trafodion supports ANSIâs grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.
The name âTrafodionâ (the Welsh word for transactions, pronounced
âTra-vod-eee-onâ) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.
HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.
We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.
We have already begun building a community but at this time the community
consists only of Trafodion developers â all HP employees â and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.
The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.
HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HPâs strategy to leverage its investment in database
software and bring software to market as open source and is similar to HPâs
efforts with OpenStack. Trafodion builds on HPâs equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.
Inexperience with Open Source
We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.
Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.
Reliance on Salaried Developers
Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.
Relationship with Other Apache Products
Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.
Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.
There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.
An Excessive Fascination with the Apache Brand
We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
Documentation and technical details on Trafodion can be found at:
The source is available today in a public github repository:
Source and Intellectual Property Submission Plan
The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.
Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.
JIRA: JIRA Trafodion (Trafodion)
Initial Committers and Affiliation
Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Michael Stack, Stack<AT>apache<DOT>org
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io
We are seeking additional mentors.
Apache Incubator PMC