Discussion:
[DISCUSS] Trafodion Incubation Proposal
(too old to reply)
Stack
2015-05-08 21:59:16 UTC
Permalink
I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.

Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .

The proposal is available on the wiki here:
https://wiki.apache.org/incubator/TrafodionProposal#preview

The proposal text is also attached to the end of this email.

Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.

Thank you in advance for giving the proposal a read.

Yours,
St.Ack


Trafodion Apache Incubator Proposal

Abstract

Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.

Proposal

Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
features of Apache Trafodion include:

* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity

Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community

Background

Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.

The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.

Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase co-processor.

Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.

Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.

Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.

Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.

Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.

Rationale

The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.

Current Status

HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.

Meritocracy

We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.

Community

We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.

The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.

Core Developers

The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.

Alignment

Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.

Known Risks

Orphaned Products

HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HP’s strategy to leverage its investment in database
software and bring software to market as open source and is similar to HP’s
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.

Inexperience with Open Source

We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.

Homogenous Developers

Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.

Reliance on Salaried Developers

Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.

Relationship with Other Apache Products

Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.

Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.

Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.

There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.

An Excessive Fascination with the Apache Brand

We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.

Documentation

Documentation and technical details on Trafodion can be found at:
http://www.trafodion.org/

Initial Source

The source is available today in a public github repository:
https://github.com/trafodion/trafodion.

Source and Intellectual Property Submission Plan

The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.

External Dependencies

Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.

Cryptography

Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.

Required Resources

Mailing Lists

***@trafodion.incubator.apache.org
***@trafodion.incubator.apache.org ***@trafodion.incubator.apache.org

Git Repository

https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git

Issue Tracking

JIRA: JIRA Trafodion (Trafodion)


Initial Committers and Affiliation

Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com

Sponsors

Champion

Michael Stack, Stack<AT>apache<DOT>org

Nominated Mentors

Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io

We are seeking additional mentors.

Sponsoring Entity

Apache Incubator PMC
Dan Di Spaltro
2015-05-11 00:25:38 UTC
Permalink
I think this would be a great addition, and I am sure this will help fix
the committer diversity.


-Dan
Post by Stack
I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.
Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .
https://wiki.apache.org/incubator/TrafodionProposal#preview
The proposal text is also attached to the end of this email.
Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.
Thank you in advance for giving the proposal a read.
Yours,
St.Ack
Trafodion Apache Incubator Proposal
Abstract
Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.
Proposal
Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity
Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community
Background
Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.
The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.
Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase
co-processor.
Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.
Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.
Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.
Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.
Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.
Rationale
The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.
Current Status
HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.
Meritocracy
We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.
Community
We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.
The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.
Core Developers
The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.
Alignment
Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.
Known Risks
Orphaned Products
HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HP’s strategy to leverage its investment in database
software and bring software to market as open source and is similar to HP’s
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.
Inexperience with Open Source
We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.
Homogenous Developers
Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.
Reliance on Salaried Developers
Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.
Relationship with Other Apache Products
Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.
Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.
Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.
There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.
An Excessive Fascination with the Apache Brand
We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.
Documentation
http://www.trafodion.org/
Initial Source
https://github.com/trafodion/trafodion.
Source and Intellectual Property Submission Plan
The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.
External Dependencies
Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.
Cryptography
Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.
Required Resources
Mailing Lists
Git Repository
https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git
Issue Tracking
JIRA: JIRA Trafodion (Trafodion)
Initial Committers and Affiliation
Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com
Sponsors
Champion
Michael Stack, Stack<AT>apache<DOT>org
Nominated Mentors
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io
We are seeking additional mentors.
Sponsoring Entity
Apache Incubator PMC
Konstantin Boudnik
2015-05-11 02:13:11 UTC
Permalink
I think it'd be great to have SQL platform for Hadoop

+1

I am mentoring 4 projects at the moment, but if you need a 1/2 time mentor -
count me in ;)

Cos
Post by Stack
I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.
Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .
https://wiki.apache.org/incubator/TrafodionProposal#preview
The proposal text is also attached to the end of this email.
Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.
Thank you in advance for giving the proposal a read.
Yours,
St.Ack
Trafodion Apache Incubator Proposal
Abstract
Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.
Proposal
Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity
Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community
Background
Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.
The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.
Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase co-processor.
Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.
Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.
Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.
Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.
Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.
Rationale
The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.
Current Status
HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.
Meritocracy
We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.
Community
We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.
The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.
Core Developers
The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.
Alignment
Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.
Known Risks
Orphaned Products
HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HP’s strategy to leverage its investment in database
software and bring software to market as open source and is similar to HP’s
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.
Inexperience with Open Source
We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.
Homogenous Developers
Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.
Reliance on Salaried Developers
Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.
Relationship with Other Apache Products
Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.
Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.
Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.
There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.
An Excessive Fascination with the Apache Brand
We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.
Documentation
http://www.trafodion.org/
Initial Source
https://github.com/trafodion/trafodion.
Source and Intellectual Property Submission Plan
The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.
External Dependencies
Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.
Cryptography
Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.
Required Resources
Mailing Lists
Git Repository
https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git
Issue Tracking
JIRA: JIRA Trafodion (Trafodion)
Initial Committers and Affiliation
Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com
Sponsors
Champion
Michael Stack, Stack<AT>apache<DOT>org
Nominated Mentors
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io
We are seeking additional mentors.
Sponsoring Entity
Apache Incubator PMC
---------------------------------------------------------------------
To unsubscribe, e-mail: general-***@incubator.apache.org
For additional commands, e-mail: general-***@incubator.apache.org
Stack
2015-05-11 03:06:32 UTC
Permalink
Post by Konstantin Boudnik
I think it'd be great to have SQL platform for Hadoop
+1
I am mentoring 4 projects at the moment, but if you need a 1/2 time mentor -
count me in ;)
Cos
We'll take you up on your kind offer if we can't get someone less loaded.

Thanks Cos,

St.Ack
Post by Konstantin Boudnik
Post by Stack
I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.
Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .
https://wiki.apache.org/incubator/TrafodionProposal#preview
The proposal text is also attached to the end of this email.
Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.
Thank you in advance for giving the proposal a read.
Yours,
St.Ack
Trafodion Apache Incubator Proposal
Abstract
Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.
Proposal
Apache Trafodion builds on the scalability, elasticity, and flexibility
of
Post by Stack
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop.
Key
Post by Stack
* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity
Hewlett-Packard Company submits this proposal to donate its Apache
License,
Post by Stack
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community
Background
Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution
targeting
Post by Stack
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June
2014.
Post by Stack
The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and
is
Post by Stack
known for its high availability, scalability, and performance. Hundreds
of
Post by Stack
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these
components
Post by Stack
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.
Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase
co-processor.
Post by Stack
Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including
full-functioned
Post by Stack
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.
Trafodion provides comprehensive and standard SQL data manipulation
support
Post by Stack
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated
and
Post by Stack
nested sub-queries, cursors, and many SQL functions.
Utilities are provided for updating table statistics used by the
optimizer
Post by Stack
for costing (i.e. selectivity/cardinality estimates) plan alternatives,
for
Post by Stack
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.
Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted,
including
Post by Stack
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.
Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.
Rationale
The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the
differentiation
Post by Stack
that Trafodion provides in closing a critical gap in the Hadoop
ecosystem.
Post by Stack
Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop.
Post by Stack
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.
Current Status
HP released the Trafodion code under the Apache License, Version 2, in
June
Post by Stack
of 2014. Since that time, we have had one major release in January 2015
and
Post by Stack
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.
Meritocracy
We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where
developers
Post by Stack
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.
Community
We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and
prospective
Post by Stack
users. We have participated in and hosted HBase Meetups and intend to
ramp
Post by Stack
up our community building efforts.
The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.
Core Developers
The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.
Alignment
Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.
Known Risks
Orphaned Products
HP Labs and HP-IT have been incubating Trafodion development for almost
two
Post by Stack
years. This is part of HP’s strategy to leverage its investment in
database
Post by Stack
software and bring software to market as open source and is similar to
HP’s
Post by Stack
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.
Inexperience with Open Source
We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open
source
Post by Stack
development, the development team received guidance and best practices
from
Post by Stack
HP developers working on OpenStack open source projects, many of whom
have
Post by Stack
experience working on Apache and other open source projects as well.
Since
Post by Stack
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints
and
Post by Stack
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools
and
Post by Stack
conduct our discussions using Apache email dlists.
Homogenous Developers
Since the initial development of Trafodion has been supported by HP, all
of
Post by Stack
the current developers are HP employees. Through the support of the
Apache
Post by Stack
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed
development
Post by Stack
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.
Reliance on Salaried Developers
Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.
Relationship with Other Apache Products
Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with
the
Post by Stack
Apache HBase project on performance optimizations, such as in the areas
of
Post by Stack
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.
Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process
should
Post by Stack
fail.
Trafodion also envisions working with the Apache Ambari project on
enabling
Post by Stack
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus
in
Post by Stack
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.
There are alternative open source projects that are providing
SQL-on-Hadoop
Post by Stack
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix.
These
Post by Stack
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security,
and
Post by Stack
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in
HDFS.
Post by Stack
An Excessive Fascination with the Apache Brand
We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community
building
Post by Stack
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand
the
Post by Stack
rules and guidelines about the use of the Apache brand and intend to
follow
Post by Stack
them.
Documentation
http://www.trafodion.org/
Initial Source
https://github.com/trafodion/trafodion.
Source and Intellectual Property Submission Plan
The source code has already been released under the Apache License,
Version
Post by Stack
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.
External Dependencies
Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache
incubator
Post by Stack
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.
Cryptography
Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.
Required Resources
Mailing Lists
Git Repository
https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git
Issue Tracking
JIRA: JIRA Trafodion (Trafodion)
Initial Committers and Affiliation
Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com
Sponsors
Champion
Michael Stack, Stack<AT>apache<DOT>org
Nominated Mentors
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io
We are seeking additional mentors.
Sponsoring Entity
Apache Incubator PMC
---------------------------------------------------------------------
Enis Söztutar
2015-05-12 17:42:54 UTC
Permalink
This looks pretty good. I can also join as a mentor.

Enis
Post by Stack
Post by Konstantin Boudnik
I think it'd be great to have SQL platform for Hadoop
+1
I am mentoring 4 projects at the moment, but if you need a 1/2 time
mentor
Post by Konstantin Boudnik
-
count me in ;)
Cos
We'll take you up on your kind offer if we can't get someone less loaded.
Thanks Cos,
St.Ack
Post by Konstantin Boudnik
Post by Stack
I would like to start up a discussion on Trafodion joining the ASF as
an
Post by Konstantin Boudnik
Post by Stack
incubating project.
Trafodion is a webscale SQL-on-Hadoop solution that enables
transactional
Post by Konstantin Boudnik
Post by Stack
or operational workloads on Hadoop, .
https://wiki.apache.org/incubator/TrafodionProposal#preview
The proposal text is also attached to the end of this email.
Trafodion is a rich, storied SQL engine that has recently been ported
to
Post by Konstantin Boudnik
Post by Stack
run on HBase and Hadoop. I think it would make for a fine addition to
the
Post by Konstantin Boudnik
Post by Stack
Apache family of projects It would be good to hear what others think.
Thank you in advance for giving the proposal a read.
Yours,
St.Ack
Trafodion Apache Incubator Proposal
Abstract
Trafodion is a webscale SQL-on-Hadoop solution enabling transactional
or
Post by Konstantin Boudnik
Post by Stack
operational workloads on Hadoop.
Proposal
Apache Trafodion builds on the scalability, elasticity, and flexibility
of
Post by Stack
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on
Hadoop.
Post by Konstantin Boudnik
Key
Post by Stack
* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including
referential
Post by Konstantin Boudnik
Post by Stack
integrity
Hewlett-Packard Company submits this proposal to donate its Apache
License,
Post by Stack
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation
in
Post by Konstantin Boudnik
Post by Stack
order to build an open source community
Background
Trafodion is an open source project sponsored by HP, incubated at HP
Labs
Post by Konstantin Boudnik
Post by Stack
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution
targeting
Post by Stack
big data transactional or operational workloads. HP publically
announced
Post by Konstantin Boudnik
Post by Stack
the open source project and uploaded the source code to GitHub in June
2014.
Post by Stack
The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online
transaction
Post by Konstantin Boudnik
Post by Stack
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and
is
Post by Stack
known for its high availability, scalability, and performance. Hundreds
of
Post by Stack
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these
components
Post by Stack
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.
Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase
co-processor.
Post by Stack
Unlike most NOSQL and other SQL-on-Hadoop open source projects,
Trafodion
Post by Konstantin Boudnik
Post by Stack
provides comprehensive ANSI SQL language support including
full-functioned
Post by Stack
data definition (DDL), data manipulation (DML), transaction control
(TCL)
Post by Konstantin Boudnik
Post by Stack
and database utility support.
Trafodion provides comprehensive and standard SQL data manipulation
support
Post by Stack
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated
and
Post by Stack
nested sub-queries, cursors, and many SQL functions.
Utilities are provided for updating table statistics used by the
optimizer
Post by Stack
for costing (i.e. selectivity/cardinality estimates) plan alternatives,
for
Post by Stack
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.
Explicit control statements are provided to allow applications to
define
Post by Konstantin Boudnik
Post by Stack
transaction boundaries and to abort transactions when warranted,
including
Post by Stack
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.
Trafodion supports ANSI’s grant/revoke semantics to define user and
role
Post by Konstantin Boudnik
Post by Stack
privileges in terms of managing and accessing the database objects.
Rationale
The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the
differentiation
Post by Stack
that Trafodion provides in closing a critical gap in the Hadoop
ecosystem.
Post by Stack
Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop.
Post by Stack
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.
Current Status
HP released the Trafodion code under the Apache License, Version 2, in
June
Post by Stack
of 2014. Since that time, we have had one major release in January 2015
and
Post by Stack
one minor release in April 2015. The focus of these releases has been
in
Post by Konstantin Boudnik
Post by Stack
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.
Meritocracy
We want to build a diverse developer community, based on the Apache
Way,
Post by Konstantin Boudnik
Post by Stack
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where
developers
Post by Stack
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and
integrating
Post by Konstantin Boudnik
Post by Stack
with other Apache projects in the Hadoop ecosystem.
Community
We have already begun building a community but at this time the
community
Post by Konstantin Boudnik
Post by Stack
consists only of Trafodion developers – all HP employees – and
prospective
Post by Stack
users. We have participated in and hosted HBase Meetups and intend to
ramp
Post by Stack
up our community building efforts.
The Trafodion project has seen interest in China, where HP has
conducted
Post by Konstantin Boudnik
Post by Stack
proof-of-concepts with multiple companies and expects to see some of
its
Post by Konstantin Boudnik
Post by Stack
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.
Core Developers
The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working
in
Post by Konstantin Boudnik
Post by Stack
this space.
Alignment
Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored
in
Post by Konstantin Boudnik
Post by Stack
HBase, native HBase tables, and Apache Hive tables.
Known Risks
Orphaned Products
HP Labs and HP-IT have been incubating Trafodion development for almost
two
Post by Stack
years. This is part of HP’s strategy to leverage its investment in
database
Post by Stack
software and bring software to market as open source and is similar to
HP’s
Post by Stack
efforts with OpenStack. Trafodion builds on HP’s equity investment in
the
Post by Konstantin Boudnik
Post by Stack
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.
Inexperience with Open Source
We have been working with open source software in building closed
source
Post by Konstantin Boudnik
Post by Stack
software for well over two decades. To help transition to doing open
source
Post by Stack
development, the development team received guidance and best practices
from
Post by Stack
HP developers working on OpenStack open source projects, many of whom
have
Post by Stack
experience working on Apache and other open source projects as well.
Since
Post by Stack
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints
and
Post by Stack
Jenkins for continuous integration. As part of the incubation process,
we
Post by Konstantin Boudnik
Post by Stack
recognize we may need to change some of our development processes/tools
and
Post by Stack
conduct our discussions using Apache email dlists.
Homogenous Developers
Since the initial development of Trafodion has been supported by HP,
all
Post by Konstantin Boudnik
of
Post by Stack
the current developers are HP employees. Through the support of the
Apache
Post by Stack
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed
development
Post by Stack
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.
Reliance on Salaried Developers
Currently all of the developers working on the project are paid by
their
Post by Konstantin Boudnik
Post by Stack
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported
distribution
Post by Konstantin Boudnik
Post by Stack
of Trafodion that HP will offer.
Relationship with Other Apache Products
Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with
the
Post by Stack
Apache HBase project on performance optimizations, such as in the areas
of
Post by Stack
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.
Trafodion uses Apache Zookeeper to coordinate and manage the
distribution
Post by Konstantin Boudnik
Post by Stack
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process
should
Post by Stack
fail.
Trafodion also envisions working with the Apache Ambari project on
enabling
Post by Stack
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus
in
Post by Stack
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database
administrators.
Post by Konstantin Boudnik
Post by Stack
There are alternative open source projects that are providing
SQL-on-Hadoop
Post by Stack
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix.
These
Post by Stack
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security,
and
Post by Stack
other capabilities that would take decades to build in these products.
On
Post by Konstantin Boudnik
Post by Stack
the other hand currently Trafodion is just focused on HBase and
querying
Post by Konstantin Boudnik
Post by Stack
Hive, whereas Hive and Drill provide access to other data formats in
HDFS.
Post by Stack
An Excessive Fascination with the Apache Brand
We understand the reputation and value of the Apache brand, and no
doubt
Post by Konstantin Boudnik
Post by Stack
believe that it will help us attract contributors and users. Our
primary
Post by Konstantin Boudnik
Post by Stack
goal is to follow a proven, open source development and community
building
Post by Stack
model that will make Trafodion successful and enable better
collaboration
Post by Konstantin Boudnik
Post by Stack
with other Apache projects in the Hadoop ecosystem. We also understand
the
Post by Stack
rules and guidelines about the use of the Apache brand and intend to
follow
Post by Stack
them.
Documentation
http://www.trafodion.org/
Initial Source
https://github.com/trafodion/trafodion.
Source and Intellectual Property Submission Plan
The source code has already been released under the Apache License,
Version
Post by Stack
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.
External Dependencies
Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache
incubator
Post by Stack
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being
able
Post by Konstantin Boudnik
Post by Stack
to use this incompatible software and do similar. All other
dependencies
Post by Konstantin Boudnik
Post by Stack
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT,
and
Post by Konstantin Boudnik
Post by Stack
BSD.
Cryptography
Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.
Required Resources
Mailing Lists
Git Repository
https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git
Issue Tracking
JIRA: JIRA Trafodion (Trafodion)
Initial Committers and Affiliation
Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com
Sponsors
Champion
Michael Stack, Stack<AT>apache<DOT>org
Nominated Mentors
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io
We are seeking additional mentors.
Sponsoring Entity
Apache Incubator PMC
---------------------------------------------------------------------
Andrew Purtell
2015-05-12 18:01:07 UTC
Permalink
If you are looking for mentors, I helped on the Phoenix incubation, happy
to do so again for Trafodion.
Post by Stack
I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.
Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .
https://wiki.apache.org/incubator/TrafodionProposal#preview
The proposal text is also attached to the end of this email.
Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.
Thank you in advance for giving the proposal a read.
Yours,
St.Ack
Trafodion Apache Incubator Proposal
Abstract
Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.
Proposal
Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity
Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community
Background
Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.
The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.
Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase
co-processor.
Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.
Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.
Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.
Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.
Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.
Rationale
The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.
Current Status
HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.
Meritocracy
We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.
Community
We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.
The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.
Core Developers
The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.
Alignment
Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.
Known Risks
Orphaned Products
HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HP’s strategy to leverage its investment in database
software and bring software to market as open source and is similar to HP’s
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.
Inexperience with Open Source
We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.
Homogenous Developers
Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.
Reliance on Salaried Developers
Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.
Relationship with Other Apache Products
Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.
Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.
Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.
There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.
An Excessive Fascination with the Apache Brand
We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.
Documentation
http://www.trafodion.org/
Initial Source
https://github.com/trafodion/trafodion.
Source and Intellectual Property Submission Plan
The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.
External Dependencies
Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.
Cryptography
Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.
Required Resources
Mailing Lists
Git Repository
https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git
Issue Tracking
JIRA: JIRA Trafodion (Trafodion)
Initial Committers and Affiliation
Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com
Sponsors
Champion
Michael Stack, Stack<AT>apache<DOT>org
Nominated Mentors
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io
We are seeking additional mentors.
Sponsoring Entity
Apache Incubator PMC
--
Best regards,

- Andy

Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Devaraj Das
2015-05-12 19:09:07 UTC
Permalink
Looks good. If you need more mentors on this, please count me in.
________________________________________
From: ***@gmail.com <***@gmail.com> on behalf of Stack <***@duboce.net>
Sent: Friday, May 08, 2015 2:59 PM
To: ***@incubator.apache.org
Subject: [DISCUSS] Trafodion Incubation Proposal

I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.

Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .

The proposal is available on the wiki here:
https://wiki.apache.org/incubator/TrafodionProposal#preview

The proposal text is also attached to the end of this email.

Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.

Thank you in advance for giving the proposal a read.

Yours,
St.Ack


Trafodion Apache Incubator Proposal

Abstract

Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.

Proposal

Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
features of Apache Trafodion include:

* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity

Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community

Background

Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.

The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.

Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase co-processor.

Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.

Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.

Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.

Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.

Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.

Rationale

The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.

Current Status

HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.

Meritocracy

We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.

Community

We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.

The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.

Core Developers

The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.

Alignment

Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.

Known Risks

Orphaned Products

HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HP’s strategy to leverage its investment in database
software and bring software to market as open source and is similar to HP’s
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.

Inexperience with Open Source

We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.

Homogenous Developers

Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.

Reliance on Salaried Developers

Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.

Relationship with Other Apache Products

Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.

Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.

Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.

There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.

An Excessive Fascination with the Apache Brand

We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.

Documentation

Documentation and technical details on Trafodion can be found at:
http://www.trafodion.org/

Initial Source

The source is available today in a public github repository:
https://github.com/trafodion/trafodion.

Source and Intellectual Property Submission Plan

The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.

External Dependencies

Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.

Cryptography

Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.

Required Resources

Mailing Lists

***@trafodion.incubator.apache.org
***@trafodion.incubator.apache.org ***@trafodion.incubator.apache.org

Git Repository

https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git

Issue Tracking

JIRA: JIRA Trafodion (Trafodion)


Initial Committers and Affiliation

Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com

Sponsors

Champion

Michael Stack, Stack<AT>apache<DOT>org

Nominated Mentors

Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io

We are seeking additional mentors.

Sponsoring Entity

Apache Incubator PMC

---------------------------------------------------------------------
To unsubscribe, e-mail: general-***@incubator.apache.org
For additional commands, e-mail: general-***@incubator.apache.org
Birdsall, Dave
2015-05-13 23:34:19 UTC
Permalink
Hi,

I just wanted to acknowledge and thank the following folks for volunteering to mentor Trafodion on its way into and through incubation as an Apache project:

Michael Stack (champion)
Roman Shaposhnik (mentor)
Andrew Purtell (mentor)
Enis Söztutar (mentor)
Devaraj Das (mentor)
Konstantin Boudnik (mentor, half time)

More mentors are welcome of course. (And please forgive my inexperience; I don't know what number of mentors is common for incubating projects.)

Thanks,

Dave Birdsall, Community Manager, Trafodion project
lars hofhansl
2015-05-14 22:39:01 UTC
Permalink
Bit belated... I'd be happy and honored to be a mentor for Trafodian.
-- Lars
From: Stack <***@duboce.net>
To: ***@incubator.apache.org
Sent: Friday, May 8, 2015 2:59 PM
Subject: [DISCUSS] Trafodion Incubation Proposal

I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.

Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .

The proposal is available on the wiki here:
https://wiki.apache.org/incubator/TrafodionProposal#preview

The proposal text is also attached to the end of this email.

Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects  It would be good to hear what others think.

Thank you in advance for giving the proposal a read.

Yours,
St.Ack


Trafodion Apache Incubator Proposal

Abstract

Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.

Proposal

Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
features of Apache Trafodion include:

* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity

Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community

Background

Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.

The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.

Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase co-processor.

Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.

Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.

Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.

Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.

Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.

Rationale

The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.

Current Status

HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.

Meritocracy

We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.

Community

We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.

The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.

Core Developers

The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.

Alignment

Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.

Known Risks

Orphaned Products

HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HP’s strategy to leverage its investment in database
software and bring software to market as open source and is similar to HP’s
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.

Inexperience with Open Source

We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.

Homogenous Developers

Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.

Reliance on Salaried Developers

Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.

Relationship with Other Apache Products

Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.

Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.

Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.

There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.

An Excessive Fascination with the Apache Brand

We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.

Documentation

Documentation and technical details on Trafodion can be found at:
http://www.trafodion.org/

Initial Source

The source is available today in a public github repository:
https://github.com/trafodion/trafodion.

Source and Intellectual Property Submission Plan

The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.

External Dependencies

Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.

Cryptography

Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.

Required Resources

Mailing Lists

***@trafodion.incubator.apache.org
***@trafodion.incubator.apache.org ***@trafodion.incubator.apache.org

Git Repository

https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git

Issue Tracking

JIRA: JIRA Trafodion (Trafodion)


Initial Committers and Affiliation

Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com

Sponsors

Champion

Michael Stack, Stack<AT>apache<DOT>org

Nominated Mentors

Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io

We are seeking additional mentors.

Sponsoring Entity

Apache Incubator PMC
Birdsall, Dave
2015-05-14 23:10:22 UTC
Permalink
Hi Lars,

Glad to have you on the mentor list!

Dave

-----Original Message-----
From: lars hofhansl [mailto:***@apache.org]
Sent: Thursday, May 14, 2015 3:39 PM
To: ***@incubator.apache.org
Subject: Re: [DISCUSS] Trafodion Incubation Proposal

Bit belated... I'd be happy and honored to be a mentor for Trafodian.
-- Lars
From: Stack <***@duboce.net>
To: ***@incubator.apache.org
Sent: Friday, May 8, 2015 2:59 PM
Subject: [DISCUSS] Trafodion Incubation Proposal

I would like to start up a discussion on Trafodion joining the ASF as an incubating project.

Trafodion is a webscale SQL-on-Hadoop solution that enables transactional or operational workloads on Hadoop, .

The proposal is available on the wiki here:
https://wiki.apache.org/incubator/TrafodionProposal#preview

The proposal text is also attached to the end of this email.

Trafodion is a rich, storied SQL engine that has recently been ported to run on HBase and Hadoop. I think it would make for a fine addition to the Apache family of projects  It would be good to hear what others think.

Thank you in advance for giving the proposal a read.

Yours,
St.Ack


Trafodion Apache Incubator Proposal

Abstract

Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or operational workloads on Hadoop.

Proposal

Apache Trafodion builds on the scalability, elasticity, and flexibility of Hadoop. Trafodion extends Hadoop to provide guaranteed transactional integrity, enabling new kinds of big data applications to run on Hadoop. Key features of Apache Trafodion include:

* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements, tables and rows
* Performance improvements for OLTP workloads with compile-time and run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential integrity

Hewlett-Packard Company submits this proposal to donate its Apache License, Version 2.0 open source project known as Trafodion, its source code, documentation, and web site content to the Apache Software Foundation in order to build an open source community

Background

Trafodion is an open source project sponsored by HP, incubated at HP Labs and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting big data transactional or operational workloads. HP publically announced the open source project and uploaded the source code to GitHub in June 2014.

The SQL compiler, optimizer and executor components of Trafodion have a rich heritage. Under development since 1993, they were released as commercial closed source software in various flavors such as HP NonStop SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is known for its high availability, scalability, and performance. Hundreds of companies and thousands of servers are running mission-critical applications today on NonStop SQL/MX. In addition, much of these components today are running internal to HP as the core of its Enterprise Data Warehouse (EDW), managing over a PB of data.

Starting in 2013, the software was modified to run on HBase and a new distributed transaction manager was written to run as an HBase co-processor.

Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion provides comprehensive ANSI SQL language support including full-functioned data definition (DDL), data manipulation (DML), transaction control (TCL) and database utility support.

Trafodion provides comprehensive and standard SQL data manipulation support including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with language options including join variants, unions, where predicates, aggregations (group by and having), sort ordering, sampling, correlated and nested sub-queries, cursors, and many SQL functions.

Utilities are provided for updating table statistics used by the optimizer for costing (i.e. selectivity/cardinality estimates) plan alternatives, for displaying the chosen SQL execution plan, plan shaping, backup and restoring the database, data loading and unloading, and a command line utility for interfacing with the database engine.

Explicit control statements are provided to allow applications to define transaction boundaries and to abort transactions when warranted, including BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.

Trafodion supports ANSI’s grant/revoke semantics to define user and role privileges in terms of managing and accessing the database objects.

Rationale

The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity, enabling new kinds of big data applications to run on Hadoop.

Current Status

HP released the Trafodion code under the Apache License, Version 2, in June of 2014. Since that time, we have had one major release in January 2015 and one minor release in April 2015. The focus of these releases has been in getting our base functionality, including security, working on top of Apache HBase, as well as improving performance, availability and scalability, and integrating better with HBase.

Meritocracy

We want to build a diverse developer community, based on the Apache Way, around Trafodion. To help developers become contributors, we have documentation on the wiki about the architecture, the source tree structure, and an example enhancement. We plan to publish our project backlog to the community, specifically highlighting areas where developers new to Trafodion may best start contributing, such as extending the database functionality with User Defined Routines (UDRs) and integrating with other Apache projects in the Hadoop ecosystem.

Community

We have already begun building a community but at this time the community consists only of Trafodion developers – all HP employees – and prospective users. We have participated in and hosted HBase Meetups and intend to ramp up our community building efforts.

The Trafodion project has seen interest in China, where HP has conducted proof-of-concepts with multiple companies and expects to see some of its first commercial deployments. To help recruit contributors and users in China, members of the team are translating Trafodion wiki content into Mandarin.

Core Developers

The core developers are very experienced in database and transaction monitor technology, with many having spent more than 20 years working in this space.

Alignment

Apache Trafodion relies on Apache HBase as its storage engine. The development team has collaborated with and gained valuable advice from working with the Apache HBase core developers. Apache Trafodion has federation capabilities as well, and can query Trafodion tables stored in HBase, native HBase tables, and Apache Hive tables.

Known Risks

Orphaned Products

HP Labs and HP-IT have been incubating Trafodion development for almost two years. This is part of HP’s strategy to leverage its investment in database software and bring software to market as open source and is similar to HP’s efforts with OpenStack. Trafodion builds on HP’s equity investment in the Hadoop ecosystem and its efforts to monetize Hadoop through hardware, software, and services. HP wants Trafodion to be successful, as HP will offer a commercially supported distribution of Trafodion.

Inexperience with Open Source

We have been working with open source software in building closed source software for well over two decades. To help transition to doing open source development, the development team received guidance and best practices from HP developers working on OpenStack open source projects, many of whom have experience working on Apache and other open source projects as well. Since releasing Trafodion as an open source project in June of 2014, the committers and contributors have moved forward using open source development processes and tools for bug tracking and design blueprints and Jenkins for continuous integration. As part of the incubation process, we recognize we may need to change some of our development processes/tools and conduct our discussions using Apache email dlists.

Homogenous Developers

Since the initial development of Trafodion has been supported by HP, all of the current developers are HP employees. Through the support of the Apache incubation project, we aim to expand the list of developers and gain contributors from related SQL-on-Hadoop projects and the Apache HBase project. Trafodion developers are experienced with distributed development processes, being primarily based in Palo Alto, CA; Austin, TX; and Shanghai, China. Trafodion is written in C++ and Java.

Reliance on Salaried Developers

Currently all of the developers working on the project are paid by their employer to work on the project. These developers will work on the open source project as well as work on the commercially supported distribution of Trafodion that HP will offer.

Relationship with Other Apache Products

Trafodion is built upon Apache HBase and extends it to support ACID transactions with HBase co-processors for distributed transaction management and recovery. Trafodion envisions future collaborations with the Apache HBase project on performance optimizations, such as in the areas of mixed workload support, High Availability, etc. It also provides transactional support and querying from native HBase tables as well.

Trafodion uses Apache Zookeeper to coordinate and manage the distribution of connection services across the cluster for load-balancing and high availability reconnection purposes in the event a Trafodion process should fail.

Trafodion also envisions working with the Apache Ambari project on enabling better Trafodion manageability. While Ambari focuses on system and component level performance metrics, Trafodion manageability will focus in a complimentary way on database workload monitoring and performance analytics with capabilities more geared towards database administrators.

There are alternative open source projects that are providing SQL-on-Hadoop capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These are more focused on reporting and analytics across data structures supported on HDFS. In comparison to all of these technologies Trafodion provides a very complete implementation of ANSI SQL, one of the most sophisticated optimizers for such workloads, a completely parallel data flow architecture that does not materialize intermediate results unless necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and other capabilities that would take decades to build in these products. On the other hand currently Trafodion is just focused on HBase and querying Hive, whereas Hive and Drill provide access to other data formats in HDFS.

An Excessive Fascination with the Apache Brand

We understand the reputation and value of the Apache brand, and no doubt believe that it will help us attract contributors and users. Our primary goal is to follow a proven, open source development and community building model that will make Trafodion successful and enable better collaboration with other Apache projects in the Hadoop ecosystem. We also understand the rules and guidelines about the use of the Apache brand and intend to follow them.

Documentation

Documentation and technical details on Trafodion can be found at:
http://www.trafodion.org/

Initial Source

The source is available today in a public github repository:
https://github.com/trafodion/trafodion.

Source and Intellectual Property Submission Plan

The source code has already been released under the Apache License, Version 2. The manuals have been released in Adobe PDF format. As part of the submission process, the source for the manuals will be converted from a proprietary DocBook XML format to AsciiDoc.

External Dependencies

Two dependencies do not have Apache compatible licenses and will be addressed as we enter incubation. One dependency is log4cpp, which is licensed under the LGPL. A compatible alternative might be Apache incubator project log4cxx. The other dependency is unixodbc, which is used as the ODBC driver manager. We will look into how Apache Hive manages being able to use this incompatible software and do similar. All other dependencies have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and BSD.

Cryptography

Trafodion does not contain any cryptographic code. It does call cryptographic libraries: OpenSSL for C++ code and Java Cryptography Extension (JCE) for Java code.

Required Resources

Mailing Lists

***@trafodion.incubator.apache.org
***@trafodion.incubator.apache.org ***@trafodion.incubator.apache.org

Git Repository

https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git

Issue Tracking

JIRA: JIRA Trafodion (Trafodion)


Initial Committers and Affiliation

Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com Sandyha Sundaresan, Hewlett-Packard Company, Sandhya.Sundaresan<AT>hp<DOT>com

Sponsors

Champion

Michael Stack, Stack<AT>apache<DOT>org

Nominated Mentors

Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io

We are seeking additional mentors.

Sponsoring Entity

Apache Incubator PMC



---------------------------------------------------------------------
To unsubscribe, e-mail: general-***@incubator.apache.org
For additional commands
Venkat Ranganathan
2015-05-15 03:02:24 UTC
Permalink
Glad to see Trafodion submitted as an Apache Incubator project.

Good luck

Venkat


On May 14, 2015, at 4:13 PM, "Birdsall, Dave" <***@hp.com> wrote:

Hi Lars,

Glad to have you on the mentor list!

Dave

-----Original Message-----
From: lars hofhansl [mailto:***@apache.org]
Sent: Thursday, May 14, 2015 3:39 PM
To: ***@incubator.apache.org
Subject: Re: [DISCUSS] Trafodion Incubation Proposal

Bit belated... I'd be happy and honored to be a mentor for Trafodian.
-- Lars
From: Stack <***@duboce.net>
To: ***@incubator.apache.org
Sent: Friday, May 8, 2015 2:59 PM
Subject: [DISCUSS] Trafodion Incubation Proposal

I would like to start up a discussion on Trafodion joining the ASF as an incubating project.

Trafodion is a webscale SQL-on-Hadoop solution that enables transactional or operational workloads on Hadoop, .

The proposal is available on the wiki here:
https://wiki.apache.org/incubator/TrafodionProposal#preview

The proposal text is also attached to the end of this email.

Trafodion is a rich, storied SQL engine that has recently been ported to run on HBase and Hadoop. I think it would make for a fine addition to the Apache family of projects It would be good to hear what others think.

Thank you in advance for giving the proposal a read.

Yours,
St.Ack


Trafodion Apache Incubator Proposal

Abstract

Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or operational workloads on Hadoop.

Proposal

Apache Trafodion builds on the scalability, elasticity, and flexibility of Hadoop. Trafodion extends Hadoop to provide guaranteed transactional integrity, enabling new kinds of big data applications to run on Hadoop. Key features of Apache Trafodion include:

* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements, tables and rows
* Performance improvements for OLTP workloads with compile-time and run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential integrity

Hewlett-Packard Company submits this proposal to donate its Apache License, Version 2.0 open source project known as Trafodion, its source code, documentation, and web site content to the Apache Software Foundation in order to build an open source community

Background

Trafodion is an open source project sponsored by HP, incubated at HP Labs and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting big data transactional or operational workloads. HP publically announced the open source project and uploaded the source code to GitHub in June 2014.

The SQL compiler, optimizer and executor components of Trafodion have a rich heritage. Under development since 1993, they were released as commercial closed source software in various flavors such as HP NonStop SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is known for its high availability, scalability, and performance. Hundreds of companies and thousands of servers are running mission-critical applications today on NonStop SQL/MX. In addition, much of these components today are running internal to HP as the core of its Enterprise Data Warehouse (EDW), managing over a PB of data.

Starting in 2013, the software was modified to run on HBase and a new distributed transaction manager was written to run as an HBase co-processor.

Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion provides comprehensive ANSI SQL language support including full-functioned data definition (DDL), data manipulation (DML), transaction control (TCL) and database utility support.

Trafodion provides comprehensive and standard SQL data manipulation support including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with language options including join variants, unions, where predicates, aggregations (group by and having), sort ordering, sampling, correlated and nested sub-queries, cursors, and many SQL functions.

Utilities are provided for updating table statistics used by the optimizer for costing (i.e. selectivity/cardinality estimates) plan alternatives, for displaying the chosen SQL execution plan, plan shaping, backup and restoring the database, data loading and unloading, and a command line utility for interfacing with the database engine.

Explicit control statements are provided to allow applications to define transaction boundaries and to abort transactions when warranted, including BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.

Trafodion supports ANSI’s grant/revoke semantics to define user and role privileges in terms of managing and accessing the database objects.

Rationale

The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity, enabling new kinds of big data applications to run on Hadoop.

Current Status

HP released the Trafodion code under the Apache License, Version 2, in June of 2014. Since that time, we have had one major release in January 2015 and one minor release in April 2015. The focus of these releases has been in getting our base functionality, including security, working on top of Apache HBase, as well as improving performance, availability and scalability, and integrating better with HBase.

Meritocracy

We want to build a diverse developer community, based on the Apache Way, around Trafodion. To help developers become contributors, we have documentation on the wiki about the architecture, the source tree structure, and an example enhancement. We plan to publish our project backlog to the community, specifically highlighting areas where developers new to Trafodion may best start contributing, such as extending the database functionality with User Defined Routines (UDRs) and integrating with other Apache projects in the Hadoop ecosystem.

Community

We have already begun building a community but at this time the community consists only of Trafodion developers – all HP employees – and prospective users. We have participated in and hosted HBase Meetups and intend to ramp up our community building efforts.

The Trafodion project has seen interest in China, where HP has conducted proof-of-concepts with multiple companies and expects to see some of its first commercial deployments. To help recruit contributors and users in China, members of the team are translating Trafodion wiki content into Mandarin.

Core Developers

The core developers are very experienced in database and transaction monitor technology, with many having spent more than 20 years working in this space.

Alignment

Apache Trafodion relies on Apache HBase as its storage engine. The development team has collaborated with and gained valuable advice from working with the Apache HBase core developers. Apache Trafodion has federation capabilities as well, and can query Trafodion tables stored in HBase, native HBase tables, and Apache Hive tables.

Known Risks

Orphaned Products

HP Labs and HP-IT have been incubating Trafodion development for almost two years. This is part of HP’s strategy to leverage its investment in database software and bring software to market as open source and is similar to HP’s efforts with OpenStack. Trafodion builds on HP’s equity investment in the Hadoop ecosystem and its efforts to monetize Hadoop through hardware, software, and services. HP wants Trafodion to be successful, as HP will offer a commercially supported distribution of Trafodion.

Inexperience with Open Source

We have been working with open source software in building closed source software for well over two decades. To help transition to doing open source development, the development team received guidance and best practices from HP developers working on OpenStack open source projects, many of whom have experience working on Apache and other open source projects as well. Since releasing Trafodion as an open source project in June of 2014, the committers and contributors have moved forward using open source development processes and tools for bug tracking and design blueprints and Jenkins for continuous integration. As part of the incubation process, we recognize we may need to change some of our development processes/tools and conduct our discussions using Apache email dlists.

Homogenous Developers

Since the initial development of Trafodion has been supported by HP, all of the current developers are HP employees. Through the support of the Apache incubation project, we aim to expand the list of developers and gain contributors from related SQL-on-Hadoop projects and the Apache HBase project. Trafodion developers are experienced with distributed development processes, being primarily based in Palo Alto, CA; Austin, TX; and Shanghai, China. Trafodion is written in C++ and Java.

Reliance on Salaried Developers

Currently all of the developers working on the project are paid by their employer to work on the project. These developers will work on the open source project as well as work on the commercially supported distribution of Trafodion that HP will offer.

Relationship with Other Apache Products

Trafodion is built upon Apache HBase and extends it to support ACID transactions with HBase co-processors for distributed transaction management and recovery. Trafodion envisions future collaborations with the Apache HBase project on performance optimizations, such as in the areas of mixed workload support, High Availability, etc. It also provides transactional support and querying from native HBase tables as well.

Trafodion uses Apache Zookeeper to coordinate and manage the distribution of connection services across the cluster for load-balancing and high availability reconnection purposes in the event a Trafodion process should fail.

Trafodion also envisions working with the Apache Ambari project on enabling better Trafodion manageability. While Ambari focuses on system and component level performance metrics, Trafodion manageability will focus in a complimentary way on database workload monitoring and performance analytics with capabilities more geared towards database administrators.

There are alternative open source projects that are providing SQL-on-Hadoop capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These are more focused on reporting and analytics across data structures supported on HDFS. In comparison to all of these technologies Trafodion provides a very complete implementation of ANSI SQL, one of the most sophisticated optimizers for such workloads, a completely parallel data flow architecture that does not materialize intermediate results unless necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and other capabilities that would take decades to build in these products. On the other hand currently Trafodion is just focused on HBase and querying Hive, whereas Hive and Drill provide access to other data formats in HDFS.

An Excessive Fascination with the Apache Brand

We understand the reputation and value of the Apache brand, and no doubt believe that it will help us attract contributors and users. Our primary goal is to follow a proven, open source development and community building model that will make Trafodion successful and enable better collaboration with other Apache projects in the Hadoop ecosystem. We also understand the rules and guidelines about the use of the Apache brand and intend to follow them.

Documentation

Documentation and technical details on Trafodion can be found at:
http://www.trafodion.org/

Initial Source

The source is available today in a public github repository:
https://github.com/trafodion/trafodion.

Source and Intellectual Property Submission Plan

The source code has already been released under the Apache License, Version 2. The manuals have been released in Adobe PDF format. As part of the submission process, the source for the manuals will be converted from a proprietary DocBook XML format to AsciiDoc.

External Dependencies

Two dependencies do not have Apache compatible licenses and will be addressed as we enter incubation. One dependency is log4cpp, which is licensed under the LGPL. A compatible alternative might be Apache incubator project log4cxx. The other dependency is unixodbc, which is used as the ODBC driver manager. We will look into how Apache Hive manages being able to use this incompatible software and do similar. All other dependencies have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and BSD.

Cryptography

Trafodion does not contain any cryptographic code. It does call cryptographic libraries: OpenSSL for C++ code and Java Cryptography Extension (JCE) for Java code.

Required Resources

Mailing Lists

***@trafodion.incubator.apache.org
***@trafodion.incubator.apache.org ***@trafodion.incubator.apache.org

Git Repository

https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git

Issue Tracking

JIRA: JIRA Trafodion (Trafodion)


Initial Committers and Affiliation

Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com Sandyha Sundaresan, Hewlett-Packard Company, Sandhya.Sundaresan<AT>hp<DOT>com

Sponsors

Champion

Michael Stack, Stack<AT>apache<DOT>org

Nominated Mentors

Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io

We are seeking additional mentors.

Sponsoring Entity

Apache Incubator PMC



---------------------------------------------------------------------
To unsubscribe, e-mail: general-***@incubator.apache.org
For additional commands, e-mail: general-***@incubator.apache.org
Stack
2015-05-18 20:25:32 UTC
Permalink
Unless there are other comments on the proposal, I will move to start a
VOTE on accepting Trafodion into incubator tomorrow.
Thanks,
St.Ack

On Thu, May 14, 2015 at 8:02 PM, Venkat Ranganathan <
Post by Venkat Ranganathan
Glad to see Trafodion submitted as an Apache Incubator project.
Good luck
Venkat
Hi Lars,
Glad to have you on the mentor list!
Dave
-----Original Message-----
Sent: Thursday, May 14, 2015 3:39 PM
Subject: Re: [DISCUSS] Trafodion Incubation Proposal
Bit belated... I'd be happy and honored to be a mentor for Trafodian.
-- Lars
Sent: Friday, May 8, 2015 2:59 PM
Subject: [DISCUSS] Trafodion Incubation Proposal
I would like to start up a discussion on Trafodion joining the ASF as an
incubating project.
Trafodion is a webscale SQL-on-Hadoop solution that enables transactional
or operational workloads on Hadoop, .
https://wiki.apache.org/incubator/TrafodionProposal#preview
The proposal text is also attached to the end of this email.
Trafodion is a rich, storied SQL engine that has recently been ported to
run on HBase and Hadoop. I think it would make for a fine addition to the
Apache family of projects It would be good to hear what others think.
Thank you in advance for giving the proposal a read.
Yours,
St.Ack
Trafodion Apache Incubator Proposal
Abstract
Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.
Proposal
Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop.
* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements, tables and rows
* Performance improvements for OLTP workloads with compile-time and run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential integrity
Hewlett-Packard Company submits this proposal to donate its Apache
License, Version 2.0 open source project known as Trafodion, its source
code, documentation, and web site content to the Apache Software Foundation
in order to build an open source community
Background
Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.
The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.
Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase co-processor.
Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.
Trafodion provides comprehensive and standard SQL data manipulation
support including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax
with language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.
Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.
Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.
Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.
Rationale
The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.
Current Status
HP released the Trafodion code under the Apache License, Version 2, in
June of 2014. Since that time, we have had one major release in January
2015 and one minor release in April 2015. The focus of these releases has
been in getting our base functionality, including security, working on top
of Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.
Meritocracy
We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.
Community
We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.
The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.
Core Developers
The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.
Alignment
Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.
Known Risks
Orphaned Products
HP Labs and HP-IT have been incubating Trafodion development for almost
two years. This is part of HP’s strategy to leverage its investment in
database software and bring software to market as open source and is
similar to HP’s efforts with OpenStack. Trafodion builds on HP’s equity
investment in the Hadoop ecosystem and its efforts to monetize Hadoop
through hardware, software, and services. HP wants Trafodion to be
successful, as HP will offer a commercially supported distribution of
Trafodion.
Inexperience with Open Source
We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.
Homogenous Developers
Since the initial development of Trafodion has been supported by HP, all
of the current developers are HP employees. Through the support of the
Apache incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.
Reliance on Salaried Developers
Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.
Relationship with Other Apache Products
Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.
Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.
Trafodion also envisions working with the Apache Ambari project on
enabling better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.
There are alternative open source projects that are providing
SQL-on-Hadoop capabilities, such as Apache Hive, Apache Drill, and Apache
Phoenix. These are more focused on reporting and analytics across data
structures supported on HDFS. In comparison to all of these technologies
Trafodion provides a very complete implementation of ANSI SQL, one of the
most sophisticated optimizers for such workloads, a completely parallel
data flow architecture that does not materialize intermediate results
unless necessary, full ACID transactional support, ANSI GRANT/REVOKE
security, and other capabilities that would take decades to build in these
products. On the other hand currently Trafodion is just focused on HBase
and querying Hive, whereas Hive and Drill provide access to other data
formats in HDFS.
An Excessive Fascination with the Apache Brand
We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.
Documentation
http://www.trafodion.org/
Initial Source
https://github.com/trafodion/trafodion.
Source and Intellectual Property Submission Plan
The source code has already been released under the Apache License,
Version 2. The manuals have been released in Adobe PDF format. As part of
the submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.
External Dependencies
Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.
Cryptography
Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.
Required Resources
Mailing Lists
Git Repository
https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git
Issue Tracking
JIRA: JIRA Trafodion (Trafodion)
Initial Committers and Affiliation
Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com Matt
Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com Tharak Capirala,
Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com Alice Chen,
Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com John DeRoo,
Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com Roberta Marton,
Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com Amanda Moran,
Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com Suresh Subbiah,
Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com Sandyha Sundaresan,
Hewlett-Packard Company, Sandhya.Sundaresan<AT>hp<DOT>com
Sponsors
Champion
Michael Stack, Stack<AT>apache<DOT>org
Nominated Mentors
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io
We are seeking additional mentors.
Sponsoring Entity
Apache Incubator PMC
---------------------------------------------------------------------
Roman Shaposhnik
2015-05-18 21:49:28 UTC
Permalink
Post by Stack
Unless there are other comments on the proposal, I will move to start a
VOTE on accepting Trafodion into incubator tomorrow.
Given that there were a few folks volunteering as mentors (thanks!!!)
should we have the updated version of the proposal posted on this
thread before opening up a VOTE?

Thanks,
Roman.

---------------------------------------------------------------------
To unsubscribe, e-mail: general-***@incubator.apache.org
For additional commands, e-mail: general-***@incubator.apache.org
Stack
2015-05-18 22:17:46 UTC
Permalink
Post by Roman Shaposhnik
Post by Stack
Unless there are other comments on the proposal, I will move to start a
VOTE on accepting Trafodion into incubator tomorrow.
Given that there were a few folks volunteering as mentors (thanks!!!)
should we have the updated version of the proposal posted on this
thread before opening up a VOTE?
Thanks,
Roman.
---------------------------------------------------------------------
Sounds good. I added volunteers Andy, Devaraj, Lars, and Enis. I left off
Konstantin since he seems well-loaded mentoring other projects for now. I
also removed the sentence soliciting mentors.

Will put up VOTE tomorrow.

St.Ack


Trafodion Apache Incubator Proposal

Abstract

Trafodion is a webscale SQL-on-Hadoop solution enabling transactional or
operational workloads on Hadoop.

Proposal

Apache Trafodion builds on the scalability, elasticity, and flexibility of
Hadoop. Trafodion extends Hadoop to provide guaranteed transactional
integrity, enabling new kinds of big data applications to run on Hadoop. Key
features of Apache Trafodion include:

* Full-functioned ANSI SQL language support
* JDBC/ODBC connectivity for Linux/Windows clients
* Distributed ACID transaction protection across multiple statements,
tables and rows
* Performance improvements for OLTP workloads with compile-time and
run-time optimizations
* Support for large data sets using a parallel-aware query optimizer
* ANSI SQL security and data integrity constraints including referential
integrity

Hewlett-Packard Company submits this proposal to donate its Apache License,
Version 2.0 open source project known as Trafodion, its source code,
documentation, and web site content to the Apache Software Foundation in
order to build an open source community

Background

Trafodion is an open source project sponsored by HP, incubated at HP Labs
and HP-IT, to develop an enterprise-class SQL-on-Hadoop solution targeting
big data transactional or operational workloads. HP publically announced
the open source project and uploaded the source code to GitHub in June 2014.

The SQL compiler, optimizer and executor components of Trafodion have a
rich heritage. Under development since 1993, they were released as
commercial closed source software in various flavors such as HP NonStop
SQL/MX and HP Neoview. NonStop SQL/MX was designed for online transaction
processing on HP’s NonStop (formerly Tandem) fault-tolerant servers and is
known for its high availability, scalability, and performance. Hundreds of
companies and thousands of servers are running mission-critical
applications today on NonStop SQL/MX. In addition, much of these components
today are running internal to HP as the core of its Enterprise Data
Warehouse (EDW), managing over a PB of data.

Starting in 2013, the software was modified to run on HBase and a new
distributed transaction manager was written to run as an HBase co-processor.

Unlike most NOSQL and other SQL-on-Hadoop open source projects, Trafodion
provides comprehensive ANSI SQL language support including full-functioned
data definition (DDL), data manipulation (DML), transaction control (TCL)
and database utility support.

Trafodion provides comprehensive and standard SQL data manipulation support
including SELECT, INSERT, UPDATE, DELETE, and UPSERT/MERGE syntax with
language options including join variants, unions, where predicates,
aggregations (group by and having), sort ordering, sampling, correlated and
nested sub-queries, cursors, and many SQL functions.

Utilities are provided for updating table statistics used by the optimizer
for costing (i.e. selectivity/cardinality estimates) plan alternatives, for
displaying the chosen SQL execution plan, plan shaping, backup and
restoring the database, data loading and unloading, and a command line
utility for interfacing with the database engine.

Explicit control statements are provided to allow applications to define
transaction boundaries and to abort transactions when warranted, including
BEGIN WORK, COMMIT WORK, ROLLBACK WORK and SET TRANSACTION.

Trafodion supports ANSI’s grant/revoke semantics to define user and role
privileges in terms of managing and accessing the database objects.

Rationale

The name “Trafodion” (the Welsh word for transactions, pronounced
“Tra-vod-eee-on”) was chosen specifically to emphasize the differentiation
that Trafodion provides in closing a critical gap in the Hadoop ecosystem.
Trafodion builds on the scalability, elasticity, and flexibility of Hadoop.
Trafodion extends Hadoop to provide guaranteed transactional integrity,
enabling new kinds of big data applications to run on Hadoop.

Current Status

HP released the Trafodion code under the Apache License, Version 2, in June
of 2014. Since that time, we have had one major release in January 2015 and
one minor release in April 2015. The focus of these releases has been in
getting our base functionality, including security, working on top of
Apache HBase, as well as improving performance, availability and
scalability, and integrating better with HBase.

Meritocracy

We want to build a diverse developer community, based on the Apache Way,
around Trafodion. To help developers become contributors, we have
documentation on the wiki about the architecture, the source tree
structure, and an example enhancement. We plan to publish our project
backlog to the community, specifically highlighting areas where developers
new to Trafodion may best start contributing, such as extending the
database functionality with User Defined Routines (UDRs) and integrating
with other Apache projects in the Hadoop ecosystem.

Community

We have already begun building a community but at this time the community
consists only of Trafodion developers – all HP employees – and prospective
users. We have participated in and hosted HBase Meetups and intend to ramp
up our community building efforts.

The Trafodion project has seen interest in China, where HP has conducted
proof-of-concepts with multiple companies and expects to see some of its
first commercial deployments. To help recruit contributors and users in
China, members of the team are translating Trafodion wiki content into
Mandarin.

Core Developers

The core developers are very experienced in database and transaction
monitor technology, with many having spent more than 20 years working in
this space.

Alignment

Apache Trafodion relies on Apache HBase as its storage engine. The
development team has collaborated with and gained valuable advice from
working with the Apache HBase core developers. Apache Trafodion has
federation capabilities as well, and can query Trafodion tables stored in
HBase, native HBase tables, and Apache Hive tables.

Known Risks

Orphaned Products

HP Labs and HP-IT have been incubating Trafodion development for almost two
years. This is part of HP’s strategy to leverage its investment in database
software and bring software to market as open source and is similar to HP’s
efforts with OpenStack. Trafodion builds on HP’s equity investment in the
Hadoop ecosystem and its efforts to monetize Hadoop through hardware,
software, and services. HP wants Trafodion to be successful, as HP will
offer a commercially supported distribution of Trafodion.

Inexperience with Open Source

We have been working with open source software in building closed source
software for well over two decades. To help transition to doing open source
development, the development team received guidance and best practices from
HP developers working on OpenStack open source projects, many of whom have
experience working on Apache and other open source projects as well. Since
releasing Trafodion as an open source project in June of 2014, the
committers and contributors have moved forward using open source
development processes and tools for bug tracking and design blueprints and
Jenkins for continuous integration. As part of the incubation process, we
recognize we may need to change some of our development processes/tools and
conduct our discussions using Apache email dlists.

Homogenous Developers

Since the initial development of Trafodion has been supported by HP, all of
the current developers are HP employees. Through the support of the Apache
incubation project, we aim to expand the list of developers and gain
contributors from related SQL-on-Hadoop projects and the Apache HBase
project. Trafodion developers are experienced with distributed development
processes, being primarily based in Palo Alto, CA; Austin, TX; and
Shanghai, China. Trafodion is written in C++ and Java.

Reliance on Salaried Developers

Currently all of the developers working on the project are paid by their
employer to work on the project. These developers will work on the open
source project as well as work on the commercially supported distribution
of Trafodion that HP will offer.

Relationship with Other Apache Products

Trafodion is built upon Apache HBase and extends it to support ACID
transactions with HBase co-processors for distributed transaction
management and recovery. Trafodion envisions future collaborations with the
Apache HBase project on performance optimizations, such as in the areas of
mixed workload support, High Availability, etc. It also provides
transactional support and querying from native HBase tables as well.

Trafodion uses Apache Zookeeper to coordinate and manage the distribution
of connection services across the cluster for load-balancing and high
availability reconnection purposes in the event a Trafodion process should
fail.

Trafodion also envisions working with the Apache Ambari project on enabling
better Trafodion manageability. While Ambari focuses on system and
component level performance metrics, Trafodion manageability will focus in
a complimentary way on database workload monitoring and performance
analytics with capabilities more geared towards database administrators.

There are alternative open source projects that are providing SQL-on-Hadoop
capabilities, such as Apache Hive, Apache Drill, and Apache Phoenix. These
are more focused on reporting and analytics across data structures
supported on HDFS. In comparison to all of these technologies Trafodion
provides a very complete implementation of ANSI SQL, one of the most
sophisticated optimizers for such workloads, a completely parallel data
flow architecture that does not materialize intermediate results unless
necessary, full ACID transactional support, ANSI GRANT/REVOKE security, and
other capabilities that would take decades to build in these products. On
the other hand currently Trafodion is just focused on HBase and querying
Hive, whereas Hive and Drill provide access to other data formats in HDFS.

An Excessive Fascination with the Apache Brand

We understand the reputation and value of the Apache brand, and no doubt
believe that it will help us attract contributors and users. Our primary
goal is to follow a proven, open source development and community building
model that will make Trafodion successful and enable better collaboration
with other Apache projects in the Hadoop ecosystem. We also understand the
rules and guidelines about the use of the Apache brand and intend to follow
them.

Documentation

Documentation and technical details on Trafodion can be found at:
http://www.trafodion.org/

Initial Source

The source is available today in a public github repository:
https://github.com/trafodion/trafodion.

Source and Intellectual Property Submission Plan

The source code has already been released under the Apache License, Version
2. The manuals have been released in Adobe PDF format. As part of the
submission process, the source for the manuals will be converted from a
proprietary DocBook XML format to AsciiDoc.

External Dependencies

Two dependencies do not have Apache compatible licenses and will be
addressed as we enter incubation. One dependency is log4cpp, which is
licensed under the LGPL. A compatible alternative might be Apache incubator
project log4cxx. The other dependency is unixodbc, which is used as the
ODBC driver manager. We will look into how Apache Hive manages being able
to use this incompatible software and do similar. All other dependencies
have Apache compatible licenses, including Apache 2.0, MIT/X11, MIT, and
BSD.

Cryptography

Trafodion does not contain any cryptographic code. It does call
cryptographic libraries: OpenSSL for C++ code and Java Cryptography
Extension (JCE) for Java code.

Required Resources

Mailing Lists

***@trafodion.incubator.apache.org
***@trafodion.incubator.apache.org ***@trafodion.incubator.apache.org

Git Repository

https://git-wip-us.apache.org/repos/afs/incubator-trafodion.git

Issue Tracking

JIRA: JIRA Trafodion (Trafodion)


Initial Committers and Affiliation

Dave Birdsall, Hewlett-Packard Company, Dave.Birdsall<AT>hp<DOT>com
Matt Brown, Hewlett-Packard Company, mattbrown<AT>hp<DOT>com
Tharak Capirala, Hewlett-Packard Company, Tharak.Capirala<AT>hp<DOT>com
Alice Chen, Hewlett-Packard Company, Alice.Chen<AT>hp<DOT>com
John DeRoo, Hewlett-Packard Company, John.Deroo<AT>hp<DOT>com
Roberta Marton, Hewlett-Packard Company, Roberta.Marton<AT>hp<DOT>com
Amanda Moran, Hewlett-Packard Company, Amanda.Kay.Moran<AT>hp<DOT>com
Suresh Subbiah, Hewlett-Packard Company, Suresh.Subbiah<AT>hp<DOT>com
Sandyha Sundaresan, Hewlett-Packard Company,
Sandhya.Sundaresan<AT>hp<DOT>com

Sponsors

Champion

Michael Stack, Stack<AT>apache<DOT>org

Nominated Mentors

Andrew Purtell apurtell<AT>apache<DOT>org
Devaraj Das, ddas<AT>apache<DOT>or
Enis Söztutar, Enis<AT>apache<DOT>org
Lars Hofhansl, larsh<AT>apache<DOT>org
Michael Stack, Stack<AT>apache<DOT>org
Roman Shaposhnik, rshaposhnik<AT>pivotal<DOT>io

Sponsoring Entity

Apache Incubator PMC
Ted Yu
2015-05-19 21:36:31 UTC
Permalink
Proposal looks good.

Please include me as mentor.

Thanks

Continue reading on narkive:
Loading...