2018-04-24

Congratulations to Oracle on MySQL 8.0


Last week, Oracle announced the general availability of MySQL 8.0. This is good news for database users, as it means Oracle is still developing MySQL.


I decide to celebrate the event by doing a quick test of MySQL 8.0. Here follows a step-by-step description of my first experience with MySQL 8.0.
Note that I did the following without reading the release notes, as is what I have done with every MySQL / MariaDB release up to date; In this case it was not the right thing to do.

I pulled MySQL 8.0 from ghit@github.com:mysql/mysql-server.git
I was pleasantly surprised that 'cmake . ; make' worked without without any compiler warnings! I even checked the used compiler options and noticed that MySQL was compiled with -Wall + several other warning flags. Good job MySQL team!

I did have a little trouble finding the mysqld binary as Oracle had moved it to 'runtime_output_directory'; Unexpected, but no big thing.

Now it's was time to install MySQL 8.0.

I did know that MySQL 8.0 has removed mysql_install_db, so I had to use the mysqld binary directly to install the default databases:
(I have specified datadir=/my/data3 in the /tmp/my.cnf file)

> cd runtime_output_directory
> mkdir /my/data3
> ./mysqld --defaults-file=/tmp/my.cnf --install

2018-04-22T12:38:18.332967Z 1 [ERROR] [MY-011011] [Server] Failed to find valid data directory.
2018-04-22T12:38:18.333109Z 0 [ERROR] [MY-010020] [Server] Data Dictionary initialization failed.
2018-04-22T12:38:18.333135Z 0 [ERROR] [MY-010119] [Server] Aborting

A quick look in mysqld --help --verbose output showed that the right command option is --initialize. My bad, lets try again,

> ./mysqld --defaults-file=/tmp/my.cnf --initialize

2018-04-22T12:39:31.910509Z 0 [ERROR] [MY-010457] [Server] --initialize specified but the data directory has files in it. Aborting.
2018-04-22T12:39:31.910578Z 0 [ERROR] [MY-010119] [Server] Aborting

Now I used the right options, but still didn't work.
I took a quick look around:

> ls /my/data3/
binlog.index

So even if the mysqld noticed that the data3 directory was wrong, it still wrote things into it.  This even if I didn't have --log-binlog enabled in the my.cnf file. Strange, but easy to fix:

> rm /my/data3/binlog.index
> ./mysqld --defaults-file=/tmp/my.cnf --initialize

2018-04-22T12:40:45.633637Z 0 [ERROR] [MY-011071] [Server] unknown variable 'max-tmp-tables=100'
2018-04-22T12:40:45.633657Z 0 [Warning] [MY-010952] [Server] The privilege system failed to initialize correctly. If you have upgraded your server, make sure you're executing mysql_upgrade to correct the issue.
2018-04-22T12:40:45.633663Z 0 [ERROR] [MY-010119] [Server] Aborting

The warning about the privilege system confused me a bit, but I ignored it for the time being and removed from my configuration files the variables that MySQL 8.0 doesn't support anymore. I couldn't find a list of the removed variables anywhere so this was done with the trial and error method.

> ./mysqld --defaults-file=/tmp/my.cnf

2018-04-22T12:42:56.626583Z 0 [ERROR] [MY-010735] [Server] Can't open the mysql.plugin table. Please run mysql_upgrade to create it.
2018-04-22T12:42:56.827685Z 0 [Warning] [MY-010015] [Repl] Gtid table is not ready to be used. Table 'mysql.gtid_executed' cannot be opened.
2018-04-22T12:42:56.838501Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2018-04-22T12:42:56.848375Z 0 [Warning] [MY-010441] [Server] Failed to open optimizer cost constant tables
2018-04-22T12:42:56.848863Z 0 [ERROR] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-001146 - Table 'mysql.component' doesn't exist
2018-04-22T12:42:56.848916Z 0 [Warning] [MY-013129] [Server] A message intended for a client cannot be sent there as no client-session is attached. Therefore, we're sending the information to the error-log instead: MY-003543 - The mysql.component table is missing or has an incorrect definition.
....
2018-04-22T12:42:56.854141Z 0 [System] [MY-010931] [Server] /home/my/mysql-8.0/runtime_output_directory/mysqld: ready for connections. Version: '8.0.11' socket: '/tmp/mysql.sock' port: 3306 Source distribution.

I figured out that if there is a single wrong variable in the configuration file, running mysqld --initialize will leave the database in an inconsistent state. NOT GOOD! I am happy I didn't try this in a production system!

Time to start over from the beginning:

> rm -r /my/data3/*
> ./mysqld --defaults-file=/tmp/my.cnf --initialize

2018-04-22T12:44:45.548960Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: px)NaaSp?6um
2018-04-22T12:44:51.221751Z 0 [System] [MY-013170] [Server] /home/my/mysql-8.0/runtime_output_directory/mysqld (mysqld 8.0.11) initializing of server has completed

Success!

I wonder why the temporary password is so complex; It could easily have been something that one could easily remember without decreasing security, it's temporary after all. No big deal, one can always paste it from the logs. (Side note: MariaDB uses socket authentication on many system and thus doesn't need temporary installation passwords).

Now lets start the MySQL server for real to do some testing:

> ./mysqld --defaults-file=/tmp/my.cnf

2018-04-22T12:45:43.683484Z 0 [System] [MY-010931] [Server] /home/my/mysql-8.0/runtime_output_directory/mysqld: ready for connections. Version: '8.0.11' socket: '/tmp/mysql.sock' port: 3306 Source distribution.

And the lets start the client:

> ./client/mysql --socket=/tmp/mysql.sock --user=root --password="px)NaaSp?6um"
ERROR 2059 (HY000): Plugin caching_sha2_password could not be loaded: /usr/local/mysql/lib/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory

Apparently MySQL 8.0 doesn't work with old MySQL / MariaDB clients by default :(

I was testing this in a system with MariaDB installed, like all modern Linux system today, and didn't want to use the MySQL clients or libraries.

I decided to try to fix this by changing the authentication to the native (original) MySQL authentication method.

> mysqld --skip-grant-tables

> ./client/mysql --socket=/tmp/mysql.sock --user=root
ERROR 1045 (28000): Access denied for user 'root'@'localhost' (using password: NO)

Apparently --skip-grant-tables is not good enough anymore. Let's try again with:

> mysqld --skip-grant-tables --default_authentication_plugin=mysql_native_password

> ./client/mysql --socket=/tmp/mysql.sock --user=root mysql
Welcome to the MariaDB monitor. Commands end with ; or \g.
Your MySQL connection id is 7
Server version: 8.0.11 Source distribution

Great, we are getting somewhere, now lets fix "root"  to work with the old authenticaion:

MySQL [mysql]> update mysql.user set plugin="mysql_native_password",authentication_string=password("test") where user="root";
ERROR 1064 (42000): You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near '("test") where user="root"' at line 1

A quick look in the MySQL 8.0 release notes told me that the PASSWORD() function is removed in 8.0. Why???? I don't know how one in MySQL 8.0 is supposed to generate passwords compatible with old installations of MySQL. One could of course start an old MySQL or MariaDB version, execute the password() function and copy the result.

I decided to fix this the easy way and use an empty password:

(Update:: I later discovered that the right way would have been to use: FLUSH PRIVILEGES;  ALTER USER' root'@'localhost' identified by 'test'  ; I however dislike this syntax as it has the password in clear text which is easy to grab and the command can't be used to easily update the mysql.user table. One must also disable the --skip-grant mode to do use this)

MySQL [mysql]> update mysql.user set plugin="mysql_native_password",authentication_string="" where user="root";
Query OK, 1 row affected (0.077 sec)
Rows matched: 1 Changed: 1 Warnings: 0
 
I restarted mysqld:
> mysqld --default_authentication_plugin=mysql_native_password

> ./client/mysql --user=root --password="" mysql
ERROR 1862 (HY000): Your password has expired. To log in you must change it using a client that supports expired passwords.

Ouch, forgot that. Lets try again:

> mysqld --skip-grant-tables --default_authentication_plugin=mysql_native_password

> ./client/mysql --user=root --password="" mysql
MySQL [mysql]> update mysql.user set password_expired="N" where user="root";

Now restart and test worked:

> ./mysqld --default_authentication_plugin=mysql_native_password

>./client/mysql --user=root --password="" mysql

Finally I had a working account that I can use to create other users!

When looking at mysqld --help --verbose again. I noticed the option:

--initialize-insecure
Create the default database and exit. Create a super user
with empty password.

I decided to check if this would have made things easier:

> rm -r /my/data3/*
> ./mysqld --defaults-file=/tmp/my.cnf --initialize-insecure


2018-04-22T13:18:06.629548Z 5 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.

Hm. Don't understand the warning as--initialize-insecure is not an option that one would use more than one time and thus nothing one would 'switch off'.

> ./mysqld --defaults-file=/tmp/my.cnf

> ./client/mysql --user=root --password="" mysql
ERROR 2059 (HY000): Plugin caching_sha2_password could not be loaded: /usr/local/mysql/lib/plugin/caching_sha2_password.so: cannot open shared object file: No such file or directory

Back to the beginning :(

To get things to work with old clients, one has to initialize the database with:
> ./mysqld --defaults-file=/tmp/my.cnf --initialize-insecure --default_authentication_plugin=mysql_native_password

Now I finally had MySQL 8.0 up and running and thought I would take it up for a spin by running the "standard" MySQL/MariaDB sql-bench test suite. This was removed in MySQL 5.7, but as I happened to have MariaDB 10.3 installed, I decided to run it from there.

sql-bench is a single threaded benchmark that measures the "raw" speed for some common operations. It gives you the 'maximum' performance for a single query. Its different from other benchmarks that measures the maximum throughput when you have a lot of users, but sql-bench still tells you a lot about what kind of performance to expect from the database.

I tried first to be clever and create the "test" database, that I needed for sql-bench, with
> mkdir /my/data3/test

but when I tried to run the benchmark, MySQL 8.0 complained that the test database didn't exist.

MySQL 8.0 has gone away from the original concept of MySQL where the user can easily
create directories and copy databases into the database directory. This may have serious
implication for anyone doing backup of databases and/or trying to restore a backup with normal OS commands.

I created the 'test' database with mysqladmin and then tried to run sql-bench:

> ./run-all-tests --user=root


The first run failed in test-ATIS:

Can't execute command 'create table class_of_service (class_code char(2) NOT NULL,rank tinyint(2) NOT NULL,class_description char(80) NOT NULL,PRIMARY KEY (class_code))'
Error: You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'rank tinyint(2) NOT NULL,class_description char(80) NOT NULL,PRIMARY KEY (class_' at line 1

This happened because 'rank' is now a reserved word in MySQL 8.0. This is also reserved in ANSI SQL, but I don't know of any other database that has failed to run test-ATIS before. I have in the past run it against Oracle, PostgreSQL, Mimer, MSSQL etc without any problems.

MariaDB also has 'rank' as a keyword in 10.2 and 10.3 but one can still use it as an identifier.

I fixed test-ATIS and then managed to run all tests on MySQL 8.0.

I did run the test both with MySQL 8.0 and MariaDB 10.3 with the InnoDB storage engine and by having identical values for all InnoDB variables, table-definition-cache and table-open-cache. I turned off performance schema for both databases. All test are run with a user without an empty password (to keep things comparable and because it's was too complex to generate a password in MySQL 8.0)

The result are as follows
Results per test in seconds:

Operation         |MariaDB|MySQL-8|

-----------------------------------
ATIS              | 153.00| 228.00|
alter-table       |  92.00| 792.00|
big-tables        | 990.00|2079.00|
connect           | 186.00| 227.00|
create            | 575.00|4465.00|
insert            |4552.00|8458.00|
select            | 333.00| 412.00|
table-elimination |1900.00|3916.00|
wisconsin         | 272.00| 590.00|
-----------------------------------

This is of course just a first view of the performance of MySQL 8.0 in a single user environment. Some reflections about the results:

  • Alter-table test is slower (as expected) in 8.0 as some of the alter tests benefits of the instant add column in MariaDB 10.3.
  • connect test is also better for MariaDB as we put a lot of efforts to speed this up in MariaDB 10.2
  • table-elimination shows an optimization in MariaDB for the  Anchor table model, which MySQL doesn't have.
  • CREATE and DROP TABLE is almost 8 times slower in MySQL 8.0 than in MariaDB 10.3. I assume this is the cost of 'atomic DDL'. This may also cause performance problems for any thread using the data dictionary when another thread is creating/dropping tables.
  • When looking at the individual test results, MySQL 8.0 was slower in almost every test, in many significantly slower.
  • The only test where MySQL was faster was "update_with_key_prefix". I checked this and noticed that there was a bug in the test and the columns was updated to it's original value (which should be instant with any storage engine). This is an old bug that MySQL has found and fixed and that we have not been aware of in the test or in MariaDB.
  • While writing this, I noticed that MySQL 8.0 is now using utf8mb4 as the default character set instead of latin1. This may affect some of the benchmarks slightly (not much as most tests works with numbers and Oracle claims that utf8mb4 is only 20% slower than latin1), but needs to be verified.
  • Oracle claims that MySQL 8.0 is much faster on multi user benchmarks. The above test indicates that they may have done this by sacrificing single user performance.
  •  We need to do more and many different benchmarks to better understand exactly what is going on. Stay tuned!


Short summary of my first run with MySQL 8.0:
  • Using the new caching_sha2_password authentication as default for new installation is likely to cause a lot of problems for users. No old application will be able to use MySQL 8.0, installed with default options, without moving to MySQL's client libraries. While working on this blog I saw MySQL users complain on IRC that not even MySQL Worklog can authenticate with MySQL 8.0. This is the first time in MySQL's history where such an incompatible change has ever been done!
  • Atomic DDL is a good thing (We plan to have this in MariaDB 10.4), but it should not have such a drastic impact on performance. I am also a bit skeptical of MySQL 8.0 having just one copy of the data dictionary as if this gets corrupted you will lose all your data. (Single point of failure)
  • MySQL 8.0 has several new reserved words and has removed a lot of variables, which makes upgrades hard. Before upgrading to MySQL 8.0 one has to check all one's databases and applications to ensure that there are no conflicts.
  • As my test above shows, if you have a single deprecated variable in your configuration files, the installation of MySQL will abort and can leave the database in inconsistent state. I did of course my tests by installing into an empty data dictionary, but one can assume that some of the problems may also happen when upgrading an old installation.



Conclusions:
In many ways, MySQL 8.0 has caught up with some earlier versions of MariaDB. For instance, in MariaDB 10.0, we introduced roles (four years ago). In MariaDB 10.1, we introduced encrypted redo/undo logs (three years ago). In MariaDB 10.2, we introduced window functions and CTEs (a year ago). However, some catch-up of MariaDB Server 10.2 features still remains for MySQL (such as check constraints, binlog compression, and log-based rollback).

MySQL 8.0 has a few new interesting features (mostly Atomic DDL and JSON TABLE functions), but at the same time MySQL has strayed away from some of the fundamental corner stone principles of MySQL:

From the start of the first version of MySQL in 1995, all development has been focused around 3 core principles:
  • Ease of use
  • Performance
  • Stability

With MySQL 8.0, Oracle has sacrifices 2 of 3 of these.

In addition (as part of ease of use), while I was working on MySQL, we did our best to ensure that the following should hold:

  • Upgrades should be trivial
  • Things should be kept compatible, if possible (don't remove features/options/functions that are used)
  • Minimize reserved words, don't remove server variables
  • One should be able to use normal OS commands to create and drop databases, copy and move tables around within the same system or between different systems. With 8.0 and data dictionary taking backups of specific tables will be hard, even if the server is not running.
  • mysqldump should always be usable backups and to move to new releases
  • Old clients and application should be able to use 'any' MySQL server version unchanged. (Some Oracle client libraries, like C++, by default only supports the new X protocol and can thus not be used with older MySQL or any MariaDB version)

We plan to add a data dictionary to MariaDB 10.4 or MariaDB 10.5, but in a way to not sacrifice any of the above principles!

The competition between MySQL and MariaDB is not just about a tactical arms race on features. It’s about design philosophy, or strategic vision, if you will.

This shows in two main ways: our respective view of the Storage Engine structure, and of the top-level direction of the roadmap.

On the Storage Engine side, MySQL is converging on InnoDB, even for clustering and partitioning. In doing so, they are abandoning the advantages of multiple ways of storing data. By contrast, MariaDB sees lots of value in the Storage Engine architecture: MariaDB Server 10.3 will see the general availability of MyRocks (for write-intensive workloads) and Spider (for scalable workloads). On top of that, we have ColumnStore for analytical workloads. One can use the CONNECT engine to join with other databases. The use of different storage engines for different workloads and different hardware is a competitive differentiator, now more than ever.

On the roadmap side, MySQL is carefully steering clear of features that close the gap between MySQL and Oracle. MariaDB has no such constraints. With MariaDB 10.3, we are introducing PL/SQL compatibility (Oracle’s stored procedures) and AS OF (built-in system versioned tables with point-in-time querying). For both of those features, MariaDB is the first Open Source database doing so. I don't except Oracle to provide any of the above features in MySQL!

Also on the roadmap side, MySQL is not working with the ecosystem in extending the functionality. In 2017, MariaDB accepted more code contributions in one year, than MySQL has done during its entire lifetime, and the rate is increasing!

I am sure that the experience I had with testing MySQL 8.0 would have been significantly better if MySQL would have an open development model where the community could easily participate in developing and testing MySQL continuously. Most of the confusing error messages and strange behavior would have been found and fixed long before the GA release.


Before upgrading to MySQL 8.0 please read https://dev.mysql.com/doc/refman/8.0/en/upgrading-from-previous-series.html to see what problems you can run into! Don't expect that old installations or applications will work out of the box without testing as a lot of features and options has been removed (query cache, partition of myisam tables etc)! You probably also have to revise your backup methods, especially if you want to ever restore just a few tables. (With 8.0, I don't know how this can be easily done).

According to the MySQL 8.0 release notes, one can't use mysqldump to copy a database to MySQL 8.0. One has to first to move to a MySQL 5.7 GA version (with mysqldump, as recommended by Oracle) and then to MySQL 8.0 with in-place update. I assume this means that all old mysqldump backups are useless for MySQL 8.0?

MySQL 8.0 seams to be a one way street to an unknown future. Up to MySQL 5.7 it has been trivial to move to MariaDB and one could always move back to MySQL with mysqldump. All MySQL client libraries has worked with MariaDB and all MariaDB client libraries has worked with MySQL. With MySQL 8.0 this has changed in the wrong direction.

As long as you are using MySQL 5.7 and below you have choices for your future, after MySQL 8.0 you have very little choice. But don't despair, as MariaDB will always be able to load a mysqldump file and it's very easy to upgrade your old MySQL installation to MariaDB :)

I wish you good luck to try MySQL 8.0 (and also the upcoming MariaDB 10.3)!

2017-05-24

MariaDB 10.2 GA released with several advanced features

MariaDB 10.2.6 GA is now released. It's a release where we have concentrated on adding new advanced features to MariaDB

The most noteworthy ones are:
  • Windows Functions gives you the ability to do advanced calculation over a sliding window.
  • Common table expressions allows you to do more complex SQL statements without having to do explicit temporary tables.
  • We finally have a DEFAULT clause that can take expressions and also CHECK CONSTRAINT.
  • Multiple triggers for the same event. This is important for anyone trying to use tools, like pt-online-schema-change, which requires multiple triggers for the same table.
  • A new storage engine, MyRocks, that gives you high compression of your data without sacrificing speed. It has been developed in cooperation with Facebook and MariaDB to allow you to handle more data with less resources.
  • flashback, a feature that can rollback instances/databases/tables to an old snapshot. The version in MariaDB 10.2 is DML only. In MariaDB 10.3 we will also allow rollback over DML (like DROP TABLE).
  • Compression of events in the binary log.
  • JSON functions added. In 10.2.7 we will also add support for CREATE TABLE ... (a JSON).
A few smaller but still noteworthy new features:
  • Connection setup was made faster by moving creation of THD to a new thread. This, in addition with better thread caching, can give a connection speedup for up to 85 % in some cases.
  • Table cache can automatically partition itself as needed to reduce the contention.
  • NO PAD collations, which means that end space are significant in comparisons.
  • InnoDB is now the default storage engine. Until MariaDB 10.1, MariaDB used the XtraDB storage engine as default. XtraDB in 10.2 is not up to date with the latest features of InnoDB and cannot be used. The main reason for this change is that most of the important features of XtraDB are nowadays implemented in InnoDB . As the MariaDB team is doing a lot more InnoDB development than ever before, we can't anymore manage updating two almost identical engines. The InnoDB version in MariaDB contains the best features of MySQL InnoDB and XtraDB and a lot more. As the InnoDB on disk format is identical to XtraDB's this will not cause any problems when upgrading to MariaDB 10.2
  • The old GPL client library is gone; now MariaDB Server comes with the LGPL Connector/C client library.

There are a lot of other new features, performance enhancements and variables in MariaDB 10.2 for you to explore!

I am happy to see that a lot of the new features have come from the MariadB community! (Note to myself; This list doesn't include all contributors to MariadB 10.2, needs to be update.)

Thanks a lot to everyone that has contributed to MariaDB!

2017-04-19

MariaDB 10.3-alpha released

While most of the MariaDB developers have been working hard on getting MariaDB 10.2 out as GA, a small team, including me, has been working on the next release, MariaDB 10.3.

The theme of MariaDB 10.2 is complex operations, like window functions, common table expressions, JSON functions, the theme of MariaDB 10.3 is compatibility.

Compatibility refers to functionality that exist in other databases but have been missing in MariaDB:
In MariaDB 10.2 ORACLE mode was limited to removing MariaDB specific options in SHOW CREATE TABLE, SHOW CREATE VIEW and setting SQL_MODE to "PIPES_AS_CONCAT, ANSI_QUOTES, IGNORE_SPACE, ORACLE, NO_KEY_OPTIONS, NO_TABLE_OPTIONS, NO_FIELD_OPTIONS, NO_AUTO_CREATE_USER".

In MariaDB 10.3, SQL_MODE=ORACLE mode allows MariaDB to understand a large subset of Oracle's PL/SQL language. The documentation for what is supported is still lacking, but the interested can find what is supported in the test suite in the "mysql-test/suite/compat/oracle" directory.

If things go as planned, the features we will add to 10.3 prior to beta are:
Most of the above features are already close to be ready (to be added in future Alphas), so I expect that it willl not take many months before we can make a first MariaDB 10.3 beta!

This is in line what was discussed on the MariaDB developer conference in New York one week ago, where most attendees wanted to see new MariaDB releases more often.

MariaDB 10.3 can be downloaded here

Happy testing!

2016-08-16

Applying the Business Source Licensing (BSL)

I believe that Open Source is one of the best ways to develop software. However, as I have written in blogs before, the Open Source model presents challenges to creating a software company that has the needed resources to continually invest in product development and innovation.

One reason for this is a lack of understanding of the costs associated with developing and extending software. As one example of what I regard to be unrealistic user expectations, here is a statement from a large software company when I asked them to support MariaDB development with financial support:

As you may remember, we’re a fairly traditional and conservative company. A donation from us would require feature work in exchange for the donation. Unfortunately, I cannot think of a feature that I would want developed that we would be willing to pay for this year.”

This thinking is flawed on many fronts -- a new feature can take more than a year to develop! It also shows that the company saw that features create value they would invest in, but was not willing to pay for features that had already been developed and was not prepared to invest into keeping alive a product they depend upon. They also don't trust the development team with the ability to independently define new features that would bring value. Without that investment, a technology company cannot invest in ongoing research and development, thereby dooming its survival.

To be able to compete with closed source technology companies who have massive profit margins, one needs income.

Dual licensing on Free Software, as we applied it at MySQL, works only for a limited subset of products (something I have called ‘infrastructure software’) that customers need to combine with their own closed source software and distribute to their customers. Most software products are not like that. This is why David Axmark and I created the Business Source license (BSL), a license designed to harmonize producing Open Source software and running a successful software company.

The intent of BSL is to increase the overall freedom and innovation in the software industry, for customers, developers, user and vendors. Finally, I hope that BSL will pave the way for a new business model that sustains software development without relying primarily on support.

For those who are interested in the background, Linus Nyman, a doctoral student from Hanken School of Economics in Finland), and I worked together on an academic article on the BSL.

Today, MariaDB Corporation is excited to introduce the beta release of MariaDB MaxScale 2.0, our database proxy, which is released under BSL. I am very happy to see MariaDB MaxScale being released under BSL, rather than under an Open Core or Closed Source license.  Developing software under BSL will provide more resources to enhance it for future releases, in similar ways as Dual Licensing did for MySQL. MariaDB Corporation will over time create more BSL products. Even with new products coming under BSL, MariaDB Server will continue to be licensed under GPL in perpetuity. Keep in mind that because MariaDB Server extends earlier MySQL GPL code it is forever legally bound by the original GPL license of MySQL.

In addition to putting MaxScale under BSL, we have also created a framework to make it easy for anyone else to license their software under BSL.

Here follows the copyright notice used in the MaxScale 2.0 source code:

/*
* Copyright (c) 2016 MariaDB Corporation Ab
*
* Use of this software is governed by the Business Source License
* included in the LICENSE.TXT file and at www.mariadb.com/bsl.
*
* Change Date: 2019-01-01
*
* On the date above, in accordance with the Business Source
* License, use of this software will be governed by version 2
* or later of the General Public License.
*/

Two out of three top characteristics of the BSL are already shown here: The Change Date and the Change License. Starting on 1 January 2019 (the Change Date), MaxScale 2.0 is governed by GPLv2 or later (the Change License).

The centrepiece of the LICENSE.TXT file itself is this text:

Use Limitation: Usage of the software is free when your application uses the Software with a total of less than three database server instances for production purposes.

This third top characteristic is in effect until the Change Date.

What this means is that the software can be distributed, used, modified, etc., for free, within the use limitation. Beyond it, a commercial relationship is required – which, in the case of MaxScale 2.0, is a MariaDB Enterprise Subscription, which permits the use of MaxScale with three or more database servers.

You can find the full license text for MaxScale at mariadb.com/bsl and a general BSL FAQ at mariadb.com/bsl-faq-adopting. Feel free to copy or refer to them for your own BSL software!

The key characteristics of BSL are as follows:
  • The source code of BSL software is available in full from day one.
  • Users of BSL software can modify, distribute and compile the source.
  • Code contributions are encouraged and accepted through the "new BSD" license.
  • The BSL is purposefully designed to avoid vendor lock-in. With vendor lock in, I here mean that users of BSL software are not depending on one single vendor for support, fixing bugs or enhancing the BSL product.
  • The Change Date and Change License provide a time-delayed safety net for users, should the vendor stop developing the software.
  • Testing BSL software is always free of cost.
  • Production use of the software is free of cost within the use limitation.
  • Adoption of BSL software is encouraged with use limitations that provide ample freedom.
  • Monetisation of BSL software is driven by incremental sales in cases where the use limitation applies.
Whether BSL will be widely adopted remains to be seen. It’s certainly my desire that this new business model will inspire companies who develop Closed Source software or Open Core software to switch to BSL, which will ultimately result in more Open Source software in the community. With BSL, companies can realize a similar amount of revenue for the company, as they could with closed source or open core, while the free of cost usage in core production scenarios establishes a much larger user base to drive testing, innovation and adoption.


2016-02-11

Foundation report for 2015

This is a repost of Otto Kekäläinen's blog of the MariaDB foundations work in 2015.

The mariadb.org website had over one million page views in 2015, a growth of about 9% since 2014. Good growth has been visible all over the MariaDB ecosystem and we can conclude that 2015 was a successful year for MariaDB.

Increased adoption

MariaDB was included for the first time in an official Debian release (version 8.0 "Jessie") and there has been strong adoption of MariaDB 10.0 in Linux distributions that already shipped 5.5. MariaDB is now available from all major Linux distributions including SUSE, RedHat, Debian and Ubuntu. Adoption of MariaDB in other platforms also increased, and MariaDB is now available as a database option on, among others, Amazon RDS, 1&1, Azure and Juju Charm Store (Ubuntu).

Active maintenance and active development

In 2015 there were 6 releases of the 5.5 series, 8 releases of the 10.0 series and 8 releases of the 10.1 series. The 10.1 series was announced for general availability in October 2015 with the release of 10.1.8. In addition, there were also multiple releases of MariaDB Galera Cluster, and the C, Java and OBDC connectors as well as many other MariaDB tools. The announcements for each release can be read on the Mariadb.org blog archives with further details in the Knowledge Base. Some of the notable new features in 10.1 include:
We are also proud that the release remains backwards compatible and it is easy to upgrade to 10.1 from any previous MariaDB or MySQL release. 10.1 was also a success in terms of collaboration and included major contributions from multiple companies and developers.

MariaDB events and talks

The main event organized by the MariaDB Foundation in the year was the MariaDB Developer Meetup in Amsterdam in October, at the Booking.com offices. It was a success with over 60 attendees In addition there were about a dozen events in 2015 at which MariaDB Foundation staff spoke.

We are planning a new MariaDB developer event in early April 2016 in Berlin. We will make a proper announcement of this as soon as we have the date and place fixed.

Staff, board and members

In 2015 the staff included:
  • Otto Kekäläinen, CEO
  • Michael "Monty" Widenius, Founder and core developer
  • Andrea Spåre-Strachan, personal assistant to Mr Widenius
  • Sergey Vojtovich, core developer
  • Alexander Barkov, core developer
  • Vicențiu Ciorbaru, developer
  • Ian Gilfillan, documentation writer and webmaster
Our staffing will slightly increase as Vicențiu will start working full time in 2016 for the Foundation. Our developers worked a lot on performance and scalability issues, ported the best features from new MySQL releases, improved MariaDB portability for platforms like ARM, AIX, IBM s390 and Power8, fixed security issues and other bugs. A lot of time was also invested in cleaning up the code base as the current 2,2 million lines of code includes quite a lot of legacy code in it. Version control and issue tracker statistics shows that the foundation staff made 528 commits, reported 373 bugs or issues and closed 424 bugs or other issues. In total there were 2400 commits made by 91 contributors in 2015.

The Board of Directors in 2015 consisted of:
  • Chairman Rasmus Johansson, VP Engineering at MariaDB Corporation
  • Michael "Monty" Widenius, Founder and CTO of MariaDB Corporation
  • Jeremy Zawodny, Software Engineer at Craigslist
  • Sergei Golubchik, Chief Architect at MariaDB Corporation
  • Espen Håkonsen, CIO of Visma and Managing Director of Visma IT & Communications
  • Eric Herman, Principal Developer at Booking.com
MariaDB Foundation CEO Otto Kekäläinen served as the secretary of the board. In 2015 we welcomed as new major sponsors Booking.com, Visma, Verkkokauppa.com. Acronis just joined to be a member for 2016. Please check out the full list of supporters. If you want to help the MariaDB Foundation in the mission to guarantee continuity and open collaboration, please support us as with individual or corporate sponsorship.

What will 2016 bring?

We expect steady growth in the adoption of MariaDB in 2016. There are many migrations from legacy database solutions underway, and as the world becomes increasingly digital, there are a ton of new software projects starting that use MariaDB to for their SQL and no-SQL data needs. In 2016 many will upgrade to 10.1 and the quickest ones will start using MariaDB 10.2 which is scheduled to be released some time during 2016. MariaDB also has a lot of plugins and storage engines that are getting more and more attention, and we expect more buzz around them when software developers figure out new ways to manage data in fast, secure and scalable ways.

2015-01-13

Foundation report for 2014

2014 was a productive year for the MariaDB Foundation.

Here is a list of some of the things MariaDB Foundation employees have
accomplished during 2014:

The 3 full-time MariaDB Foundation developers have worked hard to make MariaDB better:
  • Some 260 commits
  • Some 25 reviews of code from the MariaDB community.
  • Fixed some 170 bugs and new features. For a full list, please check Jira.
  • Reported some 160 bugs.
Some of the main new features Foundation developers have worked on in 2014 are:
  • Porting and improving MariaDB on IBM Power8.
  • Porting Galera to MariaDB 10.1 as a standard feature.
  • Query timeouts (MDEV-4427)
  • Some coding and reviews of Parallel replication in MariaDB 10.1.
  • Working with code from Google and Eperi to get table space and table level encryption for InnoDB and XtraDB.
  • Allowing storage engines to shortcut group by queries (for ScaleDB) (MDEV-6080).
  • Moronga storage engine (reviews and porting help)
  • Connect storage engine (reviews and porting help)
  • Spider storage engine (merging code with MariaDB)
  • Query timeouts (MDEV-4427)
  • Merge INET6_ATON() and INET6_NTOA() from MySQL-5.6 (MDEV-4051)
  • Make "CAST(time_expr AS DATETIME)" compatible...SQL Standard) (MDEV-5372)
  • Command line variable to choose MariaDB-5.3 vs MySQL-5.6 temporal data formats (MDEV-5528)
  • Added syntax CREATE OR REPLACE to tables, databases, stored procedures, UDF:s and Views (MDEV-5491. The original TABLE code was done by Monty, other parts was done as a Google Summer Of Code project by Sriram Patil with Alexander Barkov as a mentor.
  • Upgraded the bundled Perl Compatible Regular Expression library (PCRE) to 8.34 (MDEV-5304)
  • Reduced usage of LOCK_open (MDEV-5403) (MDEV-5492) (MDEV-5587)
  • Ported patches from WebScaleSQL to MariaDB (MDEV-6039)
  • Better preallocation of memory (MDEV-7004)
  • Lock-free hash for table definition cache (MDEV-7324)
  • A lot of speed optimizations (changing mutex usage, better memory allocations, optimized bottlenecks, memory barriers etc).
The MariaDB documentation/knowledgebase:
has now 3685 articles about MariaDB and MySQL. Foundation employees added during 2014 223 new ones and did 6045 edits.

Some of the main new articles from us are:
We also have a lot of outside contributors and translators. Thanks a lot to all of you!

We also visited and talked about MariaDB at a lot of conferences:
In addition I had several talks at different companies who were moving big installations to MariaDB and needed advice.

We where also able to finalize the MariaDB trademark agreement between the MariaDB corporation and the MariaDB Foundation. This ensures that that anyone can be part of MariaDB development on equal terms. The actual trademark agreement can be found here.

On the personnel side, we were sad to see Simon Phipps leave the position as CEO of the Foundation.

One the plus side, we just had 2 new persons join the MariaDB foundation this week:
  • We are happy to have Otto Kekäläinen join us as the new CEO for the MariaDB foundation! Otto has in the past done a great work to get MariaDB into Debian and I am looking forward to his work on improving everything we do in the MariaDB foundation.
  • Vicențiu Ciorbaru has joined the MariaDB foundation as a developer. In the past Vicențiu added ROLES to MariaDB, as part of a Google Summer of Code project and he is now interested to start working on the MariaDB optimizer. A special thanks to Jean-Paul Smets at Nexedi for sponsoring his work at the foundation!
Last, I want to give my thanks to the MariaDB foundation members who made all the foundation work possible for 2014:
For 2015 we welcome a new member, Visma. Visma will be part of the foundation board and will help push MariaDB development forwards.

As the above shows, the MariaDB Foundation is not only a guarantee that MariaDB will always be an actively developed open source project, we also do a lot of development and practical work. This is however only possible if we have active members who sponsor our work!
If you are interested in helping us, either as a member, sponsor, or by giving development resources to the MariaDB foundation, please email us at foundation at mariadb.org !

2014-10-28

MariaDB foundation trademark agreement


We have now published the trademark agreement between the MariaDB Corporation (formerly SkySQL) and the MariaDB Foundation. This agreement guarantees that MariaDB Foundation has the rights needed to protect the MariaDB server project!

With this protection, I mean to ensure that the MariaDB Foundation in turn ensures that anyone can be part of MariaDB development on equal terms (like with any other open source project).

I have received some emails and read some blog posts from people who are confusing trademarks with the rights and possibilities for community developers to be part of an open source project.

The MariaDB foundation was never created to protect the MariaDB trademark. It was created to ensure that what happened to MySQL would never happen to MariaDB: That people from the community could not be part of driving and developing MySQL on equal terms as other companies.

I have personally never seen a conflict with having one company own the trademark of an open source product, as long as anyone can participate in the development of the product! Having a strong driver for an open source project usually ensures that there are more full-time developers working on a project than would otherwise be possible. This makes the product better and makes it useful for more people. In most cases, people are participating in an open source project because they are using it, not because they directly make money on the project.

This is certainly the case with MySQL and MariaDB, but also with other projects. If the MySQL or the MariaDB trademark would have been fully owned by a foundation from a start, I think that neither project would have been as successful as they are! More about this later.

Some examples of open source projects that have the trademark used or owned by a commercial parent company are Wordpress (wordpress.com and Wordpress.org) and Mozilla.

Even when it comes to projects like Linux that are developed by many companies, the trademark is not owned by the Linux Foundation.

There has been some concern that MariaDB Corporation has more developers and Maria captains (people with write access to the MariaDB repositories) on the MariaDB project than anyone else. This means that the MariaDB Corporation has more say about the MariaDB roadmap than anyone else.

This is right and actually how things should be; the biggest contributors to a project are usually the ones that drive the project forward.

This doesn't, however, mean that no one else can join the development of the MariaDB project and be part of driving the road map.

The MariaDB Foundation was created exactly to guarantee this.

It's the MariaDB Foundation that governs the rules of how the project is developed, under what criteria one can become a Maria captain, the rights of the Maria captains, and how conflicts in the project are resolved.

Those rules are not yet fully defined, as we have had very few conflicts when it comes to accepting patches. The work on these rules have been initiated and I hope that we’ll have nice and equal rules in place soon. In all cases the rules will be what you would expect from an open source project. Any company that wants to ensure that MariaDB will continue to be a free project and wants to be part of defining the rules of the project can join the MariaDB Foundation and be part of this process!

Some of the things that I think went wrong with MySQL and would not have happened if we had created a foundation similar to the MariaDB Foundation for MySQL early on:

  • Claims that companies like Google and Ebay can't get their patches into MySQL if they don't pay (this was before MySQL was bought by Sun).
  • Closed source components in MySQL, developed by the company that owns the trademark to MySQL (almost happened to MySQL in Sun and has happened in MySQL Enterprise from Oracle).
  • Not giving community access to the roadmap.
  • Not giving community developers write access to the official repositories of MySQL.
  • Hiding code and critical test cases from the community.
  • No guarantee that a patch will ever be reviewed.

The MariaDB Foundation guarantees that the above things will never happen to MariaDB. In addition, the MariaDB Foundation employs people to perform reviews, provide documentation, and work actively to incorporate external contributions into the MariaDB project.

This doesn't mean that anyone can push anything into MariaDB. Any changes need to follow project guidelines and need to be reviewed and approved by at least one Maria captain. Also no MariaDB captain can object to the inclusion of a given patch except on technical merits. If things can't be resolved among the captains and/or the user community, the MariaDB Foundation has the final word.

I claimed earlier that MariaDB would never have been successful if the trademark had been fully owned by a foundation. The reason I can claim this is that we tried to do it this way and it failed! If we would have continued on this route MariaDB would probably be a dead project today!

To be able to understand this, you will need a little background in MariaDB history. The main points are:

  • Some parts of the MariaDB team and I left Sun in February 2009 to work on the Maria storage engine (now renamed to Aria).
  • Oracle started to acquire Sun in April 2009.
  • Monty Program Ab then hired the rest of the MariaDB engineers and started to focus on MariaDB.
  • I was part of founding SkySQL in July 2010, as a home for MySQL support, consultants, trainers, and sales people.
  • The MariaDB Foundation was announced in November 2012.
  • Monty Program Ab and SkySQL Ab joined forces in April 2013.
  • SkySQL Ab renamed itself to MariaDB Corporation in October 2014

During the 4 years before the MariaDB foundation was formed, I had contacted most of the big companies that had MySQL to thank them for their success and to ask them to be part of MariaDB development. The answers were almost all the same:

"We are very interested in you succeeding, but we can't help you with money or resources until we are using MariaDB ourselves. This is only going to happen when you have proved that MariaDB will take over MySQL."

It didn't help that most of the companies that used to pay for MySQL support had gotten scared of MySQL being sold to Oracle and had purchased 2-4 year support contracts to protect themselves against sudden price increases in MySQL support.

In May 2012, after 4 years and spending close to 4 million Euros of my own money, to make MariaDB possible, I realized that something would have to change.

I contacted some of the big technology companies in Silicon Valley and asked if they would be interested in being part of creating a MariaDB Foundation, where they could play bigger roles. The idea was that all the MariaDB developers from Monty Program Ab, the MariaDB trademark and other resources would move to the foundation. For this to happen, I need guarantees that the foundation would have resources to pay salaries to the MariaDB developers for at least the next 5 years.

In the end two companies showed interest in doing this, but after months of discussions they both said that "now was not yet the right time to do this".

In the end I created the MariaDB Foundation with a smaller role, just to protect the MariaDB server, and got some great companies to support our work:

  • Booking.com
  • SkySQL (2 years!)
  • Parallels (2 years!)
  • Automattic
  • Zenimax

There was also some smaller donations from a variety of companies.

See the whole list at https://mariadb.org/en/supporters.

During this time, SkySQL had become the biggest supporter of MariaDB and also the biggest customer of Monty Program Ab. SkySQL provided front line support for MySQL and MariaDB and Monty Program Ab did the "level 3" support (bug fixes and enhancements for MariaDB).

In the end there were only two ways to go forward to secure the financing of the MariaDB project:

a) Get investors for Monty Program Ab
b) Sell Monty Program Ab.

Note that neither of the above options would have been possible if Monty Program Ab had not owned the MariaDB trademark!

Selling to SkySQL was in the end the right and logical thing to do:

  • They have good investors who are committed to SkySQL and MariaDB.
  • Most of the people in the two companies already know each other as most come from the old MySQL team.
  • The MariaDB trademark was much more known than SkySQL and by owning it would make it much easier for SkySQL to expand their business.
  • As SkySQL was the biggest supporter of the MariaDB project this felt like the right thing to do.

However, to ensure the future of the MariaDB project, SkySQL and Monty Program Ab both agreed that the MariaDB Foundation was critically needed and we had to put a formal trademark agreement in place. Until now there was just a verbal promise for the MariaDB trademarks to the foundation and we had to do this legally right.

This took, because of a lot of reasons too boring to bring up here, much longer time than expected. You can find the trademark agreement publicly available here.

However, now this is finally done and I am happy to say that the future of MariaDB, as an open source project, is protected and there will never again be a reason for me to fork it!

So feel free to join the MariaDB project, either as a developer or community contributor or as a member of the MariaDB Foundation!