I just did a release of Package Drone 0.11.0-rc1. Yes, it is the release candidate for the long overdue 0.11.0 version.

Since 0.11.0 will break a few things, I did promise to explain what and why. Here it comes.

In short, SQL/JDBC/JPA sucked and upgrading from 0.10.x to 0.11.0 requires and export and import cycle for your channel data. The RPM/DEB packages got renamed and declare a “conflict” on the old packages to ensure manual intervention.

If you want to know the details, please bear with me and read on 😉

Storage Model – Part 1

The initial storage model of Package Drone was to store everything in an SQL database. It sounded like a simple idea in the beginning. However after a few “I-told-you-so”s caused by issues like bad locking from MySQL, the inability to actually stream BLOBs, troubles with EclipseLink and Gemini JPA in the OSGi runtime and some performance issues it was time for a re-write of the storage layer.

Sometime things just cannot be fixed and after a few releases we already learned quite a bit of what we would really like to have internally. So re-writing sounded like a great idea, and still was the right idea. But beside the fact that Package Drone is some sort of side-project and time is limited, also did we ran into the second system effect. All these ideas which we had in the past, like pluggable channel implementations, cloud storage, a better and more Java 8 like API, … the list goes on.

So instead of just changing the underlying storage model also the internal API and channel structure was cleaned up. This cost a little bit more time than originally anticipated, but finally it’s done!

The bad news first, there simply was no reasonable way to migrate from the SQL based backend to the current storage model. Initially the idea was to plug in the old model using the new pluggable channel implementation provider system, but this would mean keeping the rather complex and error prone EclipseLink/Gemini JPA runtime alive for a few more releases. Which would also cause a few problems when migrating over the Eclipse Foundation, since neither the MySQL nor PostgreSQL JDBC driver will receive clearance for distribution.

So in the end, the migration over to version 0.11.0 requires the manual step of exporting the channels and re-importing them in the new instance. This is one manual step which has to be performed and is the reason why the package names of the RPM and DEB package was changes at the same time. For the first release at the Eclipse Foundation we preferred to have “org.eclipse.packagedrone” as part of the RPM/DEB name, so this step is now already done and a conflict to the old package name (package-drone-server) was declared, so it cannot accidentally happen that an upgrade from 0.10.x to 0.11.0 is made, the system starts up and looks like a dead fish.

All you have to is to “Export all” and “Import all” for you channel content to be migrated.

Storage Model – Part 2

Now what does the new storage model actually look like? It depends! I could not be more vague 😉 But it truly does. So each storage model is represented by a StorageModule implementation, registered to a StorageManager, which coordinates the locking and requests for persisting modules. A module itself can now decide how it stores its content, and right now this is done by a few simple file storage approaches. The global meta data for example is simply stored in a property file.

The basic principle for the storage model is that each model gets registered with the manager to coordinate locking, and each module has a write and read mode. The locking is right now limited to simple read/write locks (meaning multiple readers and only one writer), but in the future there should be a lockless read mode in addition (so that reading is still possible while one writer is active).

The storage manager also allows to set up a caching model for faster reading. So the global meta data is stored in an immutable map for readers which will be replaced with a new version once the write modification has been completed successfully. The same approach is used for the actual channel data. Yes it requires a more memory, but offers a lot more performance when accessing the data. Which is what we wanted all along.

At the same time it is possible to use difference approaches in the storage module implementation. It would also be possible to store data in some of bucket storage (like S3) or again in an SQL database if you like so.

Package name & name space

One requirement for moving a project to the Eclipse Foundation is to change the Java namespace. So the new namespace is org.eclipse.packagedrone. In addition to this simple rename, the project structure in Git was also cleaned up. A task long overdue.

At the same time the DEB and RPM packages got renamed to org.eclipse.packagedrone.server and the RPM release numbers where changed in a way that milestone builds lower prioritized than release candidates and final release builds.

So what will be new?

So where are all the cool new features??

Well … it is a release which has lots of changes in the background, but not many visible changes or new features. Of course there is better performance, better compatibility with Gradle uploads and finally an OBR index which can be generated.

And of course we made it thinner, faster and the battery lasts 5% longer 😉

When will it be ready?

On thedrone.packagedrone.org we are running Package Drone versions of 0.11.0 for some time now. “rc1” got some more extensive testing, all unit, integration and performance tests look good.

I do hope we can release this as soon as possible, maybe the EclipseCon Europe 2015 is a good idea. See you there 😉

What’s next?

While the performance of the channel operations was improved, there is still some room for more. We want to have an even better performance for the actual storage implementation of the channels, and also we plan for a channel history. A thing which always caused performance issues in the past was the costly aggregation of P2, OBR and other repository formats in the end of a channel operation. 3 seconds for aggregating a 80 MB P2 repository does not sound too bad, but right now it has to be done with every small maven artifact which is uploaded if you don’t upload full P2 repositories.

So instead of synchronously aggregating repositories we want to this in a lazy fashion. Aggregation takes place in asynchronously the background, but in the meantime Package Drone will serve an older, but aggregated version to P2 consumers. But this would cause a few more months until this is finished, so we decided to not delay 0.11.0 any longer, but do this in the next release.

Since the internal API is now cleaned up, it is also possible now to move on and add a couple of new features, adding a nice REST API, more repository formats and a couple of internal features (like triggers, and spool out).

The upcoming 0.11.0 release
Tagged on:     

Leave a Reply

Your email address will not be published. Required fields are marked *