First builds at Eclipse

Yes, it took a bit longer than expected … but here we are! The first few builds of Eclipse Package Drone have been run on the Package Drone HIPP instance at Eclipse, pulling its source from the GitHub repository eclipse/packagedrone. This is a big step forward in bringing Package Drone to the Eclipse Foundation.

The next thing will be a first release, but for this we still have to wait until all CQs have been fully processed. So this may still take a while and I guess it won’t be before January 2016 that we have a first release of Eclipse Package Drone.

So what’s the difference right now? We decided to start with a fresh git repository, so the old repository will still be around in the same way it was. Old versions (0.10.x and before) will still be maintained here. However the project itself moves on and will use the new GitHub repository for active development. All links will point to the eclipse/packagedrone repository. Also if you want to contribute now (even through GitHub) it is necessary now that you sign the Eclipse CLA.

When it comes to the wiki, we will use the wiki of the Eclipse Foundation and have our space at https://wiki.eclipse.org/PackageDrone. For downloading releases, we are still undecided if we will continue to use GitHub releases or use the download space of the Eclipse Foundation. But for issues, we are glad that we can be one of the first projects which may use GitHub issues as an Eclipse project. So in the case of an issue or idea or whatever, head over the new issue space instead of using the old one. We will probably copy over a few issues, but on the same time like to use it as a fresh, clean start. So if you would like to help, copy over your issues to the new repository and close it in the old one. Thanks for helping!

So don’t be alarmed when the links on the homepage slowly start to change towards the new GitHub repository eclipse/packagedrone, it is a good sign and a lot effort went into it 😉

Package Drone 0.11.0

Klonk! … Package Drone 0.11.0 is out!

Be sure to check out the most recent release on github.

This release features the already described change in the storage model. Although this does require a clean re-installation for this release, it will allow future development to consider storage options like S3, different storage implementations in general and reduced the installation size by 10MB (about 25%).

As always the GitHub release notes are in more detail, but a summary of the changes is right here:

  • Changed storage model
  • Support for OBR
  • Performance improvements
  • The usual bug fixes and minor improvements
  • Initial support for OpenSUSE
  • Rename package names to Eclipse Foundation conventions
  • Re-structure git repository
  • GoogleBot handling

So what about the 0.10.x release? Of course it will still be available on GitHub. And if critical bugs come up which are not related to the general database setup, then these will be patches as well. However development will move on. And it is possible to migrate data by using the “export” and “import” functionality.

The upcoming 0.11.0 release

I just did a release of Package Drone 0.11.0-rc1. Yes, it is the release candidate for the long overdue 0.11.0 version.

Since 0.11.0 will break a few things, I did promise to explain what and why. Here it comes.

In short, SQL/JDBC/JPA sucked and upgrading from 0.10.x to 0.11.0 requires and export and import cycle for your channel data. The RPM/DEB packages got renamed and declare a “conflict” on the old packages to ensure manual intervention.

If you want to know the details, please bear with me and read on 😉

(more…)

Package Drone 0.10.4 & 0.11.0-M2

Whuiiii … Package Drone 0.10.4 is out!

… but wait, there is more … Package Drone 0.11.0.m2 is out as well!

Be sure to check out the most recent release on GitHub.

The 0.10.4 release is just a maintenance release which fixes a critical issue with the channel export functionality. In addition JFingerle fixes a bunch of issues for the OSGi R5 index generation.

The export functionality is a critical issue since the migration path from 0.10.x to 0.11.0 includes exporting and re-importing the full channel set. This is necessary since the storage model of Package Drone has been completely re-written in the 0.11.x branch. Instead of storing data in a relational (SQL) database and requiring a rather complex JPA setup using OSGi, the new storage manager and modules will simply store data in the local file system. Not only did the runtime setup get easier by using this approach, but also is it now possible to enhance the object model using aspects and the performance was increased by caching most information in memory.

In addition this change will allow us to enhance Package Drone in future releases to support other storage models, such as cloud storage.

There will be a more detailed blog post before the final release of 0.11.0.

The OpenShift “next” version of Package Drone has been updated to use the milestone 2 of 0.11.0 as well. So you can try out the new version.

Thanks to all contributors!

What’s up?

Long time no see … well, I guess it is time for a status update 🙂

The last few weeks were a mixture of being busy, having no time and vacation. Which sadly cut down the time I could invest in Package Drone a lot.

In the meantime we are also currently working on pushing Package Drone to the Eclipse Foundation. All external dependencies, which are being used by Package Drone and are now currently approved by the Eclipse Foundation have to be screened and approved. Since we do have quite a number of those dependencies, we must need to wait a bit longer for all CQs to be processed.

In the meantime I wanted to use the time and reduce the number of problematic dependencies and clean up an design decision I made in the beginning of Package Drone. Package Drone does store most data in an SQL database. In recent versions a file system blob store was added, since bost MySQL and Postgres, have quite a bad implementation in their JDBC driver when it comes to handling BLOB data. However, as it turned out, using JDBC and JPA for this task has a huge number of downsides. (I heard a few “I told so you” from my colleague about that).

So the idea was to replace the JDBC/JPA based storage backend with a plain and simple implementation using the filesystem. Of course when you make such a change, you actually can re-write several parts of your system, which allowed me to clean up a few areas I wanted to clean up for some time, fix up the internal API, etc …

Of course replacing the SQL based persistence with a file system persistence requires you do take responsibility over quite a few features which database already bring along (ACID). On the other side did the MySQL full table locks bring their own trouble.

Now the whole storage backend is replaced with a locking manager, which prevents deadlocks and uses read/write locks, the storage model and actually persist arbitrary data without schema updates. The setup process even got a bit simpler. The license problems of Postgres and MySQL regarding the Eclipse Foundation and the EPL are solved. The number of dependencies is reduced and last but not least, the performance greatly improved.

Also will this allow us to enhance Package Drone in a future version to make use of cloud storages like S3.

There always is a “but” … right, now of course this whole change did require some time. But today I was finally able to make a full build, successfully run all unit tests, and even successfully run the performance tests. So the next milestone release with all these changes is only a few days away. But I do guess there will be a few issues hidden, which have to be found before making a new final release.

Also is the question of how to migrate from version 0.10.x to version 0.11.x still unsolved. Simply upgrading to the new version won’t work, since he whole setup is totally different. However the export and import process can be used to migrate all data from the old package drone storage to the new one.

So I am really looking forward to a few new releases and the opportunity to finally apply a few pull requests which had to wait on GitHub.

Have a nice weekend 😉

Package Drone @ EclipseCon Europe 2015

EclipseCon Europe  2015

Package Drone will have a talk at the EclipseCon Europe 2015 in Ludwidgsburg.

We will present the current state of Package Drone and show some live demos on what you can do with Package Drone with a focus on OSGi. Hopefully we can also sit together and do some brain storming for future improvements while having a snack and a drink 😉

Since there are a lot of other talks around OSGi, Eclipse and Java, I truly recommend paying EclipseCon a visit, especially if you are located in Europe.

Package Drone 0.10.0

Schhhwapp … Package Drone 0.10.0 is out!

Be sure to check out the most recent release on github.

This release took a lot longer than most previous releases. But this was on purpose 😉 We touched a few core elements of the runtime, improved the performance a lot and also made it more stable.

As always, read the release notes! They contain the most information about this release and summarize all the milestone releases which led to 0.10.0.

Summing it all up, this would be:

  • Support for RPM/YUM
  • Huge performance improvements
  • Switch to Pax Web and Apache Felix for some OSGi services
  • Support for reCAPTCHA
  • and more…

Thanks to all contributors!

There are some minor tasks ahead, like adding sitemaps, adding support for OBR and fixing a few leftovers from the switch to Pax Web. In addition this release has provided base components necessary for implementing a decent REST API, which will lead to more integration tests. Also, since we got accepted as an official Eclipse project, it will become time relocate.

Eclipse, Eclipse & M6

Package Drone got accepted as an Eclipse Foundation project. Hooray 🙂 So before anything big will happen in this matter, it will take some time. First the project has to been provisioned, then the source and required dependencies be pushed through the IP review process. So, with a little bit of luck, we can make a first release as Eclipse project this year.

Also did we present Package Drone at the Eclipse DemoCamp Munich. If you missed the event, you can view the slides on slideshare. Although most of the time was used for demoing the actual system. So don’t expect too much from the slides.

Finally we made some more milestone releases the last weeks. So most recent, milestone 6, was released today. The focus was again on stability and performance. The plan is that the next release already is a release candidate, so the final release of 0.10.0 would be in the next few weeks.

0.10.0 – Milestone M3

Package Drone 0.10.0 Milestone 3 just got uploaded to GitHub.

The main task in this milestone release was performance. Adding some tools to be able to provide performance information and fixing the performance bugs found.

The first step was to add some profiling infrastructure, like the SLF4J Profiler. In the beginning a test with OSGi weaving was made, adding a trace like call to each method call was the idea. However it turned out that adding OSGi weaving and ASM bytecode transformation seemed quite a task. Also was the fact that each method would simply get some trace call wrapped not very promising.

So some explicit profile calls got added, which could also push some manual meta data on the profile call. Using try-with-resource and lambdas the syntax does not seem to be that intrusive while the way this can be controlled is quite helpful. In fact so helpful that two serious performance issues could be fixed.

The last performance issue, in this release, was found by using an actual Java profiler. Thanks to ej-technologies GmbH we got an open source license of JProfiler. Thanks!

Don’t put too much weight on the following numbers but for one specific use case the performance was improved from 18 seconds for uploading a single artifact (including a full channel regeneration) to about 2.5 seconds for the same operation. And while there are a few more spots which could be optimized, it think this is a good first step.