Long time no see … well, I guess it is time for a status update 🙂
The last few weeks were a mixture of being busy, having no time and vacation. Which sadly cut down the time I could invest in Package Drone a lot.
In the meantime we are also currently working on pushing Package Drone to the Eclipse Foundation. All external dependencies, which are being used by Package Drone and are now currently approved by the Eclipse Foundation have to be screened and approved. Since we do have quite a number of those dependencies, we must need to wait a bit longer for all CQs to be processed.
In the meantime I wanted to use the time and reduce the number of problematic dependencies and clean up an design decision I made in the beginning of Package Drone. Package Drone does store most data in an SQL database. In recent versions a file system blob store was added, since bost MySQL and Postgres, have quite a bad implementation in their JDBC driver when it comes to handling BLOB data. However, as it turned out, using JDBC and JPA for this task has a huge number of downsides. (I heard a few “I told so you” from my colleague about that).
So the idea was to replace the JDBC/JPA based storage backend with a plain and simple implementation using the filesystem. Of course when you make such a change, you actually can re-write several parts of your system, which allowed me to clean up a few areas I wanted to clean up for some time, fix up the internal API, etc …
Of course replacing the SQL based persistence with a file system persistence requires you do take responsibility over quite a few features which database already bring along (ACID). On the other side did the MySQL full table locks bring their own trouble.
Now the whole storage backend is replaced with a locking manager, which prevents deadlocks and uses read/write locks, the storage model and actually persist arbitrary data without schema updates. The setup process even got a bit simpler. The license problems of Postgres and MySQL regarding the Eclipse Foundation and the EPL are solved. The number of dependencies is reduced and last but not least, the performance greatly improved.
Also will this allow us to enhance Package Drone in a future version to make use of cloud storages like S3.
There always is a “but” … right, now of course this whole change did require some time. But today I was finally able to make a full build, successfully run all unit tests, and even successfully run the performance tests. So the next milestone release with all these changes is only a few days away. But I do guess there will be a few issues hidden, which have to be found before making a new final release.
Also is the question of how to migrate from version 0.10.x to version 0.11.x still unsolved. Simply upgrading to the new version won’t work, since he whole setup is totally different. However the export and import process can be used to migrate all data from the old package drone storage to the new one.
So I am really looking forward to a few new releases and the opportunity to finally apply a few pull requests which had to wait on GitHub.
Have a nice weekend 😉