udoprog.github.io

Introducing @AutoSerialize

In this post I’ll present a new framework for serialization in Java that I’ve been working on over the last year.

The initial goal was to implement a paradigm which I felt I’ve been duplicating many times; efficient, portable, and hassle-free serialization of objects.

I’m also strongly biased towards functional programming, so immutability plays a big role in my projects.

TinySerializer

Enter TinySerializer.

It’s a project that started out as two interfaces (Serializer and SerializerFramework), but recently grew into it’s shoes through the inspiration of projects like AutoValue, and AutoMatter.

I realized that annotation processors have access to a tremendous amount of information about the types and structure of your classes, and that this information could be used to generate very efficient and readable serializers.

In the following example, I’ll be using Lombok’s @Data annotation.

So, given a Person like the following:

@AutoSerialize
@Data
class Person {
    private final String name;
    private final Optional<Job> job;
}

And a Job:

@AutoSerialize
@Data
class Job {
    private final String name;
}

The @AutoSerialize processor would then generate the following serializer for Person:

@AutoSerialize
class Person_Serializer implements Serializer<Person> {
    private final Serializer<String> s_String;
    private final Serializer<Optional<Job>> s_OptionalJob;

    public Person_Serializer(final SerializerFramework framework) {
        s_String = framework.string();
        s_OptionalJob = framework.optional(new Job_Serializer(framework));
    }

    public void serialize(SerialWriter buffer, Person person) {
        s_String.serialize(buffer, person.getName());
        s_OptionalJob.serialize(buffer, person.getJob());
    }

    public Person deserialize(SerialWriter buffer) {
        final String v_name = s_String.deserialize(buffer);
        final Optional<Job> v_job = s_OptionalJob.deserialize(buffer);
        return new Person(v_name, v_job);
    }
}

The use of SerializerFramework is an important detail, this gives us a lot of freedom in deciding the exact details of how this serialization is to operate.

In the serializer, there are no concrete implementations or static initialization. Everything is Plain Old Boring Java.

You activate the serializer would be to add the following snippet to your maven dependencies:

<dependencies>
  <dependency>
    <groupId>eu.toolchain.serializer</groupId>
    <artifactId>tiny-serializer-api</artifactId>
    <version>${tiny.version}</version>
  </dependency>
  <dependency>
    <groupId>eu.toolchain.serializer</groupId>
    <artifactId>tiny-serializer-processor</artifactId>
    <version>${tiny.version}</version>
    <scope>provided</scope>
  </dependency>
</dependencies>

Implementation Details

So the above provides you with essentially free object graph serialization for your objects, but what about primitive types and containers?

To this end, I’ve built a small companion implementation in tiny-serializer-core which provides you with a relatively sane SerializerFramework implementation.

<dependency>
  <groupId>eu.toolchain.serializer</groupId>
  <artifactId>tiny-serializer-core</artifactId>
  <version>${tiny.version}</version>
</dependency>

With it, you can build a new SerializerFramework and use it with your class like this:

public class Application {
    public static void main(String argv[]) {
        final SerializerFramework framework = TinySerializer.builder().build();
        final Serializer<Person> person = new Person_Serializer(framework);

        try (final SerialWriter writer = framework.writeStream(new FileOutputStream("person.bin")) {
            person.serialize(writer, new Person("John-John Tedro", Optional.of(new Job("Programmer"))));
        }
    }
}

You can see this a larger example in action at the SerializeToFile example.

There are also plenty of configuration options available in TinySerialize that you can play around with.

Final Words

I hope you find it useful!

If you want to know more, go to udoprog/tiny-serializer-java. There is plenty more to read and play with there.

I also intend to eventually implement field-value based serialization that allows for out-of-order fields. This is the basis for semantically versioned objects with forward compatibility, and I consider it an important omission.

Becoming a Debian Maintainer

I’ve recently started looking into how exactly one would start contributing to such a huge project like Debian.

It was suprisingly difficult to get my bearings, but I’m starting to get more comfortable with the inner workings of Debian as a Project.

In this Post I will attempt to explain the things I would have liked to be explained to me when I started out.

Maintainers and Developers

The first point of contact should be the Debian New Maintainers’ Guide, pay specific attention to the Getting started The Right Way.

Before continuing you should read this chapter.

Here I noticed that there are three roles that sticks out.

An upstream maintainer which is the person currently maintaining a program. A Debian Maintainer is a person that has limited upload rights to the Debian package archive. And finally a Debian Developer, which is a person which has full upload rights to the official Debian package archive.

The chapter touches briefly on the difference between an upstream maintainer and a Debian Maintainer. But what is the practical difference?

The answer can be found on mentors.debian.net.

Only approved members of the Debian project (Debian Developers) are granted the permission to upload software packages into the Debian distribution. Still a large number of packages is maintained by non-official developers. How do they get their work into Debian when they are not allowed to upload their own packages directly? By means of a process called sponsorship. Sponsorship means that a Debian Developer uploads the package on behalf of the actual maintainer. The Debian Developer will also check the package for technical correctness and help the maintainer to improve the package if necessary. Therefore the sponsor is sometimes also called a mentor.

The mentors project will also be your testing ground, a safe place where you can receive reviews for your work without causing too much mayhem in the rest of the project, but open enough and tolerant towards newcomers (like me).

Systems

The community is built around a couple of systems. The most important (I’ve found so far) are the following.

bugs.debian.org

Bug reports are the documentation process for the entire project. Everything in Debian is tracked by a bug.

The following are the types of bugs you will initially encounter.

Mailing Lists

These are the lists that you should be apart of.

  • debian-mentors
    • Lurk and participate, this is where you interact with mentors and ask silly questions. Beware though, you are expected to do your homework.
  • debian-devel
    • Lurk mostly, this is where technical discussions are being performed for the project as a whole, they will probably be over your head for now.
  • debian-user
    • Lurk and answer any questions that you feel comfortable in answering.

First Steps

There are loads of ways to contribute to Debian.

You can write documentation, translate, maintain packages.

For me the first step will be too look for either a package that I would like to maintain. This can be either an existing, orphaned package by changing its bugs’ O (Orhpan) status to ITA (Intent to Adopt). Just make sure you actually do intend to do it.

Then, regardless of if it’s a package you’ve adopted, or a new one, after you’ve done the relevant changes and want to have a sponsor, you’d file a RFS bug according to this guide.

Links

For more detailed information, read the following pages.

Manually Troubleshooting Portage

Portage is a layered system.

The most common interface which everyone using gentoo should be familiar with is emerge. It is the definitive command-line interface to the portage system1.

Sometimes, however, you are troubleshooting that one failing build, and the minute-long wait for portage to calculate dependencies is mostly a hazzle.

ebuild

Portage ebuilds, and the corresponding ebuild comamnd, is used to build and install packages.

The sequence of commands that emerge runs is the following.

  • portage <ebuild> compile
  • portage <ebuild> install
  • portage <ebuild> qmerge

You as a user can run these manually, just find a package and replace the <ebuild> with the corresponding ebuild that you wish to install. This would cause the ebuild to be installed in the same manner as emerge would have done it, with the following exceptions.

  • Dependencies of the package will not be resolved.
  • The package will not be added to the world set (you can avoid this with --oneshot).

Apart from this, the package would be installed on your system as usual.

You as a developer can now inspect each step for errors, and fix the problems you find.

If you decide that the ebuild has to be modified it’s equally straight forward.

#> mkdir -p $HOME/temp
#> cp -R /usr/portage/<category>/<package> $HOME/temp/<package>
... modify ebuild ...
#> sudo ebuild --skip-manifest <modified-ebuild> ...

This will cause ebuild to automatically configure a temporary overlay named X-<USER>.

If you later want to find these packages, use equery.

#> equery has repository x-<user>
...

Summary

Interacting with portage can feel daunting at first, but the steps you are taking are no different from the ones emerge itself uses when building your system.

That is the power of a layered system.

P.S. — You are of course still free to thoroughly shoot yourself in the foot, so be a bit careful!

Dual Booting: Debian Wheezy & Windows 8.1

So I recently got myself in a situation where I had to convert a system that booted Windows 8.1 through EFI, and Debian through MBR legacy into grub.

Of course I had been using my Debian partition for a while and was unkeen to get rid of the rather large amount of changes I’ve already done.

What got me into this mess

I recently acquired a new computer, and being the cheap bastard I am I only purchased one new SSD for it. So to get my priorities straight: at home I mostly game. So this disk was bound to run Windows.

After a pretty happy installation I sadly discovered that my old hard drive was starting to fail. It was also causing boot times into windows to increase so I had to get rid of it.

After about a month of struggling to get a workable Windows environment I’ve had enough and ordered another SSD for work.

Installing Debian the Naive way

Grab an installer image from https://www.debian.org/distrib/

Hopefully you’ll end up with an ISO (I like torrents), now burn it onto a USB stick.

#> sudo dd if=image.iso of=/dev/<usb> bs=4M
#> sync
  • Boot into the newly created USB by messing around with your BIOS.
  • Install it on one of your newly purchased SDDs.
  • Hapilly start using your newly installed system.

By now you should have a working system and are wondering what all the EFI fuzz is about.

Growing tired of entering BIOS every day

Eventually you should be growing tired of having to go into BIOS to switch which system to boot into. In my particular case my shitty-yet-awesome keyboard decides to roll a dice if it should work pre-boot or not.

So lets figure out how to get a Windows entry into GRUB instead.

Chainloading Windows 8.1

So, lets chainload our Windows 8.1 partition from within GRUB.

What does it look like?

#> fdisk -l /dev/sdb

WARNING: GPT (GUID Partition Table) detected on '/dev/sdb'! The util fdisk doesn't support GPT. Use GNU Parted.

Ah right, GPT. We’re gonna have to use gdisk.

#> gdisk -l /dev/sdb
...
   1            2048          206847   100.0 MiB   EF00  EFI system partition
   2          206848          468991   128.0 MiB   0C01  Microsoft reserved ...
   3          468992         1083391   300.0 MiB   2700  Basic data partition
   4         1083392      1000214527   476.4 GiB   0700  Basic data partition

Nice, but this is pretty far fetched from plain old MBR partition tables.

Time to mount the EFI system partition and find the windows OS loader.

#> sudo mkdir /boot/efi
#> mount /dev/sdb1 /boot/efi
#> find /boot/efi -name bootmgfw.efi

We’ll also need to find the UUID of the EFI partition so that grub knows where to load the OS Loader from.

#> ls -l /dev/disk/by-uuid | grep sdb1
lrwxrwxrwx 1 root root 10 Oct  4 17:10 52A9-DDD7 -> ../../sdb1

Using this information, lets setup a custom boot entry for Windows 8.1.

#> cat /etc/grub.d/40_custom
... snip

menuentry "Windows 8.1" {
    insmod part_gpt
    insmod chain
    search --fs-uuid --no-floppy --set=root 52A9-DDD7
    chainloader /EFI/Microsoft/Boot/bootmgfw.efi
}

Let’s reboot and try it out!

GRUB - Error: Invalid Signature

So trying out the meny item was unsuccesful. Grub gives is an error saying Error: Invalid Signature, what the hell?

After some headscratching we remember the Naive debian installation we recently performed. The GRUB we are using uses MBR and legacy boot. None of the fancy EFI stuff is available to us, least of all the ability to verify OS Loader signatures.

Crap. Time to catch up.

Luckily Ubuntu is trying very hard to become EFI compliant and have done a ton of investigation. All of it documented on their wiki.

After reading through the available information, the gist of it is.

  • You must boot into an EFI ‘environment’ in order to modify it’s variables in NVRAM (what the hell?).
    How this is done depends on your BIOS. (Typically the boot entries that can perform this are prefixed with UEFI:)
  • You must either have enabled ‘Secure Boot’ (correctly signed EFI OS Loader), or disable it in whatever manner is available to your BIOS (if possible).
  • Lastly, GRUB can only chainload an EFI OS Loader if itself is started through an EFI OS Loader.
    It can however also load legacy MBR systems in this mode.

So our goal is: Install GRUB into the EFI system partition so it can boot both our legacy MBR disk, and Windows 8.1 through it’s EFI OS Loader

Installing GRUB into the EFI system partition

We will try to do the following.

  • Boot into a linux system through EFI.
  • chroot into our previously installed Debian system.
  • re-install grub into the EFI system partition.

The best live cd I could find to accomplish this was Ubuntu (>= 12.04.2) 64bit. As per recommendation on their wiki page, this contains the necessary partitions and EFI loader. You can find one here, (the torrent is the fastest option). take care to download a amd64 version.

Burn it to a USB in a similar manner as before and boot.

When messing around in BIOS, make sure that when you boot into it that it indicates that it is trying to boot into UEFI.

Check for the UEFI: prefix or some other indicator.

After boot, check that you have access to the EFI firmware.

#> stat /sys/firmware/efi
  File: ‘/sys/firmware/efi’
  Size: 0               Blocks: 0          IO Block: 4096   directory
...

Cool, lets setup our chroot (assuming /dev/sda1 is your debian partition).

#> sudo mkdir /debian
#> sudo mount /dev/sda1 /debian
#> sudo mount --bind /sys /debian/sys
#> sudo mount --bind /proc /debian/proc
#> sudo mount --bind /dev /debian/dev
#> sudo mount --bind /dev/pts /debian/dev/pts
#> sudo chroot /debian

Now lets reconfigure grub.

#> sudo apt-get install --reinstall grub-efi-amd64

And finally, install grub on the Windows disk with the GPT table.

#> sudo grub-install /dev/sdb

Time to reboot!

You should now be able to find another UEFI: partition in BIOS, this time called Debian.

Make this your default.

Summary

We now have two disks, sda and sdb.

  • sda contains Debian, and is partitioned using MBR.
  • sdb contains Windows 8.1, and is partitioned using GPT to boot through EFI. The EFI system partition contains OS Loaders for GRUB and Windows. GRUB is capable of reading the partition table of sda and its configuration file.

Good luck with your new dual boot system!