Building Rakudo the hard way

One can build Rakudo very easily via rakudobrew, for instance:

git clone https://github.com/tadzik/rakudobrew ~/.rakudobrew
export PATH=~/.rakudobrew/bin:$PATH
rakudobrew build moar
rakudobrew build-panda

and you’re good to go.

However, what if you want to hack on the toolchain and/or entire software stack itself? How does one go about that? Here are my notes, collected so that I don’t forget how to do it myself and presented here in the hope that they are useful to someone else.

Building the Rakudo stack

The full Rakudo (Perl6) stack is made up of these components (in build order):

• MoarVM
• NQP (not quite Perl)
• Rakudo

There is also panda which is the most common tool used to install Perl6 modules, effectively the cpanm of the Perl6 ecosystem. This can also be loosely considered to be part of the Perl6 stack.

Building MoarVM

To build MoarVM from source, first fork the repository on GitHub, then clone it, configure, build, and install:

git clone git@github.com:MoarVM/MoarVM.git
cd MoarVM
perl Configure.pl --prefix=$HOME/perl6
make
make install  # installs into the 'perl6' dir in the user's home

The configuration step will update various submodules from online sources, so you can’t run this step if you’re on the go without internet access.

Note that there is no test suite for MoarVM; this is intentional – it should be tested via the tools that build upon it, for instance NQP. Consequently there is no make test target ¹.

The MoarVM binary (moar) will then be installed into the bin subdirectory of the prefix directory (in this case $HOME/perl6). We will need to refer to this location in later configuration and build steps.

Building NQP

NQP builds on top of MoarVM and often needs a specific version of MoarVM in order to configure or build at all. Since we’ve just cloned a fresh copy this should work straight away. Similarly to MoarVM, fork the repo on GitHub, clone, configure, and build:

git clone git@github.com:perl6/nqp.git
cd nqp
perl Configure.pl --prefix=$HOME/perl6
make

However, since NQP has a test suite, we can run the tests to see if everything works before installing:

make test
make install  # assuming, of course, everything went well

The tests also serve as a test of MoarVM, hence this is an important step to pass successfully if hacking on the MoarVM sources. It is also a good idea to build and test Rakudo in order to really make sure that any changes to MoarVM (or NQP if hacking on that) didn’t break anything.

Building Rakudo

The steps to build and install Rakudo are almost identical to those for NQP; fork the repo, clone, configure, build, test and finally install locally:

git clone git@github.com:rakudo/rakudo.git
cd rakudo
perl Configure.pl --prefix=$HOME/perl6
make
make test
make install  # assuming, of course, everything went well

If you want to be very thorough, you can also run the full spec test suite:

make spectest

However, this takes a while, so depending upon which parts of the code you’ve been hacking on it might or might not be necessary to run this.

Set up the paths to the perl6 binaries

Of course the binaries need to be accessible once they have been built and installed. Simply add the $HOME/perl6/bin directory to your PATH:

export PATH=$PATH:$HOME/perl6/bin

This can also be added to your ~/.bashrc so that the PATH doesn’t have to set this up in every new shell.

Now that Rakudo has been set up, you should be able to run:

perl6 --version

and you should see something like the following:

This is Rakudo version 2016.08.1-22-g597052f built on MoarVM version 2016.08-11-gedd5839
implementing Perl 6.c.

If so, hooray! you’ve just installed a bleeding edge version of Perl6! If not, well, there were hopefully reasonably explanatory error messages which explained what went wrong. If you get really stuck you can always try asking the friendly people on the #perl6 channel on irc.freenode.net.

Building and installing panda

In order to install Perl6 modules, there are a couple of packages available for this purpose. One very popular alternative is panda (it’s like cpanm for Perl6). To set up panda, fork the repo, clone, and bootstrap:

git clone git@github.com:tadzik/panda.git
cd panda
./bootstrap.pl  # this will bootstrap and install panda

This will install panda in $HOME/perl6/share/perl6/site/bin, which you will also need to add to your PATH:

export PATH=$PATH:$HOME/perl6/share/perl6/site/bin

To install modules, now run

panda install <module-name>

E.g., if you want to install Linenoise, which will allow you to have readline-like behaviour in the perl6 REPL, you would run this:

panda install Linenoise

Rebuilding the Rakudo stack

Ok, now that we’ve got an entire stack built, we want to hack code. Or possibly you want to use the development version of Rakudo, which itself requires a specific NQP version, which itself requires a specific MoarVM version. Thus we’ll need to rebuild parts (or all) of the stack in order to check that our changes have worked as expected and that the tests still run (i.e. we haven’t broken anything). Rebuilding the stack sometimes turns out to be harder than one would imagine.

Set up the upstream sources

In order to update the code in the working copies of each of the MoarVM, NQP, Rakudo, and panda projects, we need to set up the location of the respective upstream source repositories. To do this, simply enter the respective working copy directory and add the upstream remote repository along these lines:

cd path/to/MoarVM
git remote add upstream git@github.com:MoarVM/MoarVM.git

cd path/to/nqp
git remote add upstream git@github.com:perl6/nqp.git

cd path/to/rakudo
git remote add upstream git@github.com:rakudo/rakudo.git

cd path/to/panda
git remote add upstream git@github.com:tadzik/panda.git

Rebuilding MoarVM

Now we’re ready to rebuild MoarVM. Enter the MoarVM working copy directory and fetch the latest changes from upstream.

cd MoarVM
git fetch upstream master
git fetch --tags upstream master  # just in case all tags aren't fetched
git merge upstream/master
make realclean  # sometimes "make distclean" is necessary to really clean up
perl Configure.pl --prefix=$HOME/perl6
make
make install

If you’ve changed the MoarVM C code, then you’ll just need to run:

make clean
make
make install

However, if some of the configuration code has been changed, then a reconfigure is likely necessary:

make clean
perl Configure.pl --prefix=$HOME/perl6
make
make install

By the way, when make runs, it hides the rather noisy output from the C compiler. If you want to see this extra information (which can be handy when hacking on internal stuff) then you need to add the NOISY=1 variable to the make call, e.g.:

make NOISY=1

Rebuilding NQP

If MoarVM has been updated, NQP now needs to be built with the new MoarVM version. This is necessary if you want to check changes made to MoarVM by running the NQP test suite or if you have simply updated to the latest MoarVM code.

cd nqp
git fetch upstream master
git fetch --tags upstream master  # just in case all tags aren't fetched
git merge upstream/master
make realclean
perl Configure.pl --prefix=$HOME/perl6
make
make test
make install

If you’ve only changed code in NQP, there shouldn’t be a need to update MoarVM, hence you can simply run:

make clean
make
make test
make install

Rebuilding Rakudo

If either MoarVM or NQP have been updated you will also need to update Rakudo. Enter the rakudo working directory and update like so:

make realclean
perl Configure.pl --prefix=$HOME/perl6
make
make test
make spectest   # optional; run entire spec test suite
make install

If you wish to have the latest and greatest Rakudo code as well, prepend the fetch and merge steps:

git fetch upstream nom
git fetch --tags upstream nom  # just in case
git merge upstream/nom
make realclean
perl Configure.pl --prefix=$HOME/perl6
make
make test
make spectest   # optional; run entire spec test suite
make install

However, if only code within the rakudo working directory has changed, it should only be necessary to rebuild the project, run its test suite and (if desired) reinstall.

make clean
make
make test
make spectest   # optional; run entire spec test suite
make install

General tips

• it is sometimes necessary to remove $HOME/perl6 directory. There are times when the files within this directory structure don’t get updated correctly and it can be useful to completely delete it and reinstall the stack packages as well as any modules installed via panda.
• the realclean target in MoarVM doesn’t remove the moarvm binary file. If you need to remove really everything, use the distclean target.

Hacking on core

This should be sufficient information to get the environment set up so that one can develop a feature, analyse and/or fix a bug, update docs, or try to reduce technical debt. Then one can check that the changes haven’t broken anything before submitting a pull request to the respective project.

Example

Let’s say we want to check for unused variables in MoarVM. To do this, we specify the -Wunused-variable C compiler warning option. We now want to make sure that the environment is clean before proceeding, so we use

make realclean

to clean up (make distclean is sometimes necessary, however the realclean target should be good enough for our purposes). Then we configure with the required warnings flag:

CFLAGS=-Wunused-variable perl Configure.pl --prefix=$HOME/perl6

Then we run a “noisy” make:

make NOISY=1 | tee make.out

which will print all compiler output to stdout (this output is usually suppressed); the tee will additionally write this output to the file make.out. Now we can hunt through the output in make.out for warnings which might need to be fixed.

Let’s say we’ve found an unused variable warning and, after having read the code, realise that yes, the variable is set but isn’t used. We delete it from the source file and recompile:

make realclean && CFLAGS=-Wunused-variable perl Configure.pl --prefix=$HOME/perl6 && make NOISY=1 && make install

In order to test that our change didn’t break anything, we enter the nqp directory, rebuild nqp and run its test suite:

cd ../nqp
make realclean perl Configure.pl --prefix=$HOME/perl6 && make && make test

If the tests pass, we can have some increased confidence that nothing broke. Repeating this process for as many unused variable warnings as possible, we can now prepare a pull request to the MoarVM project.

Conclusion

That’s lots of detail, however if you’re hacking code in the Perl6 stack that’s often where the devil is, hence having detailed instructions to update and rebuild it could come in handy. Happy hacking!

1. Actually, there is a make test target, however it just tells you to build NQP and test the MoarVM from there. ↩