Building Rakudo the hard way
One can build Rakudo very easily via rakudobrew
, for instance:
git clone https://github.com/tadzik/rakudobrew ~/.rakudobrew
export PATH=~/.rakudobrew/bin:$PATH
rakudobrew build moar
rakudobrew build-panda
and you’re good to go.
However, what if you want to hack on the toolchain and/or entire software stack itself? How does one go about that? Here are my notes, collected so that I don’t forget how to do it myself and presented here in the hope that they are useful to someone else.
Building the Rakudo stack
The full Rakudo (Perl6) stack is made up of these components (in build order):
- MoarVM
- NQP (not quite Perl)
- Rakudo
There is also panda
which is the most common tool used to install Perl6
modules, effectively the cpanm
of the Perl6 ecosystem. This can also be
loosely considered to be part of the Perl6 stack.
Building MoarVM
To build MoarVM from source, first fork the repository on GitHub, then clone it, configure, build, and install:
git clone git@github.com:MoarVM/MoarVM.git
cd MoarVM
perl Configure.pl --prefix=$HOME/perl6
make
make install # installs into the 'perl6' dir in the user's home
The configuration step will update various submodules from online sources, so you can’t run this step if you’re on the go without internet access.
Note that there is no test suite for MoarVM; this is intentional – it
should be tested via the tools that build upon it, for instance NQP.
Consequently there is no make test
target 1.
The MoarVM binary (moar
) will then be installed into the bin
subdirectory of the prefix directory (in this case $HOME/perl6
). We will
need to refer to this location in later configuration and build steps.
Building NQP
NQP builds on top of MoarVM and often needs a specific version of MoarVM in order to configure or build at all. Since we’ve just cloned a fresh copy this should work straight away. Similarly to MoarVM, fork the repo on GitHub, clone, configure, and build:
git clone git@github.com:perl6/nqp.git
cd nqp
perl Configure.pl --prefix=$HOME/perl6
make
However, since NQP has a test suite, we can run the tests to see if everything works before installing:
make test
make install # assuming, of course, everything went well
The tests also serve as a test of MoarVM, hence this is an important step to pass successfully if hacking on the MoarVM sources. It is also a good idea to build and test Rakudo in order to really make sure that any changes to MoarVM (or NQP if hacking on that) didn’t break anything.
Building Rakudo
The steps to build and install Rakudo are almost identical to those for NQP; fork the repo, clone, configure, build, test and finally install locally:
git clone git@github.com:rakudo/rakudo.git
cd rakudo
perl Configure.pl --prefix=$HOME/perl6
make
make test
make install # assuming, of course, everything went well
If you want to be very thorough, you can also run the full spec test suite:
make spectest
However, this takes a while, so depending upon which parts of the code you’ve been hacking on it might or might not be necessary to run this.
Set up the paths to the perl6 binaries
Of course the binaries need to be accessible once they have been built and
installed. Simply add the $HOME/perl6/bin
directory to your PATH
:
export PATH=$PATH:$HOME/perl6/bin
This can also be added to your ~/.bashrc
so that the PATH
doesn’t have
to set this up in every new shell.
Now that Rakudo has been set up, you should be able to run:
perl6 --version
and you should see something like the following:
This is Rakudo version 2016.08.1-22-g597052f built on MoarVM version 2016.08-11-gedd5839
implementing Perl 6.c.
If so, hooray! you’ve just installed a bleeding edge version of Perl6! If
not, well, there were hopefully reasonably explanatory error messages which
explained what went wrong. If you get really stuck you can always try
asking the friendly people on the #perl6
channel on irc.freenode.net
.
Building and installing panda
In order to install Perl6 modules, there are a couple of packages available
for this purpose. One very popular alternative is panda
(it’s like
cpanm
for Perl6). To set up panda
, fork the repo, clone, and bootstrap:
git clone git@github.com:tadzik/panda.git
cd panda
./bootstrap.pl # this will bootstrap and install panda
This will install panda
in $HOME/perl6/share/perl6/site/bin
, which you
will also need to add to your PATH
:
export PATH=$PATH:$HOME/perl6/share/perl6/site/bin
To install modules, now run
panda install <module-name>
E.g., if you want to install Linenoise
, which will allow you to have
readline-like behaviour in the perl6
REPL, you would run this:
panda install Linenoise
Rebuilding the Rakudo stack
Ok, now that we’ve got an entire stack built, we want to hack code. Or possibly you want to use the development version of Rakudo, which itself requires a specific NQP version, which itself requires a specific MoarVM version. Thus we’ll need to rebuild parts (or all) of the stack in order to check that our changes have worked as expected and that the tests still run (i.e. we haven’t broken anything). Rebuilding the stack sometimes turns out to be harder than one would imagine.
Set up the upstream sources
In order to update the code in the working copies of each of the MoarVM, NQP, Rakudo, and panda projects, we need to set up the location of the respective upstream source repositories. To do this, simply enter the respective working copy directory and add the upstream remote repository along these lines:
cd path/to/MoarVM
git remote add upstream git@github.com:MoarVM/MoarVM.git
cd path/to/nqp
git remote add upstream git@github.com:perl6/nqp.git
cd path/to/rakudo
git remote add upstream git@github.com:rakudo/rakudo.git
cd path/to/panda
git remote add upstream git@github.com:tadzik/panda.git
Rebuilding MoarVM
Now we’re ready to rebuild MoarVM. Enter the MoarVM working copy directory and fetch the latest changes from upstream.
cd MoarVM
git fetch upstream master
git fetch --tags upstream master # just in case all tags aren't fetched
git merge upstream/master
make realclean # sometimes "make distclean" is necessary to really clean up
perl Configure.pl --prefix=$HOME/perl6
make
make install
If you’ve changed the MoarVM C code, then you’ll just need to run:
make clean
make
make install
However, if some of the configuration code has been changed, then a reconfigure is likely necessary:
make clean
perl Configure.pl --prefix=$HOME/perl6
make
make install
By the way, when make
runs, it hides the rather noisy output from the C
compiler. If you want to see this extra information (which can be handy
when hacking on internal stuff) then you need to add the NOISY=1
variable
to the make
call, e.g.:
make NOISY=1
Rebuilding NQP
If MoarVM has been updated, NQP now needs to be built with the new MoarVM version. This is necessary if you want to check changes made to MoarVM by running the NQP test suite or if you have simply updated to the latest MoarVM code.
cd nqp
git fetch upstream master
git fetch --tags upstream master # just in case all tags aren't fetched
git merge upstream/master
make realclean
perl Configure.pl --prefix=$HOME/perl6
make
make test
make install
If you’ve only changed code in NQP, there shouldn’t be a need to update MoarVM, hence you can simply run:
make clean
make
make test
make install
Rebuilding Rakudo
If either MoarVM or NQP have been updated you will also need to update
Rakudo. Enter the rakudo
working directory and update like so:
make realclean
perl Configure.pl --prefix=$HOME/perl6
make
make test
make spectest # optional; run entire spec test suite
make install
If you wish to have the latest and greatest Rakudo code as well, prepend the
fetch
and merge
steps:
git fetch upstream nom
git fetch --tags upstream nom # just in case
git merge upstream/nom
make realclean
perl Configure.pl --prefix=$HOME/perl6
make
make test
make spectest # optional; run entire spec test suite
make install
However, if only code within the rakudo
working directory has changed,
it should only be necessary to rebuild the project, run its test suite and
(if desired) reinstall.
make clean
make
make test
make spectest # optional; run entire spec test suite
make install
General tips
- it is sometimes necessary to remove
$HOME/perl6
directory. There are times when the files within this directory structure don’t get updated correctly and it can be useful to completely delete it and reinstall the stack packages as well as any modules installed viapanda
. - the
realclean
target in MoarVM doesn’t remove themoarvm
binary file. If you need to remove really everything, use thedistclean
target.
Hacking on core
This should be sufficient information to get the environment set up so that one can develop a feature, analyse and/or fix a bug, update docs, or try to reduce technical debt. Then one can check that the changes haven’t broken anything before submitting a pull request to the respective project.
Example
Let’s say we want to check for unused variables in MoarVM. To do this, we
specify the -Wunused-variable
C compiler warning option. We now want to
make sure that the environment is clean before proceeding, so we use
make realclean
to clean up (make distclean
is sometimes necessary, however the
realclean
target should be good enough for our purposes). Then we
configure with the required warnings flag:
CFLAGS=-Wunused-variable perl Configure.pl --prefix=$HOME/perl6
Then we run a “noisy” make
:
make NOISY=1 | tee make.out
which will print all compiler output to stdout (this output is usually
suppressed); the tee
will additionally write this output to the file
make.out
. Now we can hunt through the output in make.out
for warnings
which might need to be fixed.
Let’s say we’ve found an unused variable warning and, after having read the code, realise that yes, the variable is set but isn’t used. We delete it from the source file and recompile:
make realclean && CFLAGS=-Wunused-variable perl Configure.pl --prefix=$HOME/perl6 && make NOISY=1 && make install
In order to test that our change didn’t break anything, we enter the nqp
directory, rebuild nqp
and run its test suite:
cd ../nqp
make realclean perl Configure.pl --prefix=$HOME/perl6 && make && make test
If the tests pass, we can have some increased confidence that nothing broke. Repeating this process for as many unused variable warnings as possible, we can now prepare a pull request to the MoarVM project.
Conclusion
That’s lots of detail, however if you’re hacking code in the Perl6 stack that’s often where the devil is, hence having detailed instructions to update and rebuild it could come in handy. Happy hacking!
-
Actually, there is a
make test
target, however it just tells you to build NQP and test the MoarVM from there. ↩