NEXX HYPERVISOR for ARM architecture

Author:Hector Marco
Started:01 January 2011 - I Started from scratch
Discontinued:31 March 2011 - I Stopped the development

General description

NEXX is a tiny (and incomplete) ARM hypervisor which enables you to run both bare and Linux partitions. The main purpose of NEXX was to learn operating systems internals and hardware programming.

I only spent three months developing the NEXX ARM hypervisor focussed on learning. Code quality was not in the agenda, and so, the resulting code was... well ugly. I'm not proud of the code. Even that, I decided to make it public because from the beginning of 2011 I have had no time to continue it, and it seems that I'll never will.

In these three month the most relevant work that I did it was:

  1. To start from scratch a hypervisor for ARM which:
    • Boot itself and prepare partitions execution (registers, exceptions, etc,.)
    • Provide time and space isolation (hardware and software programming)
    • Activate the MMU because was required for Linux (virtual and physical memory)
    • Implement some hypercalls like nexx_write_console. (hardware programming)
  2. NAL (NEXX abstraction layer) library
    • C support to run standalone partitions (boot.S,, etc,.)
    • Library abstraction to perform hypercalls from partitions (C and assembler)
  3. Patch Linux to run on top of NEXX hypervisor
    • Creation of a new platform (nexx-match) for Linux
    • Modification of the Linux boot process
    • Creation of an early console to print outputs etc,.
    • Built a small initramfs (rdinit=/init)
    • Provided a new configuration Linux file (.config)
  4. Loader which:
    • Unpack resident software (loader + partitions + hypervisor)
    • Load the NEXX hypervisor and the binary partitions
    • Check the loaded memory in RAM against the rsw (md5)
    • Obtain where the hypervisor is loaded and Jump to it

Obtaining and extracting NEXX ARM hypervisor sources

The NEXX ARM hypervisor 0.0.1 sources are available to download directly from my website. To download and extract the sources execute the following commands:
 $ mkdir $HOME/nexx
 $ cd $HOME/nexx
 $ wget
 $ cd NEXX-hypervisor
 $ tar xvf nexx-sources.tgz
The sources contain the following folders:

Preparing environment for building ARM NEXX hypervisor

Before start building the hypervisor, partitions or the Linux kernel, we need to install the GCC ARM compiler to build the hypervisor from sources.

1.- Download and install the GCC ARM compiler in /opt

The NEXX ARM hypervisor is compiled using the none eabi GCC cross-compiler. Following this guide, the GCC compiler will be installed in the "/opt" directory. Some files (, etc.,) expect the compiler in this path.

 $ cd /tmp
 $ wget "\
 $ cd /opt
 $ sudo tar xvf /tmp/arm-2011.09-69-arm-none-eabi-i686-pc-linux-gnu.tar.bz2

3.- Installing some packages on the host machine

We need to manipulate some ARM ELF files (objcopy, nm etc.,) as well as been able to build (make) and run the hypervisor, partitions and the Linux kernel under an ARM machine (qemu). To install these packages on Ubuntu and similar distributions execute the following command in a shell:

 $ sudo apt-get install binutils-multiarch make qemu-system

Compiling and executing bare partitions on NEXX hypervisor

NEXX hypervisor allows run on top of it bare (standalone) partitions. The following is a simple example of new bare partition.

1.- Compiling and testing an examples

Folder "user/partitions" contains two partitions "partition0.c" and "partition1.c" which are a very simple example. They print a message "I am partition 0" and "I am partition 1" respectively by invoking the hypercall NEXX_write_console(). You should be able to see these messages if you compile and execute as follow:

 $ make bare
 $ make test

After executing the last command, Qemu will start and the output should be very similar to:

 [Loader] PackedMemory used: [0x0]--[0xA000]
 [Loader] Loading NEXX hypervisor ...
 [Loader] NEXX [0x608000]--[0x60C060]
 [Loader] Loading Partitions ...
 [Loader] P1 [0xA00000]--[0xA00298]
 [Loader] P2 [0xB00000]--[0xB00298]
 NEXX_MD5(608000 , 16480, hash);
 RAM MD5 [6DC677DC22A278715CA795FC76F8450]
 [Loader] Starting NEXX at 0x608000...
 [NEXX] Starting NEXX Hypervisor ...
 [NEXX] CPU ID:41069265
 [NEXX] CR:90078
 [NEXX] PGT [0x400000]-[0x404000]
 [NEXX] PGT [0xAFC000]-[0xB00000]
 [NEXX] PGT [0xBFC000]-[0xC00000]
 [NEXX] Calling scheduler()
 Run partition id:0
 I am partition 0
 I am partition 0
 I am partition 0
 I am partition 0
 I am partition 0
 [NEXX-LOG] timerIrqHandler(). Context switch ... 
 Run partition id:1
 I am partition 1
 I am partition 1
 I am partition 1
 I am partition 1
 I am partition 1
 [NEXX-LOG] timerIrqHandler(). Context switch ... 

 Run partition id:0
 I am partition 0
 I am partition 0
 I am partition 0
 ... ...

2.- Modifying partition C code

You can build bare partitions with new code. The example partition code can be found in the extracted sources downloaded. The file "user/partitions/partition0.c" and "user/partitions/partition1.c" contains the code of the partition 1 and 2 respectively.

The first "C" function called in a partition is PartitionMain() which does not receive any argument. You can add your code inside this function to build a new partition. After that, you can compile and test the partitions as showed before.

Compiling, patching and executing Linux on top of NEXX

The NEXX ARM hypervisor implements just a few hypercalls. The hypercalls were implemented "on demand" while porting the Linux kernel to NEXX.

In order to run the Linux kernel on top of the hypervisor, the Kernel need to be patched because the NEXX is a para-virtualised hypervisor.

I started to study in deep the boot sequence as well as other internal parts of the Linux kernel in order to understand how it actually works. Then, I was able to create a new "machine" (which I called: nexx-mach). The Linux kernel version at that time was linux-3.1.1.

The steps to run the Linux kernel under NEXX hypervisor are:

  1. Download Linux kernel and uncompress it:
     $ mkdir $HOME/linux-sources
     $ cd $HOME/linux-sources
     $ wget
     $ tar xvf linux-3.1.1.tar.xz  
  2. Patch the Linux kernel to be used on top of NEXX
     $ wget
     $ cd linux-3.1.1
     $ patch -p2 < ../linux-3.1.1-nexx.patch   
  3. Configure the Linux kernel properly for NEXX hypervisor
     $ wget
     $ mv linux-3.1.1-nexx.config .config   
  4. Compile Linux Kernel for ARM from x86_64 We need to install the gcc-arm-linux-gnueabi to compile the Linux kernel for ARM. In Ubuntu and similar distributions we can install this compiler using the "apt-get install" command.
     $ sudo apt-get install gcc-arm-linux-gnueabi   
  5. Created a simple initramfs with a "Hello world" init process. Although the patch is not finished, the patched Linux start to execute the init process. A point to continue the development of the patch is by following the execution of init process. For this reason I decided substitute the init process by a simple print "Hello world" init process. To achieve this we need to do the following:
     $ mkdir $HOME/initrd
     $ cd $HOME/initrd
     $ wget
     $ arm-linux-gnueabi-gcc -static init.c -o init  
  6. Compile patched Linux kernel
     $ cd $HOME/linux-sources/linux-3.1.1
     $ export ARCH=arm
     $ export CROSS_COMPILE=arm-linux-gnueabi-
     $ make vmlinux
  7. Compile NEXX hypervisor and test the built kernel
     $ cd $HOME/nexx/NEXX-hypervisor
     $ make linux
     $ make test
    The output should be very similar to:
     [Loader] PackedMemory used: [0x0]--[0x36C500]
     [Loader] Loading NEXX hypervisor ...
     [Loader] NEXX [0x608000]--[0x60BFF8]
     [Loader] Loading Partitions ...
     [Loader] P1 [0xA00000]--[0xA00298]
     [Loader] P2 [0xB00000]--[0xB00298]
     [Loader] Linux ...
     [Loader] Linux [0xC08000]--[0xF6A484]
     NEXX_MD5(608000 , 16376, hash);
     RAM MD5 [3043D7C239ADCD2BF153B50BA388798]
     [Loader] Starting NEXX at 0x608000...
     [NEXX] Starting NEXX Hypervisor ...
     [NEXX] CPU ID:41069265
     [NEXX] CR:90078
     [NEXX] PGT [0x400000]-[0x404000]
     [NEXX] Calling scheduler()
     Run partition id:0
     console [nex2] enabled
     NEX_read_cpuid == 0x41069265
     Linux version 3.1.1 (work@work-pc) (gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) )
     CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00000000
     CPU: VIVT data cache, VIVT instruction cache
     WARNING !!!!: NOT setup stacks for re-entrant exception handlers
     __atags_pointer -> c0000100
     To execute setup_machine_tags(machine_arch_type(183))
     Machine: Virtualized ARM-Versatile PB
     Memory policy: ECC disabled, Data cache writeback
     Kernel command line1: mem=64M console=tty root=/dev/ram
     Built 1 zonelists in Zone order, mobility grouping on.  Total pages: 16256
     Kernel command line: mem=64M console=tty root=/dev/ram
     PID hash table entries: 256 (order: -2, 1024 bytes)
     Dentry cache hash table entries: 8192 (order: 3, 32768 bytes)
     Inode-cache hash table entries: 4096 (order: 2, 16384 bytes)
     Memory: 64MB = 64MB total
     Memory: 61308k/61308k available, 4228k reserved, 0K highmem
     Virtual kernel memory layout:
         vector  : 0xffff0000 - 0xffff1000   (   4 kB)
         fixmap  : 0xfff00000 - 0xfffe0000   ( 896 kB)
         DMA     : 0xffc00000 - 0xffe00000   (   2 MB)
         vmalloc : 0xc4800000 - 0xd8000000   ( 312 MB)
         lowmem  : 0xc0000000 - 0xc4000000   (  64 MB)
         modules : 0xbf000000 - 0xc0000000   (  16 MB)
           .text : 0xc0008000 - 0xc02f0144   (2977 kB)
           .init : 0xc02f1000 - 0xc0350000   ( 380 kB)
           .data : 0xc0350000 - 0xc036a460   ( 106 kB)
            .bss : 0xc036a484 - 0xc0384100   ( 104 kB)
     start_kernel(): bug: interrupts were enabled *very* early, fixing it
     NEX nex_init_irq
     NEX nex_timer_init
     start_kernel(): bug: interrupts were enabled early
     console_init !!!
     Console: colour dummy device 80x30
     console [tty0] enabled
     pid_max: default: 32768 minimum: 301
     Mount-cache hash table entries: 512
     NET: Registered protocol family 16
     NEX versatile_pb_init
     bio: create slab <bio-0> at 0
     **** [log punxos]  tty_init ****
     NET: Registered protocol family 2
     IP route cache hash table entries: 1024 (order: 0, 4096 bytes)
     TCP established hash table entries: 2048 (order: 2, 16384 bytes)
     TCP bind hash table entries: 2048 (order: 1, 8192 bytes)
     TCP: Hash tables configured (established 2048 bind 2048)
     TCP reno registered
     UDP hash table entries: 256 (order: 0, 4096 bytes)
     UDP-Lite hash table entries: 256 (order: 0, 4096 bytes)
     NET: Registered protocol family 1
     RPC: Registered named UNIX socket transport module.
     RPC: Registered udp transport module.
     RPC: Registered tcp transport module.
     RPC: Registered tcp NFSv4.1 backchannel transport module.
     NetWinder Floating Point Emulator V0.97 (double precision)
     Installing knfsd (copyright (C) 1996
     JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc.
     ROMFS MTD (C) 2007 Red Hat, Inc.
     msgmni has been set to 119
     Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254)
     io scheduler noop registered
     io scheduler deadline registered
     io scheduler cfq registered (default)
     brd: module loaded
     mousedev: PS/2 mouse device common for all mice
     TCP cubic registered
     NET: Registered protocol family 17
     VFP support v0.3: implementor 41 architecture 1 part 10 variant 9 rev 0
     Warning: unable to open an initial console.
     [NEX-linux log] Exec ... /init
     sys_access(/init) returned: 0
     [NEX-Linux log]
     Ok, we have completed the initial bootup, and
     we're essentially up and running. Get rid of the
     initmem segments and start the user-mode stuff..
     [Nex-Linux log] Halt hypervisor => While(1)


I successfully built from scratch a proof of concept hypervisor for ARM. In addition, I dealt with bare (standalone) partitions which required to solve problems like having a valid stack or to map virtual and physical memory or to build new libraries, just to mention a few.

A notable case was the Linux kernel patch that although it was not finished, it was a very refreshing and challenging goal. It worth to mention that the lack of virtualization infrastructure in the Linux ARM had make the work more "interesting".

To conclude, I successfully addressed a lot of challenges, so I'm proud to say that I achieved my goals :)

Hector Marco -