Author: | Hector Marco |
Started: | 01 January 2011 - I Started from scratch |
Discontinued: | 31 March 2011 - I Stopped the development |
NEXX is a tiny (and incomplete) ARM hypervisor which enables you to run both bare and Linux partitions. The main purpose of NEXX was to learn operating systems internals and hardware programming.
I only spent three months developing the NEXX ARM hypervisor focussed on learning. Code quality was not in the agenda, and so, the resulting code was... well ugly. I'm not proud of the code. Even that, I decided to make it public because from the beginning of 2011 I have had no time to continue it, and it seems that I'll never will.
In these three month the most relevant work that I did it was:
$ mkdir $HOME/nexx $ cd $HOME/nexx $ wget http://hmarco.org/virtualisation/nexx/nexx-0.0.1-sources.tgz $ cd NEXX-hypervisor $ tar xvf nexx-sources.tgzThe sources contain the following folders:
Before start building the hypervisor, partitions or the Linux kernel, we need to install the GCC ARM compiler to build the hypervisor from sources.
The NEXX ARM hypervisor is compiled using the none eabi GCC cross-compiler. Following this guide, the GCC compiler will be installed in the "/opt" directory. Some files (config.mk, etc.,) expect the compiler in this path.
$ cd /tmp $ wget "https://sourcery.mentor.com/sgpp/lite/arm/portal/package9740/public/arm-none-eabi/arm-20\ 11.09-69-arm-none-eabi-i686-pc-linux-gnu.tar.bz2" $ cd /opt $ sudo tar xvf /tmp/arm-2011.09-69-arm-none-eabi-i686-pc-linux-gnu.tar.bz2
We need to manipulate some ARM ELF files (objcopy, nm etc.,) as well as been able to build (make) and run the hypervisor, partitions and the Linux kernel under an ARM machine (qemu). To install these packages on Ubuntu and similar distributions execute the following command in a shell:
$ sudo apt-get install binutils-multiarch make qemu-system
Folder "user/partitions" contains two partitions "partition0.c" and "partition1.c" which are a very simple example. They print a message "I am partition 0" and "I am partition 1" respectively by invoking the hypercall NEXX_write_console(). You should be able to see these messages if you compile and execute as follow:
$ make bare $ make test
After executing the last command, Qemu will start and the output should be very similar to:
[Loader] PackedMemory used: [0x0]--[0xA000] [Loader] Loading NEXX hypervisor ... [Loader] NEXX [0x608000]--[0x60C060] [Loader] Loading Partitions ... [Loader] P1 [0xA00000]--[0xA00298] [Loader] P2 [0xB00000]--[0xB00298] NEXX_MD5(608000 , 16480, hash); RAM MD5 [6DC677DC22A278715CA795FC76F8450] [Loader] Starting NEXX at 0x608000... [NEXX] Starting NEXX Hypervisor ... [NEXX] CPU ID:41069265 [NEXX] CR:90078 [NEXX] PGT [0x400000]-[0x404000] [NEXX] PGT [0xAFC000]-[0xB00000] [NEXX] PGT [0xBFC000]-[0xC00000] [NEXX] Calling scheduler() Run partition id:0 I am partition 0 I am partition 0 I am partition 0 I am partition 0 I am partition 0 [NEXX-LOG] timerIrqHandler(). Context switch ... Run partition id:1 I am partition 1 I am partition 1 I am partition 1 I am partition 1 I am partition 1 [NEXX-LOG] timerIrqHandler(). Context switch ... Run partition id:0 I am partition 0 I am partition 0 I am partition 0 ... ...
You can build bare partitions with new code. The example partition code can be found in the extracted sources downloaded. The file "user/partitions/partition0.c" and "user/partitions/partition1.c" contains the code of the partition 1 and 2 respectively.
The first "C" function called in a partition is PartitionMain() which does not receive any argument. You can add your code inside this function to build a new partition. After that, you can compile and test the partitions as showed before.
The NEXX ARM hypervisor implements just a few hypercalls. The hypercalls were implemented "on demand" while porting the Linux kernel to NEXX.
In order to run the Linux kernel on top of the hypervisor, the Kernel need to be patched because the NEXX is a para-virtualised hypervisor.
I started to study in deep the boot sequence as well as other internal parts of the Linux kernel in order to understand how it actually works. Then, I was able to create a new "machine" (which I called: nexx-mach). The Linux kernel version at that time was linux-3.1.1.
The steps to run the Linux kernel under NEXX hypervisor are:
$ mkdir $HOME/linux-sources $ cd $HOME/linux-sources $ wget https://www.kernel.org/pub/linux/kernel/v3.x/linux-3.1.1.tar.xz $ tar xvf linux-3.1.1.tar.xz
$ wget http://hmarco.org/virtualisation/nexx/linux-3.1.1-nexx.patch $ cd linux-3.1.1 $ patch -p2 < ../linux-3.1.1-nexx.patch
$ wget http://hmarco.org/virtualisation/nexx/linux-3.1.1-nexx.config $ mv linux-3.1.1-nexx.config .config
$ sudo apt-get install gcc-arm-linux-gnueabi
$ mkdir $HOME/initrd $ cd $HOME/initrd $ wget http://hmarco.org/virtualisation/nexx/init.c $ arm-linux-gnueabi-gcc -static init.c -o init
$ cd $HOME/linux-sources/linux-3.1.1 $ export ARCH=arm $ export CROSS_COMPILE=arm-linux-gnueabi- $ make vmlinux
$ cd $HOME/nexx/NEXX-hypervisor $ make linux $ make testThe output should be very similar to:
[Loader] PackedMemory used: [0x0]--[0x36C500] [Loader] Loading NEXX hypervisor ... [Loader] NEXX [0x608000]--[0x60BFF8] [Loader] Loading Partitions ... [Loader] P1 [0xA00000]--[0xA00298] [Loader] P2 [0xB00000]--[0xB00298] [Loader] Linux ... [Loader] Linux [0xC08000]--[0xF6A484] NEXX_MD5(608000 , 16376, hash); RAM MD5 [3043D7C239ADCD2BF153B50BA388798] [Loader] Starting NEXX at 0x608000... [NEXX] Starting NEXX Hypervisor ... [NEXX] CPU ID:41069265 [NEXX] CR:90078 [NEXX] PGT [0x400000]-[0x404000] [NEXX] Calling scheduler() Run partition id:0 console [nex2] enabled NEX_read_cpuid == 0x41069265 Linux version 3.1.1 (work@work-pc) (gcc version 4.7.3 (Ubuntu/Linaro 4.7.3-1ubuntu1) ) CPU: ARM926EJ-S [41069265] revision 5 (ARMv5TEJ), cr=00000000 CPU: VIVT data cache, VIVT instruction cache WARNING !!!!: NOT setup stacks for re-entrant exception handlers __atags_pointer -> c0000100 To execute setup_machine_tags(machine_arch_type(183)) Machine: Virtualized ARM-Versatile PB Memory policy: ECC disabled, Data cache writeback Kernel command line1: mem=64M console=tty root=/dev/ram Built 1 zonelists in Zone order, mobility grouping on. Total pages: 16256 Kernel command line: mem=64M console=tty root=/dev/ram PID hash table entries: 256 (order: -2, 1024 bytes) Dentry cache hash table entries: 8192 (order: 3, 32768 bytes) Inode-cache hash table entries: 4096 (order: 2, 16384 bytes) Memory: 64MB = 64MB total Memory: 61308k/61308k available, 4228k reserved, 0K highmem Virtual kernel memory layout: vector : 0xffff0000 - 0xffff1000 ( 4 kB) fixmap : 0xfff00000 - 0xfffe0000 ( 896 kB) DMA : 0xffc00000 - 0xffe00000 ( 2 MB) vmalloc : 0xc4800000 - 0xd8000000 ( 312 MB) lowmem : 0xc0000000 - 0xc4000000 ( 64 MB) modules : 0xbf000000 - 0xc0000000 ( 16 MB) .text : 0xc0008000 - 0xc02f0144 (2977 kB) .init : 0xc02f1000 - 0xc0350000 ( 380 kB) .data : 0xc0350000 - 0xc036a460 ( 106 kB) .bss : 0xc036a484 - 0xc0384100 ( 104 kB) start_kernel(): bug: interrupts were enabled *very* early, fixing it NR_IRQS:192 NEX nex_init_irq NEX nex_timer_init start_kernel(): bug: interrupts were enabled early console_init !!! Console: colour dummy device 80x30 console [tty0] enabled pid_max: default: 32768 minimum: 301 Mount-cache hash table entries: 512 NET: Registered protocol family 16 NEX versatile_pb_init bio: create slab <bio-0> at 0 **** [log punxos] tty_init **** NET: Registered protocol family 2 IP route cache hash table entries: 1024 (order: 0, 4096 bytes) TCP established hash table entries: 2048 (order: 2, 16384 bytes) TCP bind hash table entries: 2048 (order: 1, 8192 bytes) TCP: Hash tables configured (established 2048 bind 2048) TCP reno registered UDP hash table entries: 256 (order: 0, 4096 bytes) UDP-Lite hash table entries: 256 (order: 0, 4096 bytes) NET: Registered protocol family 1 RPC: Registered named UNIX socket transport module. RPC: Registered udp transport module. RPC: Registered tcp transport module. RPC: Registered tcp NFSv4.1 backchannel transport module. NetWinder Floating Point Emulator V0.97 (double precision) Installing knfsd (copyright (C) 1996 okir@monad.swb.de). JFFS2 version 2.2. (NAND) © 2001-2006 Red Hat, Inc. ROMFS MTD (C) 2007 Red Hat, Inc. msgmni has been set to 119 Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254) io scheduler noop registered io scheduler deadline registered io scheduler cfq registered (default) brd: module loaded mousedev: PS/2 mouse device common for all mice TCP cubic registered NET: Registered protocol family 17 VFP support v0.3: implementor 41 architecture 1 part 10 variant 9 rev 0 Warning: unable to open an initial console. [NEX-linux log] Exec ... /init sys_access(/init) returned: 0 [NEX-Linux log] Ok, we have completed the initial bootup, and we're essentially up and running. Get rid of the initmem segments and start the user-mode stuff.. [Nex-Linux log] Halt hypervisor => While(1)
I successfully built from scratch a proof of concept hypervisor for ARM. In addition, I dealt with bare (standalone) partitions which required to solve problems like having a valid stack or to map virtual and physical memory or to build new libraries, just to mention a few.
A notable case was the Linux kernel patch that although it was not finished, it was a very refreshing and challenging goal. It worth to mention that the lack of virtualization infrastructure in the Linux ARM had make the work more "interesting".
To conclude, I successfully addressed a lot of challenges, so I'm proud to say that I achieved my goals :)
Hector Marco - http://hmarco.org