2 % This document is released under the GPL
3 % Initially written by Stefan Reinauer, <stepan@openbios.org>
6 \documentclass[titlepage,12pt]{article}
10 \usepackage[pdftex]{hyperref}
11 % \usepackage{makeidx}
15 urlbordercolor={1 1 1},
16 menubordercolor={1 1 1},
17 linkbordercolor={1 1 1},
19 % pdfpagemode=None, % PDF-Viewer starts without TOC
21 pdftitle={LinuxBIOS on AMD64},
22 pdfauthor={Stefan Reinauer},
23 pdfsubject={LinuxBIOS configuration and build process},
24 pdfkeywords={LinuxBIOS, Opteron, AMD64, Athlon64, Build}
28 % \newcommand{\sh}[1]{\begin{verbatim}\texttt{#1}\end{verbatim}}
29 % \newcommand{\prog}[1]{\textit{#1}}
31 \setlength{\parindent}{0pt}
33 \title{LinuxBIOS on AMD64}
34 \author{Stefan Reinauer $<$stepan@openbios.org$>$}
35 \date{February 10th, 2004}
53 This document targets porting LinuxBIOS to new Motherboards and creating
54 custom firmware images using LinuxBIOS. It describes how to build
55 LinuxBIOS images for the AMD64 platform, including hypertransport
56 configuration and pertinent utilities. If you are missing information or
57 find errors in the following descriptions, contact
58 \href{mailto:stepan@openbios.org}{\textit{Stefan Reinauer $<$stepan@openbios.org$>$}}
67 \item 2003/11/18 initial release
68 \item 2004/02/10 acpi and option rom updates
77 \section{What is LinuxBIOS?}
79 LinuxBIOS aims to replace the normal BIOS found on PCs, Alphas, and
80 other machines with a Linux kernel that can boot Linux from a cold
81 start. The startup code of an average LinuxBIOS port is about 500 lines
82 of assembly and 5000 lines of C. It executes 16 instructions to get into
83 32bit mode and then performs DRAM and other hardware initializations
84 required before Linux can take over.
86 The projects primary motivation initially was maintenance of large
87 clusters. Not surprisingly interest and contributions have come from
88 people with varying backgrounds. Nowadays a large and growing number of
89 Systems can be booted with LinuxBIOS, including embedded systems,
90 Desktop PCs and Servers.
93 % 4 Build Requirements
96 \section{Build Requirements}
97 To build LinuxBIOS for AMD64 from the sources you need a recent Linux
98 system for x86 or AMD64. SUSE Linux 8.2 or 9.0 are known to work fine.
99 The following toolchain is recommended:
103 \item binutils 2.14.90.0.5
108 \textbf{NOTE:} Later versions should also work. Prior versions might lead to problems.
113 % 5 Getting the Sources
116 \section{Getting the Sources}
118 The latest LinuxBIOS sources are available via CVS. The CVS repository
119 is maintained at SourceForge.net (the project name is \emph{FreeBIOS}). You
120 can get the entire source tree via CVS:
124 % cvs -d:pserver:anonymous@cvs.freebios.sourceforge.net:/cvsroot/freebios login
128 Hit return when you are asked for a password. Then checkout (or update)
129 the freebios source tree as follows:
133 % cvs -z3 -d:pserver:anonymous@cvs.freebios.sourceforge.net:/cvsroot/freebios co freebios2
137 Once the source tree is checked out, it can be updated with:
145 Due to recent problems with SourceForge's CVS infrastructure we set up a
146 snapshot site that keeps hourly source trees of the last four days. It
147 is available at \url{http://snapshots.linuxbios.org/}.
148 Due to major structural enhancements to \hbox{LinuxBIOS}, AMD64 support
149 is only available in the \texttt{freebios2} tree. This tree reflects (as
150 of November 2003) LinuxBIOS version 1.1.5 and will lead to LinuxBIOS 2.0
151 when finished. Most x86 hardware is currently only supported by the
155 % 6 LinuxBIOS configuration overview
158 \section{LinuxBIOS configuration overview}
159 To support a large variety of existing hardware LinuxBIOS allows for a
160 lot of configuration options that can be tweaked in several ways:
164 Firmware image specific configuration options can be set in the image
165 configuration file which is usually found in
166 \texttt{freebios2/targets/$<$vendor$>$/$<$motherboard$>$/}. Such
167 options are the default amount of output verbosity during bootup, image
168 size, use of fallback mechanisms, firmware image size and payloads
169 (Linux Kernel, Bootloader...) The default configuration file name is
170 \texttt{Config.lb}, but LinuxBIOS allows multiple configurations to
171 reside in that directory.
173 \item Motherboard specific configuration options can be set in the
174 motherboard configuration file placed in
175 \texttt{freebios2/src/mainboard/$<$vendor$>$/$<$motherboard$>$}. The
176 motherboard configuration file is always called \texttt{Config.lb}. It
177 contains information on the onboard components of the motherboard like
178 CPU type, northbridge, southbridge, hypertransport configuration and
179 SuperIO configuration. This configuration file also allows to include
180 addon code to hook into the LinuxBIOS initialization mechanism at
185 This document describes different approaches of changing and configuring the
186 LinuxBIOS source tree when building for AMD64.
189 % 7 Building LinuxBIOS
192 \section{Building LinuxBIOS}
193 One of the design goals for building LinuxBIOS was to keep object files
194 out of the source tree in a seperate place. This is mandatory for
195 building parallel LinuxBIOS images for several distinct motherboards
196 and/or platforms. Therefore building LinuxBIOS consists of two steps:
199 creating a build tree which holds all files automatically created by the
200 configuration utility and the object files
202 compiling the LinuxBIOS code and creating a flashable firmware image.
205 The first of these two steps is accomplished by the \texttt{buildtarget}
206 script found in \texttt{freebios2/targets/}. To build LinuxBIOS for
207 instance for the AMD Solo Athlon64 motherboard enter:
210 % cd freebios2/targets
211 % ./buildtarget amd/solo
214 This will create a directory containing a Makefile and other software
215 components needed for this build. The directory name is defined in the
216 firmware image specific configuration file. In the case of AMD's Solo
217 motherboard the default directory resides in
218 \texttt{freebios2/targets/amd/solo/solo}. To build the LinuxBIOS image, do
225 The LinuxBIOS image filename is specified in the firmware image specific
226 configuration file. The default filename for AMD's Solo motherboard is
230 % 8 Programming LinuxBIOS to flash memory
233 \section{Programming LinuxBIOS to flash memory}
234 The image resulting from a LinuxBIOS build can be directly programmed to
235 a flash device, either using a hardware flash programmer or by using the
236 Linux flash driver devbios or mtd. This document assumes that you use a
237 hardware flash programmer. If you are interested in doing in-system
238 software flash programming, find detailed information:
241 \item \url{http://www.openbios.org/development/devbios.html} (/dev/bios)
242 \item \url{http://www.linuxmtd.infradead.org/} (Memory Technology Device Subsystem MTD)
248 % 9 LinuxBIOS configuration
251 \section{LinuxBIOS configuration}
252 The following chapters will cope with configuring LinuxBIOS. All
253 configuration files share some basic rules
256 The default configuration file name in LinuxBIOS is \texttt{Config.lb}.
258 All variables used in a configuration file have to be declared in this
259 file with \texttt{uses VARNAME} before usage.
261 Comments can be added on a new line by using the comment identifier
262 \texttt{\#} at the beginning of the line.
264 LinuxBIOS distinguishes between statements and options. Statements cause
265 the LinuxBIOS configuration mechanism to act, whereas options set
266 variables that are used by the build scripts or source code.
268 Default configuration values can be set in the motherboard configuration
269 files (keyword default)
271 Option overrides to the default configuration can only be specified in
272 the build target configuration file
273 \texttt{freebios2/targets/$<$vendor$>$/$<$mainboard$>$/Config.lb}
277 \subsection{Common Configuration Statements}
281 \item \begin{verbatim}uses\end{verbatim}
283 All local configuration variables have to be declared before they can be
289 \textbf{NOTE:} Only configuration variables known to the configuration
290 system can be used in configuration files. LinuxBIOS checks
291 \texttt{freebios2/src/config/Options.lb} to see whether a configuration
294 \item \begin{verbatim}default\end{verbatim}
296 The \texttt{default} statement is used to set a configuration variable
297 with an overridable default value. It is commonly used in motherboard
303 default ROM_IMAGE_SIZE=0x10000
306 It is also possible to assign the value of one configuration variable to
310 default FALLBACK_SIZE=ROM_SIZE
313 Also, simple expressions are allowed:
316 default FALLBACK_SIZE=(ROM_SIZE - NORMAL_SIZE)
319 If an option contains a string, this string has to be protected with
323 default CC="gcc m32"
326 \item \begin{verbatim}option\end{verbatim}
328 The \texttt{option} statement basically behaves identically to the
329 \texttt{default} statement. But unlike default it can only be used in
330 build target configuration files
331 (\texttt{freebios2/targets/$<$vendor$>$/$<$mainboard$>$}). The option
332 statement allows either to set new options or to override default values
333 set with the default statement in a motherboard configuration file.
334 Syntax and application are the same as with default.
338 \subsection{Firmware image specific configuration}
339 LinuxBIOS allows to create different firmware images for the same
340 hardware. Such images can differ in the amount of output they produce,
341 the payload, the number of subimages they consist of etc.
343 The firmware image specific configuration file can be found in
344 \texttt{freebios2/targets/$<$vendor$>$/<motherboard$>$}.
346 \subsubsection{Firmware image specific keywords}
347 In addition to the above described keywords the following statements are
348 available in firmware image specific configuration files:
351 \item \begin{verbatim}romimage\end{verbatim}
353 The \texttt{romimage} definition describes a single rom build within the
354 final LinuxBIOS image. Normally there are two romimage definitions per
355 LinuxBIOS build: \texttt{normal} and \texttt{fallback}.
357 Each \texttt{romimage} section needs to specify a mainboard directory and a
358 payload. The mainboard directory contains the mainboard specific
359 configuration file and source code. It is specified relatively to
360 \texttt{freebios2/src/mainboard}. The payload definition is an absolute
361 path to a static elf binary (i.e Linux kernel or etherboot)
365 option USE_FALLBACK_IMAGE=0
366 option ROM_IMAGE_SIZE=0x10000
367 option LINUXBIOS_EXTRA_VERSION=".0Normal"
369 payload /suse/stepan/tg3ide_
374 \item \begin{verbatim}buildrom\end{verbatim}
376 The \texttt{buildrom} statement is used to determine which of the
377 LinuxBIOS image builds (created using \texttt{romimage}) are packed
378 together to the final LinuxBIOS image. It also specifies the order of
379 the images and the final image size:
382 buildrom ./solo.rom ROM_SIZE "normal" "fallback"
388 \subsubsection{Firmware image configuration options}
389 In addition to the definitions described above there are a number of
390 commonly used options. Configuration options set in the firmware image
391 specific configuration file can override default selections from the
392 Motherboard specific configuration. See above examples about
393 option on how to set them.
397 \item \begin{verbatim}CC\end{verbatim}
399 Target C Compiler. Default is \texttt{\$(CROSS\_COMPILE)gcc}. Set to
400 \texttt{gcc m32} for compiling AMD64 LinuxBIOS images on an AMD64
403 \item \begin{verbatim}CONFIG_CHIP_CONFIGURE \end{verbatim}
405 Use new \textit{chip\_configure} method for configuring (nonpci)
406 devices. Set to \texttt{1} for all AMD64 motherboards.
408 \item \begin{verbatim}MAXIMUM_CONSOLE_LOGLEVEL\end{verbatim}
410 Errors or log messages up to this level can be printed. Default is
411 \texttt{8}, minimum is \texttt{0}, maximum is \texttt{10}.
413 \item \begin{verbatim}DEFAULT_CONSOLE_LOGLEVEL\end{verbatim}
415 Console will log at this level unless changed. Default is \texttt{7},
416 minimum is \texttt{0}, maximum is \texttt{10}.
418 \item \begin{verbatim}CONFIG_CONSOLE_SERIAL8250\end{verbatim}
420 Log messages to 8250 uart based serial console. Default is \texttt{0}
421 (don't log to serial console). This value should be set to \texttt{1}
422 for all AMD64 builds.
424 \item \begin{verbatim}ROM_SIZE\end{verbatim}
426 Size of final ROM image. This option has no default value.
428 \item \begin{verbatim}FALLBACK_SIZE\end{verbatim}
430 Fallback image size. Defaults to \texttt{65536} bytes. \textbf{NOTE:}
431 This does not include the fallback payload.
433 \item \begin{verbatim}HAVE_OPTION_TABLE\end{verbatim}
435 Export CMOS option table. Default is \texttt{0}. Set to \texttt{1} if
436 your motherboard has CMOS memory and you want to use it to store
437 LinuxBIOS parameters (Loglevel, serial line speed, ...)
439 \item \begin{verbatim}CONFIG_ROM_STREAM\end{verbatim}
441 Boot image is located in ROM (as opposed to \texttt{CONFIG\_IDE\_STREAM}, which
442 will boot from an IDE disk)
444 \item \begin{verbatim}HAVE_FALLBACK_BOOT\end{verbatim}
446 Set to \texttt{1} if fallback booting is required. Defaults to
452 The following options should be used within a romimage section:
456 \item \begin{verbatim}USE_FALLBACK_IMAGE\end{verbatim}
458 Set to \texttt{1} to build a fallback image. Defaults to \texttt{0}
460 \item \begin{verbatim}ROM_IMAGE_SIZE\end{verbatim}
462 Default image size. Defaults to \texttt{65535} bytes.
464 \item \begin{verbatim}LINUXBIOS_EXTRA_VERSION\end{verbatim}
466 LinuxBIOS extra version. This option has an empty string as default. Set
467 to any string to add an extra version string to your LinuxBIOS build.
473 \subsection{Motherboard specific configuration}
475 Motherboard specific configuration files describe the onboard
476 motherboard components, i.e. bridges, number and type of CPUs. They also
477 contain rules for building the low level start code which is translated
478 using romcc and/or the GNU assembler. This code enables caches and
479 registers, early mtrr settings, fallback mechanisms, dram init and
482 \textbf{NOTE:} The option keyword can not be used in motherboard specific
483 configuration files. Options shall instead be set using the default
484 keyword so that they can be overridden by the image specific
485 configuration files if needed.
487 \subsubsection{Motherboard specific keywords}
489 The following statements are used in motherboard specific configuration
493 \item \begin{verbatim}arch\end{verbatim}
495 Sets the CPU architecture. This should be \texttt{i386} for AMD64 boards.
502 \item \begin{verbatim}cpu\end{verbatim}
504 The cpu statement is needed once per possibly available CPU. In a
505 one-node system, write:
511 \item \begin{verbatim}driver\end{verbatim}
513 The \texttt{driver} keyword adds an object file to the driver section of a
514 LinuxBIOS image. This means it can be used by the PCI device
515 initialization code. Example:
521 \item \begin{verbatim}object\end{verbatim}
523 The \texttt{object} keyword adds an object file to the LinuxBIOS image.
524 Per default the object file will be compiled from a \texttt{.c} file
525 with the same name. Symbols defined in such an object file can be used
526 in other object files and drivers. Example:
532 \item \begin{verbatim}makerule\end{verbatim}
534 This keyword can be used to extend the existing file creation rules
535 during the build process. This is useful if external utilities have to
536 be used for the build. LinuxBIOS on AMD64 uses romcc for it's early
537 startup code placed in auto.c.
539 To tell the configuration mechanism how to build \texttt{romcc} files,
544 depends "$(MAINBOARD)/auto.c"
545 action "$(CPP) I$(TOP)/src $(ROMCCPPFLAGS) $(CPPFLAGS) \
546 $(MAINBOARD)/auto.c > ./auto.E"
549 depends "./auto.E ./romcc"
550 action "./romcc mcpu=k8 O ./auto.E > auto.inc"
554 Each makerule section contains file dependencies (using the depend
555 keyword) and an action that is taken when the dependencies are satisfied
556 (using the action keyword).
558 \item \begin{verbatim}mainboardinit\end{verbatim}
560 With the mainboardinit keyword it's possible to include assembler code
561 directly into the LinuxBIOS image. This is used for early infrastructure
562 initialization, i.e. to switch to protected mode. Example:
565 mainboardinit cpu/i386/entry16.inc
568 \item \begin{verbatim}ldscript\end{verbatim}
570 The GNU linker ld is used to link object files together to a LinuxBIOS
573 Since it is a lot more comfortable and flexible to use the GNU linker
574 with linker scripts (ldscripts) than to create complex command line
575 calls, LinuxBIOS features including linker scripts to control image
579 ldscript /cpu/i386/entry16.lds
583 \item \begin{verbatim}dir\end{verbatim}
585 LinuxBIOS reuses as much code between the different ports as possible.
586 To achieve this, commonly used code can be stored in a seperate
587 directory. For a new motherboard, it is enough to know that the code in
588 that directory can be used as is.
590 LinuxBIOS will also read a \texttt{Config.lb} file stored in that
591 directory. This happens with:
598 \item \begin{verbatim}config\end{verbatim}
600 This keyword is needed by the new chip configuration scheme. Should be
607 \item \begin{verbatim}register\end{verbatim}
608 The \texttt{register} keyword can occur in any section, passing
609 additional parameters to the code handling the according device.
613 register "com1" = "{1, 0, 0x3f8, 4}"
616 \item \begin{verbatim}northbridge\end{verbatim}
618 The \texttt{northbridge} keyword describes a system northbridge. Some
619 systems, like AMD64, can have more than one northbridge, i.e. one per
620 CPU node. Each northbridge is described by the path to the northbridge
621 code in LinuxBIOS (relative to \texttt{freebios2/src/northbridge}), i.e.
622 \texttt{amd/amdk8} and a unique name (i.e ´mc0' ) Example:
625 northbridge amd/amdk8 "mc0"
630 \item \begin{verbatim}southbridge\end{verbatim}
632 To simplify the handling of bus bridges in a LinuxBIOS system, all
633 bridges available in a system that are not northbridges (i.e AGP, IO,
634 PCIX) are seen as southbridges.
636 Since from the CPUs point of view any southbridge is connected via the
637 northbridge, a southbridge section is declared within the northbridge
638 section of the north bridge it is attached to.
640 Like the northbridge, any other bridge is described by the path to it's
641 driver code, and a unique name. If the described bridge is a
642 hypertransport device, the northbridge's hypertransport link it connects
643 to can be specified using the \texttt{link} keyword. Example:
646 northbridge amd/amdk8 "mc0"
648 southbridge amd/amd8111 "amd8111" link 0
655 \item \begin{verbatim}pci\end{verbatim}
657 The \texttt{pci} keyword can only occur within a \texttt{northbridge} or
658 \texttt{southbridge} section. It is used to describe the PCI devices
659 that are provided by the bridge. Generally all bridge sections have a
660 couple of \texttt{pci} keywords.
662 The first occurrence of the \texttt{pci} keyword tells LinuxBIOS where
663 the bridge devices start, relative to the PCI configuration space used
664 by the bridge. The following occurences of the \texttt{pci} keyword
665 describe the provided devices.
667 Adding the option \texttt{on} or \texttt{off} to a PCI device will
668 enable or disable this device. This feature can be used if some bridge
669 devices are not wired to hardware outputs and thus are not used.
674 northbridge amd/amdk8 "mc1"
687 southbridge amd/amd8111 "amd8111" link 0
703 \item \begin{verbatim}superio\end{verbatim}
705 SuperIO devices are basically handled like brigdes. They are taking
706 their driver code from \texttt{freebios2/src/superio/}. They don't
707 provide a PCI compatible configuration interface, but instead are ISA
708 PnP devices. Normally they are connected to a southbridge. If this is
709 the case, the \texttt{superio} section will be a subsection of the
710 \texttt{southbridge} section of the southbridge it is connected to.
714 superio NSC/pc87360 link 1
726 register "com1" = "{1, 0, 0x3f8, 4}"
727 register "lpt" = "{1}"
735 \subsubsection{Motherboard specific configuration options}
737 The following options are commonly used in motherboard specific
740 They should be set using the \texttt{default} keyword:
744 \item \begin{verbatim}HAVE_HARD_RESET\end{verbatim}
746 If set to \texttt{1}, this option defines that there is a hard reset
747 function for this mainboard. This option is not defined per default.
749 \item \begin{verbatim}HAVE_PIRQ_TABLE\end{verbatim}
751 If set to \texttt{1}, this option defines that there is an IRQ Table for
752 this mainboard. This option is not defined per default.
754 \item \begin{verbatim}IRQ_SLOT_COUNT\end{verbatim}
756 Number of IRQ slots. This option is not defined per default.
758 \item \begin{verbatim}HAVE_MP_TABLE\end{verbatim}
760 Define this option to build an MP table (v1.4). The default is not to
763 \item \begin{verbatim}HAVE_OPTION_TABLE\end{verbatim}
765 Define this option to export a CMOS option table. The default is not to
766 export a CMOS option table.
768 \item \begin{verbatim}CONFIG_SMP\end{verbatim}
770 Set this option to \texttt{1} if the mainboard supports symmetric
771 multiprocessing (SMP). This option defaults to \texttt{0} (no SMP).
773 \item \begin{verbatim}CONFIG_MAX_CPUS\end{verbatim}
775 If \begin{verbatim}CONFIG_SMP\end{verbatim} is set, this option defines
776 the maximum number of CPUs (i.e. the number of CPU sockets) in the
777 system. Defaults to \texttt{1}.
779 \item \begin{verbatim}CONFIG_IOAPIC\end{verbatim}
781 Set this option to \texttt{1} to enable IOAPIC support. This is
782 mandatory if you want to boot a 64bit Linux kernel on an AMD64 system.
784 \item \begin{verbatim}STACK_SIZE\end{verbatim}
786 LinuxBIOS stack size. The size of the function call stack defaults to
787 \texttt{0x2000} (8k).
789 \item \begin{verbatim}HEAP_SIZE\end{verbatim}
791 LinuxBIOS heap size. The heap is used when LinuxBIOS allocates memory
792 with malloc(). The default heap size is \texttt{0x2000}, but AMD64 boards
793 generally set it to \texttt{0x4000} (16k)
795 \item \begin{verbatim}XIP_ROM_BASE\end{verbatim}
797 Start address of area to cache during LinuxBIOS execution directly from
800 \item \begin{verbatim}XIP_ROM_SIZE\end{verbatim}
802 Size of area to cache during LinuxBIOS execution directly from ROM
804 \item \begin{verbatim}CONFIG_COMPRESS\end{verbatim}
806 Set this option to \texttt{1} for a compressed image. If set to
807 \texttt{0}, the image is bigger but will start slightly faster (since no
808 decompression is needed).
816 % 10. Tweaking the source code
819 \section{Tweaking the source code}
820 Besides configuring the existing code it is sometimes necessary or
821 wished to tweak certain parts of LinuxBIOS by direct changes to the
822 code. This chapter covers some possible enhancements and changes that
823 are needed when porting LinuxBIOS to a new motherboard or just come
826 \subsection{Hypertransport configuration}
827 Before LinuxBIOS is able to activate all CPUs and detect bridges
828 attached to these CPUs (and thus, see all devices attached to the
829 system) it has to initialize the coherent hypertransport devices.
831 The current algorithms to do coherent hypertransport initialization are
832 not fully automatically evaluating the hypertransport chain graph.
833 Therefore the code needs to be adapted when porting LinuxBIOS to a new
834 AMD64 motherboard. An example arrangement of hypertransport devices
839 \includegraphics[scale=1.0]{hypertransport.pdf}
840 \caption{Example: Hypertransport Link Connections}
841 \label{fix:hypertransport}
844 Each hypertransport device has one to three hypertransport links that
845 are used for device interconnection. These links are called LDT$[$012$]$, or
846 accordingly UP, ACROSS, DOWN. Communication between the hypertransport
847 devices can be freely routed, honoring the physical wiring. Teaching the
848 coherent hypertransport initialization algorithm this wiring happens in
854 \item Setting outgoing connections
855 The algorithm needs to know which outgoing port of a CPU node is
856 connected to the directly succeeding node. This is done in
857 \texttt{freebios2/src/mainboard/$<$vendor$>$/$<$mainboard$>$/auto.c}
858 with a number of preprocessor defines (one define for two-node systems,
859 three defines for four-node systems).
861 The ports in question are flagged with a circle in the graph for
862 illustration. For the example graph above (all outgoing connections are
863 realized using LDT1/ACROSS) the defines are:
866 #define CONNECTION_0_1 ACROSS
867 #define CONNECTION_0_2 ACROSS
868 #define CONNECTION_1_3 ACROSS
871 \item Describing routing information between CPUs.
873 There are basically three different message types for hypertransport
876 \item request packages
877 \item response packages
878 \item broadcast packages
881 These three message types are routed using different hypertransport
882 ports. The routing information is written to the AMD K8 routing table.
883 In an Nnode system this routing table consists of 3*N*N entries , one
884 for each message type and for each possible CPU --> CPU communication. For
885 simplicity LinuxBIOS keeps the 3 routing entries for each CPU --> CPU
886 communication in one machine word. The routing table of each node looks
890 /* Routing Table for Node i
892 * F0: 0x40, 0x44, 0x48, 0x4c, 0x50, 0x54, 0x58, 0x5c
893 * i: 0, 1, 2, 3, 4, 5, 6, 7
895 * [ 0: 3] Request Route
896 * [0] Route to this node
897 * [1] Route to Link 0
898 * [2] Route to Link 1
899 * [3] Route to Link 2
900 * [11: 8] Response Route
901 * [0] Route to this node
902 * [1] Route to Link 0
903 * [2] Route to Link 1
904 * [3] Route to Link 2
905 * [19:16] Broadcast route
906 * [0] Route to this node
907 * [1] Route to Link 0
908 * [2] Route to Link 1
909 * [3] Route to Link 2
913 The routing table is passed to the coherent hypertransport
914 initialization algorithm by defining a function called
915 \texttt{generate\_row()} in \texttt{auto.c}:
918 static unsigned int generate_row
919 (uint8_t node, uint8_t row, uint8_t maxnodes)
922 This function is trivial if there is only one CPU in the system, since
923 no routing has to be done:
926 static unsigned int generate_row
927 (uint8_t node, uint8_t row, uint8_t maxnodes)
929 return 0x00010101; /* default row entry */
933 On a two-node system things look slightly more complicated. Since the
934 coherent hypertransport initialization algorithm works by consecutively
935 enabling CPUs, it contains routing information for driving the system
936 with one node and two nodes:
939 static unsigned int generate_row
940 (uint8_t node, uint8_t row, uint8_t maxnodes)
942 uint32_t ret=0x00010101; /* default row entry */
943 static const unsigned int rows_2p[2][2] = {
944 { 0x00050101, 0x00010404 },
945 { 0x00010404, 0x00050101 }
947 if(maxnodes>2) maxnodes=2;
948 if (!(node>=maxnodes || row>=maxnodes)) {
949 ret=rows_2p[node][row];
955 Systems with four nodes have to contain routing information for one, two
956 and four-node setups:
959 static unsigned int generate_row
960 (uint8_t node, uint8_t row, uint8_t maxnodes)
962 uint32_t ret=0x00010101; /* default row entry */
963 static const unsigned int rows_2p[2][2] = {
964 { 0x00030101, 0x00010202 },
965 { 0x00010202, 0x00030101 }
967 static const unsigned int rows_4p[4][4] = {
968 { 0x00070101, 0x00010202, 0x00030404, 0x00010204 },
969 { 0x00010202, 0x000b0101, 0x00010208, 0x00030808 },
970 { 0x00030808, 0x00010208, 0x000b0101, 0x00010202 },
971 { 0x00010204, 0x00030404, 0x00010202, 0x00070101 }
973 if (!(node>=maxnodes || row>=maxnodes)) {
975 ret=rows_2p[node][row];
977 ret=rows_4p[node][row];
984 \subsection{DRAM configuration}
985 Setting up the RAM controller(s) is probably the most complex part of
986 LinuxBIOS. Basically LinuxBIOS serially initializes all RAM controllers
987 in the system, using SPDROM (serial presence detect) data to set
988 timings, size and other properties. The SPD data is usually read
989 utilizing the I2C SMBUS interface of the southbridge.
991 There is one central data structure that describes the RAM controllers
992 available on an AMD64 system and the concerned devices:
995 struct mem_controller {
997 device_t f0, f1, f2, f3;
1003 Available motherboard implementations and CPUs create the need to add
1004 special setup code to RAM initialization in a number of places.
1005 LinuxBIOS provides hooks to easily add code in these places without
1006 having to change the generic code. Whether these hooks have to be used
1007 depends on the motherboard design. In many cases the functions executed
1008 by the hooks just carry out trivial default settings or they are even
1011 Some motherboard/CPU combinations need to trigger an additional memory
1012 controller reset before the memory can be initialized properly. This is,
1013 for example, used to get memory working on preC stepping AMD64
1014 processors. LinuxBIOS provides two hooks for triggering onboard memory
1018 \item \begin{verbatim}static void memreset_setup(void)\end{verbatim}
1019 \item \begin{verbatim}static void memreset(int controllers, const struct
1020 mem_controller *ctrl)\end{verbatim}
1023 Some motherboards utilize an SMBUS hub or possibly other mechanisms to
1024 allow using a large number of SPDROMs and thus ram sockets. The result
1025 is that only the SPDROM information of one cpu node is visible at a
1026 time. The following function, defined in \texttt{auto.c}, is called every time
1027 before a memory controller is initialized and gets the memory controller
1028 information of the next controller as a parameter:
1031 static inline void activate_spd_rom (const struct mem_controller *ctrl)
1034 The way SMBUS hub information is coded into the \texttt{mem\_controller}
1035 structure is motherboard implementation specific and not closer
1036 described here. See \texttt{freebios2/src/mainboard/amd/quartet/auto.c}
1039 LinuxBIOS folks have agreed on SPD data being the central information
1040 source for RAM specific information. But not all motherboards/RAM
1041 modules feature a physical SPD ROM. To still allow an easytouse SPD
1042 driven setup, there is a hook that abstracts reading the SPD ROM
1043 ingredients that are used by the memory initialization mechanism:
1046 static inline int spd_read_byte(unsigned device, unsigned address)
1049 This function, defined in \texttt{auto.c}, directly maps it's calls to
1050 \texttt{smbus\_read\_byte()} calls if SPD ROM information is read via
1054 static inline int spd_read_byte(unsigned device, unsigned address)
1056 return smbus_read_byte(device & 0xff, address);
1060 If there is no SPD ROM available in the system design, this function
1061 keeps an array of SPD ROM information hard coded per logical RAM module.
1062 It returns the ´faked' SPD ROM information using device and address
1063 as indices to this array.
1066 \subsection {IRQ Tables}
1068 Motherboards that provide an IRQ table should have the following two
1069 variables set in their \texttt{Config.lb} file:
1072 default HAVE_PIRQ_TABLE=1
1073 default IRQ_SLOT_COUNT=7
1076 This will make LinuxBIOS look for the file
1077 \texttt{freebios2/src/mainboard/<vendor>/<motherboard>/irq\_tables.c} which
1078 contains the source code definition of the IRQ table. LinuxBIOS corrects
1079 small inconsistencies in the IRQ table during startup (checksum and
1080 number of entries), but it is not yet writing IRQ tables in a completely
1083 \textbf{NOTE:} To get Linux to understand and actually use the IRQ
1084 table, it is not always a good idea to specify the vendor and device id
1085 of the actually present interrupt router device. Linux 2.4 for example
1086 does not know about the interrupt router of the AMD8111 southbridge. In
1087 such cases it is advised to choose the vendor/device id of a compatible
1088 device that is supported by the Linux kernel. In case of the AMD8111
1089 interrupt router it is advised to specify the AMD768/Opus interrupt
1090 controller instead (vendor id=\texttt{0x1022}, device id=\texttt{0x7443})
1092 \subsection {MP Tables}
1094 LinuxBIOS contains code to create MP tables conforming the
1095 Multiprocessor Specification V1.4. To include an MP Table in a LinuxBIOS
1096 image, the following configuration variables have to be set (in the
1097 mainboard specific configuration file
1098 \texttt{freebios2/src/mainboard/<vendor><mainboard>/Config.lb}):
1101 default CONFIG_SMP=1
1102 default CONFIG_MAX_CPUS=1 # 2,4,..
1103 default HAVE_MP_TABLE=1
1106 LinuxBIOS will then look for a function for setting up the MP table in
1107 the file \texttt{freebios2/src/mainboard<vendor>/<mainboard>/mptable.c}:
1110 void *smp_write_config_table(void *v, unsigned long * processor_map)
1113 MP Table generation is still somewhat static, i.e. changing the bus
1114 numbering will force
1115 you to adopt the code in mptable.c. This is subject to change in future
1118 \subsection {ACPI Tables}
1120 There is initial ACPI support in LinuxBIOS now. Currently the only gain with
1121 this is the ability to use HPET timers in Linux. To achieve this, there is a
1122 framework that can generate the following tables:
1130 To enable ACPI in your LinuxBIOS build, add the following lines to your
1131 configuration files:
1133 uses HAVE_ACPI_TABLES
1135 option HAVE_ACPI_TABLES=1
1138 To keep Linux doing it's pci ressource allocation based on IRQ tables and MP
1139 tables, you have to specify the kernel parameter \texttt{pci=noacpi} otherwise
1140 your PCI devices won't get interrupts.
1141 It's likely that more ACPI support will follow, when there is need for certain
1145 LinuxBIOS has three different methods of handling POST codes. They can
1146 be triggered using configuration file options.
1149 \emph{Ignore POST completely}. No early code debugging is possible with
1150 this setting. Set the configuration variable \texttt{NO\_POST} to
1151 \texttt{1} to switch off all POST handling in LinuxBIOS.
1153 \emph{Normal IO port 80 POST}. This is the default behavior of
1154 LinuxBIOS. No configuration variables have to be set. To be able to see
1155 port 80 POST output, you need a POST expansion card for ISA or PCI. Port
1156 80 POST allows simple debugging without any other output method
1157 available (serial interface or VGA display)
1160 This option allows to push POST messages to the serial interface instead
1161 of using IO ports. \textbf{NOTE:} The serial interface has to be
1162 initialized before serial POST can work. To use serial POST, set the
1163 configuration variable \texttt{CONFIG\_SERIAL\_POST} to the value 1.
1167 \subsection{HDT Debugging}
1168 If you are debugging your LinuxBIOS code with a Hardware Debug Tool
1169 (HDT), you can find the source code line for a given physical EIP
1170 address as follows: Look the address up in the file linuxbios.map. Then
1171 search the label Lxx in the file auto.inc created by romcc. The original
1172 source code file and line number is mentioned in auto.inc.
1175 \subsection{Device Drivers}
1176 With only a few data structures LinuxBIOS features a simple but flexible
1177 device driver interface. This interface is not intended for autonomously
1178 driving the devices but to initialize all system components so that they
1179 can be used by the booted operating system.
1181 Since nowadays most systems are PCI centric, the data structures used
1182 are tuned towards (onboard and expansion bus) PCI devices. Each driver
1183 consists of at least two structures.
1185 The \texttt{pci\_driver} structure maps PCI vendor/device id pairs to a
1186 second structure that describes a set of functions that together
1187 initialize and operate the device:
1190 static void adaptec_scsi_init(struct device *dev)
1194 static struct device_operations lsi_scsi_ops = {
1195 .read_resources = pci_dev_read_resources,
1196 .set_resources = pci_dev_set_resources,
1197 .enable_resources = pci_dev_enable_resources,
1198 .init = lsi_scsi_init,
1201 static struct pci_driver lsi_scsi_driver __pci_driver = {
1202 .ops = &lsi_scsi_ops,
1208 By separating the two structures above, M:N relations between compatible
1209 devices and drivers can be described. The driver source code containing
1210 above data structures and code have to be added to a LinuxBIOS image
1211 using the driver keyword in the motherboard specific configuration file
1212 \texttt{freebios2/src/mainboard/<vendor>/<mainboard>/Config.lb}:
1218 \subsection{Bus Bridges}
1220 Currently all bridges supported in the LinuxBIOS2 tree are transparent
1221 bridges. This means, once the bridge is initialized, it's remote devices
1222 are visible on one of the PCI buses without special probing. LinuxBIOS
1223 supports also bridges that are nontransparent. The driver support code
1224 can provide a \texttt{scan\_bus} function to scan devices behind the bridge.
1226 \subsection{CPU Reset}
1227 When changing speed and width of hypertransport chain connections
1228 LinuxBIOS has to either assert an LDTSTOP or a reset to make the changes
1229 become active. Additionally Linux can do a firmware reset, if LinuxBIOS
1230 provides the needed infrastructure. To use this capability, define the
1231 option \texttt{HAVE\_HARD\_RESET} and add an object file specifying the
1232 reset code in your mainboard specific configuration file
1233 \texttt{freebios2/src/mainboard/$<$vendor$>$/$<$mainboard$>$/Config.lb}:
1236 default HAVE_HARD_RESET=1
1240 The C source file \texttt{reset.c} (resulting in \texttt{reset.o}
1241 during compilation) shall define the following function to take care
1242 of the system reset:
1245 void hard_reset(void);
1248 See \texttt{freebios2/src/mainboard/arima/hdama/reset.c} for an example
1254 % 11. LinuxBIOS Internals
1257 \section{LinuxBIOS Internals}
1258 This chapter covers some of the internal structures and algorithms of
1259 LinuxBIOS that have not been mentioned so far.
1261 \subsection{Code Flow}
1265 \includegraphics[scale=0.7]{codeflow.pdf}
1266 \caption{LinuxBIOS rough Code Flow}
1267 \label{fix:codeflow}
1272 \subsection{Fallback mechanism}
1273 LinuxBIOS provides a mechanism to pack two different LinuxBIOS builds
1274 within one LinuxBIOS ROM image. Using the system CMOS memory LinuxBIOS
1275 determines whether the last boot with a default image succeeded and
1276 boots a failsafe image on failure. This allows insystem testing without
1277 the risk to render the system unusable. See
1278 \texttt{freebios2/src/mainboard/arima/hdama/failover.c} for example
1279 code. The fallback mechanism can be used with the \texttt{cmos\_util}.
1281 \subsection{(Un) Supported Standards}
1282 LinuxBIOS supports the following standards
1284 \item Multiprocessing Specification (MPSPEC) 1.4
1285 \item IRQ Tables (PIRQ)
1286 \item ACPI (initial support on AMD64)
1289 However, the following standards are not supported until now, and will
1290 probably not be supported in future revisions:
1295 \subsection{LinuxBIOS table}
1296 LinuxBIOS stores information about the system in a data structure called
1297 the LinuxBIOS table. This table can be read under Linux using the tool
1298 lxbios from the Lawrence Livermore National Laboratory.
1300 Get more information about lxbios and the utility itself at
1301 \url{http://www.llnl.gov/linux/lxbios/lxbios.html}
1303 \subsection{ROMCC limitations}
1304 ROMCC, part of the LinuxBIOS project, is a C compiler that translates to
1305 completely rommable code. This means the resulting code does not need
1306 any memory to work. This is one of the major improvements in LinuxBIOS
1307 V2, since it allows almost all code to be written in C. DRAM
1308 initialization can be factored and reused more easily among mainboards
1311 Since no memory is available during this early initialization point,
1312 romcc has to map all used variables in registers for their time being.
1313 Same applies for their stack usage. Generally the less registers are
1314 used up by the algorithms, the better code can be factored, resulting in
1315 a smaller object size. Since getting the best register usage is an NP
1316 hard problem, some heuristics are used to get reasonable translation
1317 time. If you run out of registers during compilation, try to refactor
1320 \subsection{CMOS handling}
1321 LinuxBIOS can use the motherboard's CMOS memory to store information
1322 defined in a data structure called the CMOS table . This information
1323 contains serial line speed, fallback boot control, output verbosity,
1324 default boot device, ECC control, and more. It can be easily enhanced by
1325 enhancing the CMOS table. This table, if present, is found at
1326 \texttt{freebios2/src/mainboard/$<$vendor$>$/$<$mainboard$>$/cmos.layout}.
1327 It describes the available options, their possible values and their
1328 position within the CMOS memory. The layout file looks as follows:
1330 # startbit length config configID name
1335 # configid value human readable description
1347 To change CMOS values from a running Linux system, use the
1348 \texttt{cmos\_util}, provided by Linux Networks as part of the LinuxBIOS
1349 utilities suite. Get it at
1350 \textit{ftp://ftp.lnxi.com/pub/linuxbios/utilities/}
1352 \subsection {Booting Payloads}
1353 LinuxBIOS can load a payload binary from a Flash device or IDE. This
1354 payload can be a boot loader, like FILO or Etherboot, a kernel image, or
1355 any other static ELF binary.
1357 To create a Linux kernel image, that is bootable in LinuxBIOS, you have
1358 to use mkelfImage. The command line I used, looks like follows:
1361 objdir/sbin/mkelfImage t bzImagei386 kernel /boot/vmlinuz \
1362 commandline="console=ttyS0,115200 root=/dev/hda3" \
1363 initrd=/boot/initrd output vmlinuz.elf
1367 This will create the file \texttt{vmlinuz.elf} from a distribution
1368 kernel, console redirected to the serial port and using an initial
1371 \subsection{Kernel on dhcp/tftp}
1373 One possible scenario during testing is that you keep your kernel (or
1374 any additional payload) on a different machine on the network. This can
1375 quickly be done using a DHCP and TFTP server.
1377 Use for example following \texttt{/etc/dhcpd.conf} (adapt to your
1381 subnet 192.168.1.0 netmask 255.255.255.0 {
1382 range 192.168.1.0 192.168.1.31;
1383 option broadcastaddress 192.168.1.255;
1386 ddnsupdatestyle adhoc;
1389 hardware ethernet 00:04:76:EA:64:31;
1390 fixedaddress 192.168.1.24;
1391 filename "vmlinuz.elf";
1396 Additionally you have to run a \texttt{tftp} server. You can start one
1397 using \texttt{inetd}. To do this, you have to remove the comment from
1398 the following line in \texttt{/etc/inetd.conf}:
1401 tftp dgram udp wait root /usr/sbin/in.tftpd in.tftpd -s /tftpboot
1404 Then put your kernel image \texttt{vmlinuz.elf} to \texttt{/tftpboot} on
1405 the \texttt{tftp} server.
1411 % 12. Advanced Device Initialization Mechanisms
1414 \section{Advanced Device Initialization Mechanisms}
1416 Like software, today's hardware is getting more and more complex. To
1417 stay flexible many hardware vendors start breaking hardware
1418 compatibility to old standards as VGA. Thus, VGA is still supported by
1419 most cards, but emulation has to be enabled by the firmware for the
1420 device to operate properly. Also, many expansion cards are small
1421 discrete systems that have to initialize attached ram, download
1422 controller firmware and similar. Without this initialization, an
1423 operating system can not take advantage of the hardware, so there needs
1424 to be a way to address this issue. There are several alternatives:
1426 \subsection{Native LinuxBIOS Support}
1428 For some devices (ie Trident Cyberblade 3d) there is native LinuxBIOS
1429 support This means there is a small driver bound to the PCI id of the
1430 device that is called after PCI device ressources are allotted.
1435 \item minimal driver
1441 \item need an additional driver
1442 \item viable for onboard devices only
1443 \item not flexible for pci cards
1446 \subsection{Using Native Linux Support}
1448 A simple way of getting a whole lot of drivers available for LinuxBIOS
1449 is to reuse Linux drivers by putting a Linux kernel to flash. This
1450 works, because no drivers are needed to get the Linux kernel (as opposed
1451 to store the kernel on a harddisk connected to isa/scsi/raid storage)
1455 \item large number of open source drivers
1460 \item need Linux in Flash (BLOAT!)
1461 \item drivers expect devices to be initialized (LSI1020/1030)
1463 \item large flash needed (4MBit minimum, normal operations 8+ MBit)
1467 \subsection{Running X86 Option ROMs}
1469 Especially SCSI/RAID controllers and graphics adapters come with a
1470 special option rom. This option rom usually contains x86 binary code
1471 that uses a legacy PCBIOS interface for device interaction. If this code
1472 gets executed, the device becomes operable in Linux and other operating
1477 \item really flexible
1478 \item no need for additional drivers on firmware layer
1479 \item large number of supported devices
1484 \item non-x86 platforms need complex emulation
1485 \item x86 platforms need legacy API
1486 \item outdated concept
1490 \subsection{Running Open Firmware Option ROMs}
1492 Some PCI devices come with open firmware option roms. These devices are
1493 normally found in computers from SUN, Apple or IBM. Open Firmware is a
1494 instruction set architecture independent firmware standard that allows
1495 device specific initialization using simple, small, but flexible
1496 bytecode that runs with minimal footprint on all architectures that have
1497 an Open Firmware implementation.
1499 There is a free Open Firmware implementation available, called OpenBIOS,
1500 that runs on top of LinuxBIOS. See www.openbios.org
1504 \item architecture independence
1505 \item small footprint
1506 \item clean concept, less bugs
1511 \item only small number of devices come with OpenFirmware capable option roms
1523 LinuxBIOS only cares about lowlevel machine initialization, but also has
1524 very simple mechanisms to boot a file either from FLASHROM or IDE. That
1525 file, possibly a Linux Kernel, a boot loader or Etherboot, are called
1526 payload, since it is the first software executed that does not cope with
1527 pure initialization.
1531 Flash devices are commonly used in all different computers since unlike
1532 ROMs they can be electronically erased and reprogrammed.
1541 \section{Bibliography}
1542 \subsection{Additional Papers on LinuxBIOS}
1546 \textit{\url{http://www.linuxnetworx.com/products/linuxbios_white_paper.pdf}}
1549 \textit{\url{http://www.linuxbios.org/papers/}}
1551 \textit{\url{http://www.lysator.liu.se/upplysning/fa/linuxbios.pdf}}
1553 \textit{\url{http://portal.acm.org/citation.cfm?id=512627}}
1559 \item Etherboot: \textit{\url{http://www.etherboot.org/}}
1560 \item Filo: \textit{\url{http://te.to/~ts1/filo/}}
1561 \item OpenBIOS: \textit{\url{http://www.openbios.org/}}