This is a basic guide to getting started with Spinach.  Included
are directions for installing Liberty from source (if necessary),
applying any necessary changes to Liberty, a walkthrough of an example
configuration, and some tips about building your own configurations.

A NOTE ABOUT CODE AND DOCUMENTS:  Most of this code and almost all
documents were created using an editor that forces a tabstop to be
viewed as 3 spaces, not 8.  So, if the code appears with terrible
spacing in your editor, it's most likely a tab-stop issue.  For
example, you can use the "nano" editor to view these files as we
created them by using the command syntax "nano -T 3 filename" from
the command line.  Similarly, both emacs and vi can be configured
to change the size of a tabstop.

*****************************
* 1 - Spinach - What is it? *
*****************************

For a detailed explanation of Spinach, how it fits into Liberty, and why
we wrote it, you can read our paper which appeared at the 2004 Conference
on Languages, Tools, and Compilers for Embedded Systems.  You can download
this paper at:

http://www.ece.rice.edu/~willmann/pubs/spinach.pdf

To summarize, Spinach is a set of add-on modules for the Liberty simulation
environment (LSE).  Spinach modules are targeted specifically at embedded systems,
including (but not limited to) programmable network interfaces.  Spinach
simulators use no global state; rather, state is maintained locally at each
module, and timing decisions (such as memory bus arbitration) are evaluated
on a cycle-by-cycle basis.  Consequently, you can rearrange these modules in 
any way you see fit, and they will seamlessly support and accurately model
asynchronous interactions that are common in embedded systems (such as DMA
transfers and unsolicited I/O events).  Since memory-path widths and latencies
are explicitly defined, and since memory transfers are evaluated cycle-by-cycle, 
memory bandwidth, contention, and latency are modeled precisely.

Fundamental to Spinach's extensibility and reconfigurability is the notion of
"no global state" - since no module assumes anything about the state of another
module, it's possible to reconfigure these modules in any way you see fit and
they should "just work".  This means that you can do things like add additional
processors by adding a processor instance and corresponding memory paths - note
also, however, that just as in a real machine, you must take care of coherence
issues (if necessary).

**************************
* 2.  Installing Spinach *
**************************

Spinach can be downloaded from Sourceforge at http://spinach.sourceforge.net.
The tarball distribution contains this getting started guide, several tarballs
containing Spinach's various components, and an installer script. Extract
the outer tarball and follow the directions below.

2.1 - Liberty Requirements

Host Requirements:  Linux of some variety running on a little-endian
host.  It is possible to get Liberty to install on FreeBSD, but there
are significant issues regarding library and tool compatibility.  
The current generation of Spinach has been tested on Debian 3.1 and 
the "testing" version of its successor, on i386 and AMD64 architectures.
Previously Spinach has been tested on Redhat 7.3, Redhat 9.0, and Fedora
Core 2 for x86, and on Redhat 7.3 for Itanium 2. 64-bit machines need
the 32-bit x86 runtime support in order to use Java (see below).

Basic tool requirements:  You will need many (if not all) of the
tools that are installed by default in the "developers" option of the
Redhat-style distributions.  Your best bet is to make sure to install
everything that's remotely developer-related.  Liberty requires
python 2.x (and its development files), so you may need to modify your 
path accordingly on redhat 7.3 so that your default python toolset is 2.x. 
Liberty also requires the GNU autotools. We use automake 1.4, and recent 
versions of automake such as 1.9 will issue warnings during installation
(but may work anyway).

2.2 - Satisfying additional tool requirements:  
Beyond the basic included packages, you will need:

Sun's Java SDK version 1.4 - note that you need the SDK, not
just the JRE. Java 1.5/5.0 currently does not work because of name
collisions between Liberty and the Java SDK. Java 1.4 is only available
for x86 architectures, but there is no problem with using it on x86-64
machines.

Apache Ant version 1.5.2 or greater (available at http://ant.apache.org/)

There are some installation issues regarding the Java SDK; 
Make sure you have your JAVA_HOME, PATH, and CLASSPATH environment variables 
set correctly. JAVA_HOME should point to the base directory of your Java
installation.


2.3 - Building Liberty 

Stock Liberty distributions do not support some of the extensions that
we have added to Liberty for Spinach modules.  The upcoming 2.0 release
of Liberty will have implementations of our modifications (though
perhaps using slightly different syntax), but in the meantime you will
need to install our modified version of Liberty, with its dependences.

Spinach's version of Liberty and its addon libraries are distributed
along with an install script in a file called spinach_framework.tar.
To install Liberty, you need to create a directory where all the LSE
files will reside, set and export the environment variable LIBERTY
to this directory, and extract spinach_framework.tar in this directory.
Herafter, I'll presume this location is /my_liberty_location, but it 
can be whatever you want. Something like /usr/local/liberty or ~/liberty
will do fine. Before running the install script, run:

export LIBERTY=/my_liberty_location

Copy spinachLibertyInstall.sh, liberty_spinach.tar.gz, and spinach_libs.tar.gz
into /my_liberty_location and cd there.

Now you can run the install script:

sh spinachLibertyInstall.sh

The install script will extract the tarballs and run the LSE installer.
If there are errors in the LSE install, the installer should quit, and
will need to be run again (or you can look at the script and execute
the commands yourself, it's very simple).

LSE requires several environment variables to be set in order to run.
The installer script will create a file called liberty_env_setup which
defines these variables. You must source this before you can run LSE,
by using the following command:

source liberty_env_setup

You can add a line like the following to your .bashrc or similar file
so that it is sourced each time you log in:

source /my_liberty_location/liberty_env_setup


2.4 - Spinach Modules

The Spinach modules are distributed in a file called spinach_modules.tar.gz.
You can extract and use them anywhere - put them wherever your working
directory will be. 

That's it - The LSE/Spinach toolset is installed!!

***************************
* 3 - Building an Example *
***************************

3.1 - Explanation of file layout

Spinach modules and the demo files are distributed in a file called 
spinach_modules.tar.gz.
All modules are under the "lss" directory - they're broken up into
directories under there according to type.  Each core module has a .clm file
and possibly several .h files.  These core files get rolled into corresponding
.tar files, which is done by the "make_tars" script in each module directory.
Before getting started (or after changing any .clm file), you must (re)run the
"make_tars" script, which will copy the .tar files and .lss definitions to the
"build" directory - thus, you should build ALL machines in the "build"
directory.  In the "build" directory, there is also a "make_tars" script
for your convenience. If you run it with no arguments it will run "make_tars"
in each of the spinach directories (processor, nic, memory, interconnect, 
test_configs), and if you run it with arguments, it will run "make_tars" 
in each directory specified in the arguments.

3.2 - Description of the demo

The demo configuration uses the functional MIPS processor model to plug into
a cycle-accurate memory organization, as depicted in the figure "demo.jpg".
(This figure does not include some glue logic necessary to convert between
signal types and so forth, but it closely matches the actual modules used
in the model).  The demo and the demo memory organization are both documented
heavily - these documents will shed some light on how to write extensible
configurations.  In this case, you can change the number of processors with
just one variable in the toplevel demo.lss file (with the num_cores assignment
statement).

Each processor has two associated cache controllers (one for the "fetch" stage,
one for the "mem" stage).  Coherence is maintained via token coherence.  The
DRAM controller used is Scott Rixner's reordering DRAM simulator using the
timing specifications of a Micron GDDR SDRAM operating at 500MHz.  (No timing
synchronization between the processors and memory is provided, hence all
modules in the entire simulation operate with 500MHz clocks).  The "token
master" module is the token coherence master controller, which is responsible
for managing persistent memory requests.

3.3 - Building the demo

After running:

./make_tars

in the "build" directory, or in each of the "test_configs", "interconnect",
 "memory", "nic", and "processor" subdirectories, run:

ls-build demo.lss

This will build your simulator.  Upon successful completion you should see
a statement at the end that says:

touch valid_machine

IMPORTANT NOTE - any time you change your machine, a parameter,
or modify a source .clm file, you need to rebuild, then relink the machine.  
You do not need to do this if you're merely changing the executable or some
other file that is read by the simulator at runtime.

3.4 - Linking the simulator

From the command line, run:

mp_link.sh demo

Presuming you don't see any unresolved symbol errors, your machine will be
linked correctly when you get back to the command prompt.

3.5 - Running the simulator

3.5.1 - The file details

Each processor starts fetching executing code at the location indicated by the
"bootPC" parameter, which is assigned in demo.lss.  By default, this is the
elf-standard location 0x4000.  Upon simulation start, the simulator loads
several state files into per-module memory.  Each processor loads a register
file image ("regfile_init.dmp"), and the main memory is initialized with the
contents of the file "hello_world".  Conversely, register file state is saved
into "regfile_final.dmp", and main memory contents will be written into 
"data_memory.dmp".  Since MIPS is a big-endian machine, the data in these
files will be in big-endian format (regardless of the endianess of the host).

FILE LOCATIONS - All state files (loaded & saved) by Spinach modules are looked
for in the directory pointed to by the environment variable SPINACH_STATE_PATH.
If SPINACH_STATE_PATH is not set, the simulator assumes /tmp.  If requisite
files are not found, the simulator may terminate immediately, so you may need to
create empty files (with touch, for instance).  

Debug messages printed via the LSE_report_warn() API call (inside .clm files and
some .lss userpoints) go to the file pointed to by the LIBERTY_DEBUG_FILE
environment variable.  This is a buffered file (with lots of buffering), so
you may not see these messages arrive in realtime as the simulation runs, but they
are flushed on simulation exit (or termination, if the simulation ends abnormally).
If you don't set LIBERTY_DEBUG_FILE, the system will presume "/tmp/result.out".
There currently is a bug in which, upon simulation end, the system may leave that
file with odd permissions in which the file isn't readable or writeable - so you
may have to manually change the permissions of this file.  It's best to just
touch the file prior to running simulation, as the warning system will not override
the initial permissions - it just may not create a file with reasonable permissions.

Regarding the MIPS model (and every processor model), there's no on-line loading
process, because there's no operating system.  Normally, a shell would inspect the
executable type & apply the proper loader to the executable, which would interpret
where the code & data segments are.  We have an offline utility that takes an 
ELF executable (like what's produced by gcc for MIPS R4000) and creates a flat
memory image with the code & data segments properly located at their real addresses.
Notice there is NO virtual addressing.

3.5.2 - No really, how do I run this?

The demo firmware and simulator state initialization files are in the "demo_state"
directory, wherever you extraced spinach_modules.tar.gz. First, you need to create
the flat memory image which contains the relocated code and data segments at their
proper addresses (as described above).  To do this, you'll need the create_fw_image
utility.  The source for this, in the $LIBERTY/spinach_libs directory, can be 
compiled with gcc.

./create_fw_image input_elf_file output_image_for_spinach

In our case, you'd do:

./create_fw_image hello_world.elf hello_world

Then, you'd need to copy this file (hello_world) into the directory pointed to
by the environment variable SPINACH_STATE_PATH.

hello_world.elf was created using the Tigon utilities, which has a version of gcc
that targets the Tigon's MIPS R4000 processors - you can obtain these utilities 
at http://alteon.shareable.org/. Additionally, other compilers for the Tigon
can be found at http://www.osc.edu/~pw/tigon/.

The dissassembly of this executable is hello_world.txt.  All this does is copy
the string "hello, world!" to the memory location 0x8000, which you will be able
to verify by inspecting the generated data_memory.dmp - the range 0x8000 through
0x8100 is marked "uncacheable" by parameters in demo.lss.  If you make it cacheable,
then the data will be written into local cache state (which, by default is *not* 
dumped - you can change this by changing the dcaches[i].saveState statement
in demo_memory_organization.lss to "TRUE" and then setting the dcaches[i].dumpStateFile
to whatever file you'd like).

Finally, you actually invoke the simulator with the following command (presuming 
you've already linked the machine with mp_link.sh):

	echo wc | ls-run-bench -nolink machinename

For the demo, this means:

	echo wc | ls-run-bench -nolink demo


3.5.3 - How do I know this actually worked?

To verify the correct execution of this code, you should see the
string "hello, world!" at memory location 0x8000 after simulation 
has completed.  Remember, the contents of memory is dumped to the
file data_memory.dmp inside the directory pointed to by 
SPINACH_STATE_PATH.  So, if you do:

xxd $SPINACH_STATE_PATH/data_memory.dmp | less

and then search for "8000:"

you'll see the string "hello, world!" in memory.

3.5.4 - A word about simulation termination

Simulation terminates upon some egregious error (which will probably
be indicated in the LIBERTY_DEBUG_FILE or 
processor/runs/machinename/results/result.err file, where "machinename"
in this case is "demo".  Simulation also terminates some set number of
cycles (determined by the halt_padding parameter of the processors) after
encountering a MIPS "halt" instruction.  This padding exists to let any 
buffers in the system (such as pending memory writes) flush.  If you
set the halt_padding value too low and you're writing to DRAM, it's
likely that not all of your changes will commit to memory prior to
simulation exit.

******************************
* 4 - Contacting the Authors *
******************************

Please read this guide thoroughly before contacting the authors.
However, it's quite possible that we left something out.  We are
available on a limited basis via email at:

willmann (at) rice.edu (Paul Willmann)
brogioli (at) rice.edu (Mike Brogioli)
dschuff (at) purdue.edu (Derek Schuff)