ECT Installation Enhancements Notes

Chris Greenhalgh 2005-03-09

Background

Almost no-one is using the WebStart version, either as webstart or from script. Most people are using the separate batch files in the install directory. This includes platform and tool developers (e.g. Chris, Jan, Shahram), component developers (e.g. Ted) processing (scripting) component developers (e.g. Andy B/L), assembly builders/users (RCA, Eric).
Why?

"real" webstart:

requires broadband network access to origin server for download

can't do it from an unconnected machine (e.g. developer laptop on the move)
can't do it viably over a dialup connection (too slow)

start-up delay each time

boot manager

doesn't auto-start installation (even as a GUI option) => startup delay

waits for discovery fail before starting an installation master (common initial case) => startup delay

although useful to avoid potential failure mode of duplicate master(s)

have had problems spawning host manager and other applications e.g. under OSX

host manager/installation

downloads components etc. from built-in URL

requires broadband network access... (although does fail through with last version)
removes local versions
hard to add component - host jar on server, add to config file, restart installation
only downloads when installation first started

have had problems spawning container and other applications
doesn't pick up changes to e.g. editor without complete restart (from webstart)
no detailed status/error reporting, e.g.

component instantiation failure
component execution/update failure

difficult to access console log files

location not obvious
locked while being written to
no remote access

persistence files can grow very large

hard to find
not obvious how to prune them
no GUI support for management

no remote management/restart facilities
dataspace persistence can be a problem

not all things come back - half/half complexity
if a component is broken can keep being remade and breaking the installation

installation files can be oddly (fixed) placed

on network drive (MRL), causing very slow performance
not clear if should be per user or per host

Bottom line:

more hastle; no immediate benefits; a few situation-specific show-stoppers :-)

Key tasks/activities

platform developer

changes to underlying system; recompile; restart all

=> fast start option?!

run test scenario
check results
check diagnostics, logs, etc.

=> easy access to console/logs/etc,

tool developer

add/distribute new tool

=> extensible tool framework - distribute, run, log/console access, etc.

changes to tool; recompile; restart tool (redistribute tool?!)
check diagnostics/logs/console output

Java/native component developer

add new component

=> local component management framework/tool(s)

change, compile, redeploy

=> component distribution support (across installation)
=> container restart support (across installation)

distribute across installation
restart containers/reload,with/without restart old components running
try test scenario
check results
check diagnostics, logs, etc.

Processing component developer

author/edit in processing
convert to ect component; distribute/redeploy

=> special processing development container/tool?! (watches directory; converts; re/starts container)

restart container
try test scenario
check results
check diagnostics/logs/etc

Assembler/editor

add/update components
restart
re/connect

Local operator/user

check status including other machines

=> whole installation status - other machines; all errors/warnings (explicit?!)

get report info for errors to send

=> "crash report" generation

change local config e.g. autostart

Parts...

host/installation manager - webstarted, minimal, etc.
tools

graph editor
configuration editor
...

components

Distribution, Building, Running...

Kinds of files:

JARs of Java class files (and resources used by those class files such as icons) for

core library(s)
component(s)
editors/viewers/tools
third party libraries on which these depend

Native DLLs

JNI for core, editors, components, etc
Dependencies of these
External applications?

Source
Data files

configurations
properties
persistence
logs (events, console)

Documentation files

HTML
text
word
pdf
...

Versioning issues

identification

of file
of version

releases, bug fixes, etc.

of dependencies

run-time substitutability

Java 2 allows (does it actually do it??) native libraries to be unloaded when the Class Loader that loaded them is garbage collected. It does not allow two Class Loaders in the same JVM to load the same native library. (see JNI_OnLoad and JNI_OnUnload)

To do

Build process to set codebase and download URLs in ect.jnlp
Include rendezvous properties in ect.jnlp (commented out?!)
Boot manager to allow autostart (after delay) of a (suitably configured) installation

allow configure in installation [later/optional]

Normal build/install to populate directories for webstart from local directory or web directory (recursive)

implementation - file
implementation - web/html
change directory/url in Boot
change directory/url in Installation

explicit
automatic - for a slave?! (gets from master via dataspace?!)

refresh Installation (just terminate an autostart installation?!)

but kill tools?!

convert component build.xml targets to work in this way

Installation manager to present tool buttons dynamically (from tool jars) [later]
Installation manager give access to console

access to tool consoles?

Installation manager give remote access to download directories
Container give access to console

download.xml File Format

Current, representative:

<?xml version="1.0" encoding="UTF-8"?>
<download>
  <cache>
    <directory path="java/components">
	  <url>http://www.crg.cs.nott.ac.uk/~cmg/Equator/Downloads/docs/ect/webstart/processing_asteroids-component.jar</url>
	  <url ifosname="Windows*">http://www.crg.cs.nott.ac.uk/~cmg/Equator/Downloads/docs/ect/webstart/phidgquip-component.jar</url>
    </directory>
    <directory path="java/common">
    </directory>
    <directory path="java/config">
    </directory>
    <!-- javax containing packages -->
    <directory path="java/common/ext">
    </directory>
  </cache>
</download>

Proposed:

<?xml version="1.0" encoding="UTF-8"?>
<download baseurl="..."(opt) purge="true(default)|false" purgeonfail="true|false(default)" recurse="false(default)|true">
  <include name="..." baseurl="..."(opt.) ifosname="Windows*" skip="true|false(default)"/>
  <directory name="java" purge="true(default)|false" purgeonfail="true|false(default)" baseurl="..."(opt) ifosname="Windows*" skip="true|false(default)" recurse="false(default)|true">
    <directory name="components" ...>
      <file name="..." baseurl="..."(opt.) purgeonfail="true|false(default)"
        ifosname="Windows*" requires="(name),..." skip="true|false(default)" tool-class="..."(opt.) tool-text="Tool..."(opt.)/>
    </directory>
  </directory>
</download>

Notes:

default for baseurl is baseurl of containing (directory or download) element if any, else baseurl of include if this file is included, else directory containing download file.
default for purge is value of purge in containing (directory or download) element if any, else true.
default for purgeonfail is value of purgeonfail in containing (directory or download) element if any, else true.
if the same directory is specified more than once (in the same download file or via include), then purge false takes priority over purge true.
if the same directory is specified more than once (in the same download file or via include), purgeonfail, baseurl, ifosname and skip affect only the occurance within which they are specified.
if the same target file is specified (directory path and name) is specified more than once then:

ifosname values are concatenated with ',' (treated as logical OR) (optionally, duplicates may be removed)
requires values are concatenated with ',' (treated as logical AND) (optionally, duplicates may be removed)
skip = true takes precedance over skip = false

[rationale: mix in with skip="true" allows a problematic file to be ignored]

values in the requires attribute are checked against file name attribute values (ignoring path/directory)
tool-class should only be set if the file is a jar containing an application (e.g. editor) to be run from the installation

Implementation notes:

by default a component is a single jar with no dependencies in java/components, and download info is autogenerated.
by default, the complete "download" directory hierarchy ...

Hmm. How about you just give it a directory URL and it recursively copies it...?! And any jar in "tools" is assumed to be a tool (look for main...)?!

download/

java/

common/

ext/

components/
tools/
config/