CEDARS: A multi-site UK project to
create exemplars in Digital Archiving
David Holdsworth / Leeds University
Link to DADs info in Leeds
both a pun and an
indication that we have been quietly operating archival
activities for many years.
Some Tales from the Backwoods
CEDARS is a very new project, and has not yet formed
truly collective views.
This presentation focusses on experiences from building in-house
systems for digital preservation, and relates these both to the OAIS model
and to the needs of research libraries.
My personal perspective comes from decades of dealing with
data storage at the University of Leeds, where we have a history
of in-house systems which have been taken up elsewhere.
CURL, the Consortium of Research Libraries is the project owner.
There are three lead sites: Oxford, Cambridge and Leeds.
These are the sites with existing preservation systems in service.
use IBM's ADSM to provide an archive for the whole University.
began operation in 199x.
Strong interest in meta-data.
About 3 Tbytes of data.
use EPOCH to provide an archive with an in-house user interface
began operation in 199x.
Holds data from earliest computation in Cambridge.
use an in-house system to provide an archive for the whole University.
began operation in 1992.
Current system is designed for indefinitely long life.
Holds data from previous systems back to 1980 (maybe 1970s).
About 1 Tbyte of data, in approx 2.7 million files
belonging to several thousand users.
The following are some issues which I perceive as important.
- Distributed stores v central
I personally strongly favour a distributed archive,
hence my preference for a global namespace (below).
- Keep the format - or convert
i.e. emulation v conversion
I like the OAIS's notion of being able to recover the original
data stream. An easy wasy to do that is to keep the original.
I think that we shall keep something very close to the original.
- Keep the media
examples for the technology,
but not for storing the information
I regard copying of the data as inevitable, due more
to the march of technology than to drop-out.
- Underpinning abstract form (e.g. byte-stream, bitfile, Virtual Storage Object)
personal peference for byte-stream (bit-stream?)
A distributed archive needs such an underlying form in order that
interchange takes place easily.
- How much meta-data is bound to the data?
enough to guarantee subsequent human readability (Representation Information)
plus something to understand the data's origins
I'd love some guidance on this.
It is probably a good idea to attach enough information such that
someone finding a storage volume in the distant future could decode it --
or do we explicitly want to prevent such discoveries?
- Is there any separate meta-data?
surely a good idea
My present personal model has the prospect of
multiple meta-data engines (special-purpose
search facilities) driven by meta-data of all sorts.
- Global namespace
We need a global namespace for digitally archived objects (c.f. ISBN)
At present we have several to choose from, of which
digital object ID (DOI)
seems to be a front runner.
An early agreement on the fomat of such a name seems a good idea.
How about using Internet domain names in the same way as Java package names?
protects against relocation
We can distinguish the storage location from the information.
enables separation of data and meta-data.
enables multiple meta-data implementations
empowers federations of archival stores.
i.e. allows distribution in general.
Each system that we have built has had
more indirection than its predecessors.
We now have all indexing of files and owners by internal names
which are largely invisible.
This allows easy renaming of files and owners in indexing systems,
without implying the need to modify the stored data which is
probably in near-line storage at best and may be off-line.
Experience of George3 makes hierarchical systems the norm
The George3 operating system brought the concept of automatically
managed hierarchical filestore to the UK mainstream in the 1970s.
- First implemented and used hierarchical storage in 1968
Our in-house Eldon2 system used automatic migration between tape and
disk to the extent that the total filestore was about 10 times the on-line
disk space (48 Mbytes).
- Migrated data:
With the LEEDS system we have an architecture designed for indefinite preservaton.
|1960s||Eldon2(in-house)||KDF9 (approx IBM7090)|
|1980s||VM/CMS ||Amdahl 470|
|1990s||LEEDS ||EXB-120 / EXB-480 / ATL4/52
UNIX, NetWare3&4, NT?
|2000s||LEEDS ||absorbing new technology|
- Indirection is vital in a hierarchical system
Our systems have used increasing indirection with each new system
- Versioning of data formats is important
We need to attach version numbers to the volumes and to the stored data objects.
A Librarian's Perspective
From Peter Fox, Librarian at Cambridge University
What I as a librarian want is:
I see Peter's last comment as indicating an archival store which would have
simple access control, but in front of that an access facility that knew all
the complex rights of access, and was allowed access by the archival store.
- Somewhere that I can be sure that a database can be stored securely and
retrieved at any time in the future by anyone who has approved access.
- In our discussions in Cambridge it has emerged that the file stores that
Leeds, Cambridge and Oxford have are just that - stores where the people that
have deposited the data can retrieve it.
- I fear that, in earlier
discussions, 'access' has meant different things to the librarians and
the computer people.
- What I as a librarian want is for there to be an
archive for academia which operates in perpetuity,
like the commercial ones do now.
- Anyone (subject to whatever security/financial arragements were in place)
could have access to archived information.
- This might mean two
separate archives - one like the existing file stores as an 'archival
archive' and one behaving like the commercial suppliers' machines as an
'archive for access'.
The Oxford System
- Uses IBMís ADSM.
- back-up -- mirrors userís file space
- archive -- users can copy fils into the archive
- HSM -- a file system is coupled to ADSM with automatic migration
- Long-term future may be locked with IBM, as ADSM is necessary for moving data
The Cambridge System
- An EPOCH HSM filestore
- is accessed via an in-house user interface
- Data transfer uses FTP
The LEEDS File Archive
A network file archiver written at Leeds University
- Operational in 1992
but it also contains data migrated from the VM/CMS system,
going back into 1980, and raw images of the filestore tapes
of the previous George3 system, which shut down in 1980.
- Based on IEEE-MSS v4
This standard (well-known to the audience) introduced the "bitfile"
concept, and this abstraction is central.
My understanding is that the "virtual storage object" of version 5 of the model
and the "Information Package" of OAIS are almost equivalent.
This diagram (93K JPEG)
shows part of the network of systems
linked into the LEEDS archive.
The purple part is the actual archive system itself.
The rest of the diagram represents the client systems, with UNIX systems on
the left and NetWare systems on the right.
Not shown is the FTP link used to transfer he contents of PC hard disks to
- We own the data format
i.e. data is always accessible
This guarantees us against loss of data owing to supplier bankruptcy
(We don't have chapter 11 in the UK), or system obsolescence.
- Minimalist assumptions about the medium
i.e. write from start, fwd skip, read
no overwriting, no appending
This gives us easy portability, as proven by the move from helical scan
on Solaris to DLT on AIX.
- Copying is inevitable
no virtue in indestructible media
Even with an indestructible medium, the capability for
reading the medium will become obsolete.
Perhaps when we are storing at the quantum-mechanical level this will no
longer be true.
- User interface allows submission to and
recovery from the archive
There is also a facility for getting directory information
and also a (rarely used) facility for the deletion of unwanted files.
formulated by a nationally based working group of 47 UK universities in November 1990.
A university has a large floating population of computer users who own many
It is in the nature of academia that there is much data whose value is not known.
The major points were:
- Files can be sent to the archive from any of the participating file systems on
- Recovery of a file can be onto any system, not necessarily the originating
- The retention of indexing information is done by the system.
- It should be easy for an end-user to rename files.
- The overheads per file must be very small, as many of the files are themselves
- The system should be able to exploit new storage technology seamlessly.
- The system should cope with data for a shifting population of many thousands
- Data should be safe.
- There should be no reliance on operating system modifications.
actual list of requirements and their rationale
are available on the Web.
Overview of concepts in LEEDS system
I suspect that this is much too much for a slide, but the
concepts would be covered by talking.
Many of the properties which I plan to use in the CEDARS demonstrator
map well onto OAIS.
- Unit of storage is the BITFILE
No attempt is made to alter the contents of bitfiles.
They are treated as indivisible units, except that they contain
a header which gives some (but not much) meta-data information.
- A BITFILE has a name (c.f. IP number) and an owner.
An owner concept is vital if access is to be permitted to
This tends to be vital where several systems have access.
- Only the bitfile-ID is required to request information
from the archive. Name space of bitfile-IDs is flat.
- Owners are allocated numbers (another flat namespace).
- Bitfile names are not secret, but only the owner
or a permitted reader can actually receive the data.
- A bitfile is a sequence of bits (bytes actually).
The archive undertakes to deliver the byte-stream
intact on request.
- Against each owner is kept a list of bitfiles with
the full UNC pathname that the file had when it was
- The owner is also given a stub file, which
holds the bitfile-ID. Stub is often kept where the
original file lived, but can be kept anywhere, and can
also be regenerated from the directory in the archive.
- The owner of the file has the responsibility of remembering
how to make sense of the data (or another authorised requester).
Is LEEDS an OAIS?
Well only partly. One can identify traces of
OAIS concepts such as
- Content Information (CI) -- clearly the body of the bitfile,
- Repesentation Information (RI) -- only identity of originating system, and
- Preservation Description Information (PDI) -- details of ownership and origin.
- The goal of indefinite preservation
What can we feed into the OAIS debate?
- We need indirection for digital objects -- we need a digital ISDON now
(International Standard Digital Object Number)
This needs to be the definitive way of referring to objects,
not an optional feature of the PDI.
For CEDARS we will probably plump for one of the current candidates,
but on a national/global scale we could use agreement.
An early agreement on the fomat of such a name seems a good idea.
How about using Internet domain names in the same way as Java does?
-- perhap as a fall-back when no other ID is available.
- We need indirection for user identities (maybe use certificates)
If you do not have some such mechanism, only trusted systems
can be allowed to access the store. They must be trusted
by each other as well as by the archive.
- Is a "collector" a special kind of "producer" or a new element in
the OAIS environment (section 2.1)
In the world of libraries the concept of a collector is
probably different from a producer.
The collector is not the same as the management.
- Include enough PDI with the objects on media to allow last-ditch recovery.
Things sometimes go wrong, and to have enough identity information
physically next to the data helps in recovery operations.
- CEDARS will demonstrate OAIS-style operation in a limited service environment
- It is addressing bothe issues of data storage, and collection management
- If we cannot make the ideas work on a small scale, they certainly
won't work on a national/international scale.