DRS METS Archive Tool (Dmart) for Audio Deposit

Source Directory Structure | Dmart tool ( Examples, Configuration, Download, Bugs and Known Issues) | DRS Audio Deposit Package | DRS Database Issues

The Dmart tool reads a source directory of audio files and produces an audio METS deposit package, mets.xml, and a batch.xml file for DRS deposit.

1 Source Directory Structure

All of the audio files for a single audio work are generated or moved to a file system directory tree. The root-level directory name, workid, may be a unique descriptive name appropriate to the audio work. The audio files associated with the work are categorized into five sub-directories based on the function of the files:

workid/
       original/
       archival/
       production/
       deliverable/
       misc/

  1. The original sub-directory contains the optional core audio metadata about the original source audio for the work
  2. The archival sub-directory contains the archival audio files for the work;
  3. The production sub-directory contains the production master audio files for the work;
  4. The deliverable sub-directory contains the the deliverable audio files for the work; and
  5. The misc sub-directory contains any additional files produced during the production process for the audio work.

The various types of files associated with a deposit package are:

  1. Archival master audio (AIFF): [name.aif,name.n] or (Broadcast Wave): [name.wav,name.n.wav]
  2. Production master audio (AIFF): [name.aif,name.n] or (Broadcast Wave): [name.wav,name.n.wav]
  3. Deliverable audio (RealAudio): name.ra
  4. Core audio metadata (XML): name.n.xml
  5. Process history metadata (XML): digiprov.xml
  6. Waveform reduction (binary) name.1.r (from Sonic) or name.gpk (from Wavelab)
  7. AES31 metadata (ASCII): aes31.adl
  8. AES31 fade files (AIFF and XML): in aes31_fades subdirectory
  9. Synchronization (SMIL): name.smi
  10. DMart Batch Override File (XML): [name.dbof.xml or name.n.dbof.xml]
  11. Miscellaneous (binary or XML): name or name.xml

Within the archival, production, and deliverable directories, the audio files for a given work are grouped into multiple segments, each stored in a separate sub-directory, segmentid. Each segment in turn may be composed of multiple channels of audio. If a set of audio is not segmented, it will only have one segment subdirectory. If audio multiple segments (and sub-directories) are present, then an aes31.adl file will be present in the parent directory.

The processing history files, digiprov.xml, can occur:

  1. Once at the level of the archival or production directories, in which case it applies against all segments

The digital audio for a particular segment can be multi-channel, with each channel saved to a separate AIFF or Broadcast Wave file. The AIFF filenames are of the form:

name.n
and Broadcase Wave are of the form:
name.n.wav
where name is an arbitary name assigned to all of the audio channel files for a given audio segment, and n is a numeric value, 1,2,..., identifying the channel. Thus, a stereo pair of AIFFs would be named:
name.1
name.2

Each audio file is associated with a AES core audio metadata file:
name.n.xml
where the numeric value, n, indicates the channel.

Archival or production directories with only a single file can optionally have the same naming convention minus the numeric value, 1,2,....

name.aif
or name.wav

Regardless of the number of channels of audio stored for a segment, there may be a single external waveform reduction file, with a filename of the form:

name.1.r
from Sonic, or
name.gpk
from Wavelab.

The deliverable RealAudio files are of the form name.ra

The deliverable directory may contain SMIL files, name.smi to provide appropriate delivery synchronization of the RealAudio files.

DRS batch metadata for submission may be overriden for each content file with the DMart Batch Override File (dbof). One can add or change metadata values for the generated batch.xml file on a per-audio-file basis. Here's an example file which specifies a different DRS ownerSuppliedName (local id) than the default, which is the the file path and name:

<batch>
  <ownerSuppliedName>my-special-id</ownerSuppliedName>
</batch>

DMart Batch Override Files have the filename form name.n.dbof.xml or name.dbof.xml.

The following diagram illustrates the source directory tree structure in standard Unix notation. Brackets [ and ] enclose optional elements.

workid/
        original/
                    [ coreaudio.xml ]
        archival/
                    [ aes31.adl ]
                    [ digiprov.xml ]
                      segmentid1/
                                  name.1.xml  or  name.xml
                                [ name.2.xml ...
                                  name.n.xml ]
                                  name.1  or  name.aif  or  name.1.wav
                                [ name.2  or  name.2.wav  ...
                                  name.n  or  name.n.wav ]
                                [ name.1.r  or  name.gpk ]
                                [ name.1.dbof.xml  or  name.dbof.xml ]
                    [ segmentid2/
                                  ... ]
                    [ aes31_fades/
                                  name.xml
                                [ name.n.xml ]
                                  name  or name.aif  or  name.wav

                                  ... ]
                    [ ... ]
        production/
                    [ aes31.adl ]
                    [ digiprov.xml ]
                      segmentid1/
                                  name.1.xml  or  name.xml
                                [ name.2.xml ...
                                  name.n.xml ]
                                  name.1  or  name.aif  or  name.1.wav
                                [ name.2  or  name.2.wav  ...
                                  name.n  or  name.n.wav ]
                                [ name.1.r  or  name.gpk ]
                                [ name.1.dbof.xml  or  name.dbof.xml ]
                    [ segmentid2/
                                  ... ]
                    [ ... ]
        deliverable/
                    [ digiprov.xml ]
                    [ name.smi ]
                      segmentid1/
                                  name.1.xml
                                [ name.2.xml ...
                                  name.n.xml ]
                                  name.1.ra
                                [ name.2.ra ...
                                  name.n.ra ]
                                [ name.1.dbof.xml  or  name.dbof.xml ]
                    [ segmentid2/
                                  ... ]
                    [ ... ]
        misc/
                    [ name  or  name.xml ]
                    [ ... ]

2 Dmart

The Dmart tool processes an audio source directory and produces a DRS deposit package.

The invocation syntax for Dmart is:

./dmart.sh -i workid | -x metsfile [-hv] [-a access] [-b billing] [-c config]
     [-l label] [-o output ] [-s status] [-u urnmask] [-w ownercode] [-z buffer] directory

-i specifies that you are supplying a workid and creating a METS package for deposit. -x specifies you are extracting data from a METS package, previously created with the dmart tool.

Options for Packaging (-i)

where-h displayes a help message;
-v displays the Dmart version;
-a access specifies the access flag for deliverables: "P" for public access (default) or "R" for restricted access;
-b billing specifies the DRS billing code;
-c config specifies the configuration file (defaults to "dmart.conf");
-g agent specifies the depositing agent name;
-i workid specifies the audio work ID (required);
-l label specifies the audio work label;
-o output specifies the METS file output file name (defaults to "mets.xml");
-s status specifies the audio work status: "PROD", "QA", "DEV" (default);
-u urnmask specifies the URN mask for assigning persistent identifiers for the deliverable files;
-w ownercode specifies the DRS owner code;
-z buffer specifies the MD5 checksum buffer (defaults to 131072); and
directory specifies the source directory (required).

Note that arguments containing whitespace (e.g., workid, label) must be enclosed in "quotation marks."

Options for Extracting (-x)

where-x metsfile specifies path of the metsfile you are extracting
directory specifies the directory to extract into (required).

2.1 Examples

Here's some examples of how to use the tool.

Example packaging

./dmart.sh -c conf/dmart.conf.example -i 12345 example

uses the configuration values in dmart.conf.example, workid 12345 to package the files in directory example and create the following files in your current directory:

batch.xml
mets.xml

You would then sftp the batch.xml, mets.xml and the example directory to the "incoming" directory of your sftp drop box.

Example extracting

Extraction is very similar to unzipping or un-tarring a packaged source file. Here we extract the mets.xml into a test directory.

[julian@fledge drs_dmart]$ ./dmart.sh -x mets.xml /users/julian/dmart_test
creating file: /users/julian/junk/dmart_test/archivalCreator
creating file: /users/julian/junk/dmart_test/deliverableCreator
creating file: /users/julian/junk/dmart_test/productionCreator
creating file: /users/julian/junk/dmart_test/smilCreator
creating file: /users/julian/junk/dmart_test/waveformCreator
creating file: /users/julian/junk/dmart_test/original/coreaudio.xml
creating file: /users/julian/junk/dmart_test/archival/seg1/name.1.xml
creating file: /users/julian/junk/dmart_test/archival/seg2/name.1.xml
creating file: /users/julian/junk/dmart_test/production/seg1/name.1.xml
creating file: /users/julian/junk/dmart_test/production/seg2/name.2.xml
creating file: /users/julian/junk/dmart_test/deliverable/seg1/name.1.xml
creating file: /users/julian/junk/dmart_test/deliverable/seg2/name.2.xml
creating file: /users/julian/junk/dmart_test/misc/extra.xml
creating file: /users/julian/junk/dmart_test/archival/aes31.adl
creating file: /users/julian/junk/dmart_test/production/aes31.adl
creating file: /users/julian/junk/dmart_test/misc/PMasterTrack1edits
creating file: /users/julian/junk/dmart_test/misc/PMasterTrack2edits

2.2 Configuration

The Dmart configuration file is formatted as:

<configFile app="Dmart">
 <accessFlag>access</accessFlag>
 <batchSchema>batchuri</batchSchema>
 <billingCode>billing</access>
 <emailSuccess>success</emailSuccess>
 <emailFailure>failure</emailFailure>
 <ownerCode>ownercode</access>
 <service>service</service>
 <urnMask>urnmask</urnmark>

 <productionUsage>LOWUSE</productionUsage>
 <archivalUsage>LOWUSE</archivalUsage>
 <deliverableUsage>HIGHUSE</deliverableUsage>
 <metsUsage>LOWUSE</metsUsage>

 <agentName>agent</agentName>
 <label>label</label>
 <status>status</status>
 <workID>workID</workID>

 <archivalCreator>acreate</archivalCreator>
 <deliverableCreator>dcreate</deliverableCreator>
 <productionCreator>pcreate</productionCreator>
 <smilCreator>screate</smilCreator>
 <waveformCreator>wcreate</waveformCreator>

 <saxParser>parser</saxParser>
 <bufferSize>buffer</bufferSize>
<configFile>

whereaccess is the access flag: "P" for public access (default) or "R" for restricted access;
batchuri URI of the batch.xml DTD (defaults to http://hul.harvard.edu/ois/xml/xsd/drs/drs_batch.dtd);
billing is the DRS billing code;
success comma-separated list of email addresses to receive deposit success confirmation (required);
failure comma-separated list of email addresses to receive deposit failure notification (required);
ownercode is the DRS owner code;
service is the delivery service for deliverable files (defaults to STREAMING_DELIVERY_SERVICE);
urnmask is the URN mask for naming deliverable files;
agent is the deposting agent name;
label is the audio work label;
status is the audio work status: "PROD", "QA", "DEV" (default);
workid is the audio work ID;
acreator is the Mac OS filetype for the creating software for the archival files;
dcreator is the Mac OS filetype for the creating software for the delivery files;
pcreator is the Mac OS filetype for the creating software for the production files;
screator is the Mac OS filetype for the creating software for the SMIL files;
wcreator is the MacOS filetype for the creating software for the waveform files;
parser is the SAX parser class (defaults to org.apache.xerces.parsers.SAXParser); and
buffer is the MD5 checksum buffer size (defaults to 131072).

The configuration file is read first, and can be overridden by values specified on the command line.

An example configuration file is available in the example directory of the relase.

2.3 Download

The latest version of drs_dmart-1.1.9.tar.gz (1-31-2008).

It can be uncompressed and disaggregated by the following commands:

gunzip drs_dmart-x.y.z.tar.gz
tar xvf drs_dmart-x.y.z.tar
to form the following directory structure:
dmart-x-y-z/
           dmart.sh                shell script to run dmart
           LICENSE                 GPL license
           README.txt              simple readme file
           RELEASE.txt             changes between dmart versions
           lib/                    necessary jars including drs_dmart.jar
           classes/                source code
           conf/                   configuration directory with example
           example/                complete example directory structure

2.4 Bugs and Known Issues

The following are known bugs and issues:

3 DRS Audio Deposit Package

A compliant DRS audio deposit package is created by the Dmart tool from an audio source directory.

The deposit package includes:

  1. batch.xml
  2. mets.xml, with encapsulated core audio and process history metadata files (name.n.xml, digiprov.xml), AES31 files (aes31.adl), and miscellaneous processing files; and pointers to external files:
    1. Archival master audio, name.aif or name.n or name.n.wav
    2. Production master audio, name.aif or name.n or name.n.wav
    3. Deliverable RealAudio files, name.n.ra
    4. Waveform reduction files, name.1.r or name.gpk
    5. SMIL files, name.smi

3.1 METS file

The general form of the METS file is as follows.

Note that since the traditional character marking directory separation ('/') is not allowed by the ID/IDREF mechanism in XML, the file IDs are encoded with a replacement separator ('-_-').

<?xml version="1.0" standalone="no"?>
<mets OBJID="workid" LABEL="label" TYPE="AUDIO_SIP" PROFILE="Harvard" ... >
  <metsHdr ...>
    <agent ROLE="creator" TYPE="OTHER" OTHERTYPE="DepositingAgent">
      <name>depositing agent name</name>
    </agent>
  </metsHdr>
  <amdSec ID="AmdSec">
    <techMD ID="TechMD">
      <mdWrap ID="original-_-coreaudio.xml" MIMETYPE="text/xml" ...>
        <xmlData>
          <coreAudioMD ...>
            ...
          </coreAudioMD>
        </xmlData>
      </mdWrap>
      ...
    </techMD>
  </amdSec>
  <fileSec>
    <fileGrp ID="AES31">
      <file ID="archival-_-aes31.adl" MIMETYPE="text/plain" ...>
        <FContent>
          <binData>...base64-encoded data...</binData>
        </FContent>
      </file>
      ... 
    </fileGrp>
    <fileGrp ID="AIFF">
      <file ID="archival-_-segmentid1" MIMETYPE="audio/x-aiff" ...>
        <FLocat ... xlink:href="archival-_-segmentid1-_-name.1">
      </file>
      ...
    </fileGrp>
    <fileGrp ID="Misc"> ... </fileGrp>
    <fileGrp ID="Real"> ... </fileGrp>
    <fileGrp ID="SMIL"> ... </fileGrp>
    <fileGrp ID="Wave"> ... </fileGrp>
    <fileGrp ID="Waveform"> ... </fileGrp>
  </fileSec>
  <structMap TYPE="LOGICAL">
    <div LABEL="archival">
      <fptr FILEID="archival-_-aes31.adl"/>
      <div LABEL="archival-_-segmentid1">
        <fptr FILEID="archival-_-segmentid1/name.1"
        ...
      </div>
      ...
    </div>
    <div LABEL="production"> ... </div>
    <div LABEL="deliverable"> ... </div>
    <div LABEL="misc"> ... </div>
  </structMap>
</mets>

The METS schema and documentation is available from the METS website.

4 DRS Database Issues

Technical metadata properties about the audio files is extracted from the core audio (name.n.xml) files and copied into the appropriate fields of the audioMetadata block of the batch.xml file, as follows:

DRS batch.xmlcoreAudio.n.xmlNote
<bitdepth><bitDepth>
<blocksize> <audioDataBlockSize>
<channelassign> <channelAssignment>
<channelmap> <formatRegion> and <region> data is distributed underneath both
<channelnumber> <channelNum>
<channels> <numChannels>
<codec> <bitrateReduction>
<codecname> <codecName>
<codecnameversion> <codecNameVersion>
<codeccreator> <codecCreatorApplication>
<codeccreatorversion> <codecCreatorApplicationVersion>
<codecquality> <codecQuality> 0 = CODE_REGENERATING, 1 = LOSSY
<dataformattype> <audioDataEncoding>
<dataorientation> <byteOrder> 0 = BIG_ENDIAN, 1 = LITTLE_ENDIAN
<datarate><dataRate>
<dataratemode> <dataRateMode> 0 = FIXED, 1 = VARIABLE
<duration><value> if parent element is <duration>
<firstvalidbyte> <firstValidByteOfBlock>
<lastvalidbyte> <lastValidByteOfBlock>
<maplocation> <mapLocation>
<offset> <firstSampleOffset>
<samplerate> <sampleRate>
<soundfield> <soundField>
<timestampend><value> if parent element is <endTime>
<timestampstart><value> if parent element is <startTime>
<wordsize><wordSize>

Metadata TypeDescriptorRepertoire CharmapMIME typeRoleFiles
TEXT AUDIO_SIP * Unicode * UTF-8 * text/xml NA mets.xml
SMIL * DELIVERABLE name.smi
AUDIO audio/x-aiff
audio/x-wave
ARCHIVAL_MASTER channel.n
channel.n.wav
PRODUCTION_MASTER channel.n
channel.n.wav
audio/x-pn-realaudio DELIVERABLE channel.n.ra
APPLICATION * WAVEFORM_REDUCTION * application/x-sonic-waveform-reduction *
application/x-wavelab-waveform-reduction *
NA channel.1.r
channel.gpk

An asterisk (*) indicates new values added to DRS specifically for audio.

The following DRS relationships are specified:

  1. IS_PART_OF
  2. IS_RELATED_CHANNEL_OF *
  3. IS_WAVEFORM_FOR *

An asterisk (*) indicates new values added to the DTD specifically for audio.


Source Directory Structure | Dmart tool ( Examples, Configuration, Download) | DRS Audio Deposit Package

Copyright 2008 by the President and Fellows of Harvard College

Rev.: 2008/1/31 - SM