Source Directory Structure | Dmart tool ( Examples, Configuration, Download, Bugs and Known Issues) | DRS Audio Deposit Package | DRS Database Issues
The Dmart tool reads a source directory of audio files and produces an audio METS deposit package, mets.xml, and a batch.xml file for DRS deposit.
All of the audio files for a single audio work are generated or moved to a file system directory tree. The root-level directory name, workid, may be a unique descriptive name appropriate to the audio work. The audio files associated with the work are categorized into five sub-directories based on the function of the files:
workid/
original/
archival/
production/
deliverable/
misc/
The various types of files associated with a deposit package are:
Within the archival, production, and deliverable directories, the audio files for a given work are grouped into multiple segments, each stored in a separate sub-directory, segmentid. Each segment in turn may be composed of multiple channels of audio. If a set of audio is not segmented, it will only have one segment subdirectory. If audio multiple segments (and sub-directories) are present, then an aes31.adl file will be present in the parent directory.
The processing history files, digiprov.xml, can occur:
The digital audio for a particular segment can be multi-channel, with each channel saved to a separate AIFF or Broadcast Wave file. The AIFF filenames are of the form:
and Broadcase Wave are of the form:name.n
where name is an arbitary name assigned to all of the audio channel files for a given audio segment, and n is a numeric value, 1,2,..., identifying the channel. Thus, a stereo pair of AIFFs would be named:name.n.wav
Each audio file is associated with a AES core audio metadata file:name.1 name.2
where the numeric value, n, indicates the channel.name.n.xml
Archival or production directories with only a single file can optionally have the same naming convention minus the numeric value, 1,2,....
name.aif or name.wav
Regardless of the number of channels of audio stored for a segment, there may be a single external waveform reduction file, with a filename of the form:
from Sonic, orname.1.r
from Wavelab.name.gpk
The deliverable RealAudio files are of the form name.ra
The deliverable directory may contain SMIL files, name.smi to provide appropriate delivery synchronization of the RealAudio files.
DRS batch metadata for submission may be overriden for each content file with the DMart Batch Override File (dbof). One can add or change metadata values for the generated batch.xml file on a per-audio-file basis. Here's an example file which specifies a different DRS ownerSuppliedName (local id) than the default, which is the the file path and name:
<batch> <ownerSuppliedName>my-special-id</ownerSuppliedName> </batch>
DMart Batch Override Files have the filename form name.n.dbof.xml or name.dbof.xml.
The following diagram illustrates the source directory tree structure in standard Unix notation. Brackets [ and ] enclose optional elements.
workid/
original/
[ coreaudio.xml ]
archival/
[ aes31.adl ]
[ digiprov.xml ]
segmentid1/
name.1.xml or name.xml
[ name.2.xml ...
name.n.xml ]
name.1 or name.aif or name.1.wav
[ name.2 or name.2.wav ...
name.n or name.n.wav ]
[ name.1.r or name.gpk ]
[ name.1.dbof.xml or name.dbof.xml ]
[ segmentid2/
... ]
[ aes31_fades/
name.xml
[ name.n.xml ]
name or name.aif or name.wav
... ]
[ ... ]
production/
[ aes31.adl ]
[ digiprov.xml ]
segmentid1/
name.1.xml or name.xml
[ name.2.xml ...
name.n.xml ]
name.1 or name.aif or name.1.wav
[ name.2 or name.2.wav ...
name.n or name.n.wav ]
[ name.1.r or name.gpk ]
[ name.1.dbof.xml or name.dbof.xml ]
[ segmentid2/
... ]
[ ... ]
deliverable/
[ digiprov.xml ]
[ name.smi ]
segmentid1/
name.1.xml
[ name.2.xml ...
name.n.xml ]
name.1.ra
[ name.2.ra ...
name.n.ra ]
[ name.1.dbof.xml or name.dbof.xml ]
[ segmentid2/
... ]
[ ... ]
misc/
[ name or name.xml ]
[ ... ]
The Dmart tool processes an audio source directory and produces a DRS deposit package.
The invocation syntax for Dmart is:
./dmart.sh -i workid | -x metsfile [-hv] [-a access] [-b billing] [-c config]
[-l label] [-o output ] [-s status] [-u urnmask] [-w ownercode] [-z buffer] directory
-i specifies that you are supplying a workid and creating a METS package for deposit. -x specifies you are extracting data from a METS package, previously created with the dmart tool.
where -h displayes a help message; -v displays the Dmart version; -a access specifies the access flag for deliverables: "P" for public access (default) or "R" for restricted access; -b billing specifies the DRS billing code; -c config specifies the configuration file (defaults to "dmart.conf"); -g agent specifies the depositing agent name; -i workid specifies the audio work ID (required); -l label specifies the audio work label; -o output specifies the METS file output file name (defaults to "mets.xml"); -s status specifies the audio work status: "PROD", "QA", "DEV" (default); -u urnmask specifies the URN mask for assigning persistent identifiers for the deliverable files; -w ownercode specifies the DRS owner code; -z buffer specifies the MD5 checksum buffer (defaults to 131072); and directory specifies the source directory (required). Note that arguments containing whitespace (e.g., workid, label) must be enclosed in "quotation marks."
where -x metsfile specifies path of the metsfile you are extracting directory specifies the directory to extract into (required).
Here's some examples of how to use the tool.
./dmart.sh -c conf/dmart.conf.example -i 12345 example
uses the configuration values in dmart.conf.example, workid 12345 to package the files in directory example and create the following files in your current directory:
batch.xml mets.xml
You would then sftp the batch.xml, mets.xml and the example directory to the "incoming" directory of your sftp drop box.
Extraction is very similar to unzipping or un-tarring a packaged source file. Here we extract the mets.xml into a test directory.
[julian@fledge drs_dmart]$ ./dmart.sh -x mets.xml /users/julian/dmart_test creating file: /users/julian/junk/dmart_test/archivalCreator creating file: /users/julian/junk/dmart_test/deliverableCreator creating file: /users/julian/junk/dmart_test/productionCreator creating file: /users/julian/junk/dmart_test/smilCreator creating file: /users/julian/junk/dmart_test/waveformCreator creating file: /users/julian/junk/dmart_test/original/coreaudio.xml creating file: /users/julian/junk/dmart_test/archival/seg1/name.1.xml creating file: /users/julian/junk/dmart_test/archival/seg2/name.1.xml creating file: /users/julian/junk/dmart_test/production/seg1/name.1.xml creating file: /users/julian/junk/dmart_test/production/seg2/name.2.xml creating file: /users/julian/junk/dmart_test/deliverable/seg1/name.1.xml creating file: /users/julian/junk/dmart_test/deliverable/seg2/name.2.xml creating file: /users/julian/junk/dmart_test/misc/extra.xml creating file: /users/julian/junk/dmart_test/archival/aes31.adl creating file: /users/julian/junk/dmart_test/production/aes31.adl creating file: /users/julian/junk/dmart_test/misc/PMasterTrack1edits creating file: /users/julian/junk/dmart_test/misc/PMasterTrack2edits
The Dmart configuration file is formatted as:
<configFile app="Dmart"> <accessFlag>access</accessFlag> <batchSchema>batchuri</batchSchema> <billingCode>billing</access> <emailSuccess>success</emailSuccess> <emailFailure>failure</emailFailure> <ownerCode>ownercode</access> <service>service</service> <urnMask>urnmask</urnmark> <productionUsage>LOWUSE</productionUsage> <archivalUsage>LOWUSE</archivalUsage> <deliverableUsage>HIGHUSE</deliverableUsage> <metsUsage>LOWUSE</metsUsage> <agentName>agent</agentName> <label>label</label> <status>status</status> <workID>workID</workID> <archivalCreator>acreate</archivalCreator> <deliverableCreator>dcreate</deliverableCreator> <productionCreator>pcreate</productionCreator> <smilCreator>screate</smilCreator> <waveformCreator>wcreate</waveformCreator> <saxParser>parser</saxParser> <bufferSize>buffer</bufferSize> <configFile>
| where | access | is the access flag: "P" for public access (default) or "R" for restricted access; |
| batchuri | URI of the batch.xml DTD (defaults to http://hul.harvard.edu/ois/xml/xsd/drs/drs_batch.dtd); | |
| billing | is the DRS billing code; | |
| success | comma-separated list of email addresses to receive deposit success confirmation (required); | |
| failure | comma-separated list of email addresses to receive deposit failure notification (required); | |
| ownercode | is the DRS owner code; | |
| service | is the delivery service for deliverable files (defaults to STREAMING_DELIVERY_SERVICE); | |
| urnmask | is the URN mask for naming deliverable files; | |
| agent | is the deposting agent name; | |
| label | is the audio work label; | |
| status | is the audio work status: "PROD", "QA", "DEV" (default); | |
| workid | is the audio work ID; | |
| acreator | is the Mac OS filetype for the creating software for the archival files; | |
| dcreator | is the Mac OS filetype for the creating software for the delivery files; | |
| pcreator | is the Mac OS filetype for the creating software for the production files; | |
| screator | is the Mac OS filetype for the creating software for the SMIL files; | |
| wcreator | is the MacOS filetype for the creating software for the waveform files; | |
| parser | is the SAX parser class (defaults to org.apache.xerces.parsers.SAXParser); and | |
| buffer | is the MD5 checksum buffer size (defaults to 131072). |
The configuration file is read first, and can be overridden by values specified on the command line.
An example configuration file is available in the example directory of the relase.
The latest version of drs_dmart-1.1.9.tar.gz (1-31-2008).
It can be uncompressed and disaggregated by the following commands:
to form the following directory structure:gunzip drs_dmart-x.y.z.tar.gz tar xvf drs_dmart-x.y.z.tar
dmart-x-y-z/
dmart.sh shell script to run dmart
LICENSE GPL license
README.txt simple readme file
RELEASE.txt changes between dmart versions
lib/ necessary jars including drs_dmart.jar
classes/ source code
conf/ configuration directory with example
example/ complete example directory structure
The following are known bugs and issues:
A compliant DRS audio deposit package is created by the Dmart tool from an audio source directory.
The deposit package includes:
The general form of the METS file is as follows.
Note that since the traditional character marking directory separation ('/') is not allowed by the ID/IDREF mechanism in XML, the file IDs are encoded with a replacement separator ('-_-').
<?xml version="1.0" standalone="no"?>
<mets OBJID="workid" LABEL="label" TYPE="AUDIO_SIP" PROFILE="Harvard" ... >
<metsHdr ...>
<agent ROLE="creator" TYPE="OTHER" OTHERTYPE="DepositingAgent">
<name>depositing agent name</name>
</agent>
</metsHdr>
<amdSec ID="AmdSec">
<techMD ID="TechMD">
<mdWrap ID="original-_-coreaudio.xml" MIMETYPE="text/xml" ...>
<xmlData>
<coreAudioMD ...>
...
</coreAudioMD>
</xmlData>
</mdWrap>
...
</techMD>
</amdSec>
<fileSec>
<fileGrp ID="AES31">
<file ID="archival-_-aes31.adl" MIMETYPE="text/plain" ...>
<FContent>
<binData>...base64-encoded data...</binData>
</FContent>
</file>
...
</fileGrp>
<fileGrp ID="AIFF">
<file ID="archival-_-segmentid1" MIMETYPE="audio/x-aiff" ...>
<FLocat ... xlink:href="archival-_-segmentid1-_-name.1">
</file>
...
</fileGrp>
<fileGrp ID="Misc"> ... </fileGrp>
<fileGrp ID="Real"> ... </fileGrp>
<fileGrp ID="SMIL"> ... </fileGrp>
<fileGrp ID="Wave"> ... </fileGrp>
<fileGrp ID="Waveform"> ... </fileGrp>
</fileSec>
<structMap TYPE="LOGICAL">
<div LABEL="archival">
<fptr FILEID="archival-_-aes31.adl"/>
<div LABEL="archival-_-segmentid1">
<fptr FILEID="archival-_-segmentid1/name.1"
...
</div>
...
</div>
<div LABEL="production"> ... </div>
<div LABEL="deliverable"> ... </div>
<div LABEL="misc"> ... </div>
</structMap>
</mets>
The METS schema and documentation is available from the METS website.
Technical metadata properties about the audio files is extracted from the core audio (name.n.xml) files and copied into the appropriate fields of the audioMetadata block of the batch.xml file, as follows:
| DRS batch.xml | coreAudio.n.xml | Note |
|---|---|---|
| <bitdepth> | <bitDepth> | |
| <blocksize> | <audioDataBlockSize> | |
| <channelassign> | <channelAssignment> | |
| <channelmap> | <formatRegion> and <region> | data is distributed underneath both |
| <channelnumber> | <channelNum> | |
| <channels> | <numChannels> | |
| <codec> | <bitrateReduction> | |
| <codecname> | <codecName> | |
| <codecnameversion> | <codecNameVersion> | |
| <codeccreator> | <codecCreatorApplication> | |
| <codeccreatorversion> | <codecCreatorApplicationVersion> | |
| <codecquality> | <codecQuality> | 0 = CODE_REGENERATING, 1 = LOSSY |
| <dataformattype> | <audioDataEncoding> | |
| <dataorientation> | <byteOrder> | 0 = BIG_ENDIAN, 1 = LITTLE_ENDIAN |
| <datarate> | <dataRate> | |
| <dataratemode> | <dataRateMode> | 0 = FIXED, 1 = VARIABLE |
| <duration> | <value> | if parent element is <duration> |
| <firstvalidbyte> | <firstValidByteOfBlock> | |
| <lastvalidbyte> | <lastValidByteOfBlock> | |
| <maplocation> | <mapLocation> | |
| <offset> | <firstSampleOffset> | |
| <samplerate> | <sampleRate> | |
| <soundfield> | <soundField> | |
| <timestampend> | <value> | if parent element is <endTime> |
| <timestampstart> | <value> | if parent element is <startTime> |
| <wordsize> | <wordSize> |
| Metadata Type | Descriptor | Repertoire | Charmap | MIME type | Role | Files |
|---|---|---|---|---|---|---|
| TEXT | AUDIO_SIP * | Unicode * | UTF-8 * | text/xml | NA | mets.xml |
| SMIL * | DELIVERABLE | name.smi | ||||
| AUDIO | audio/x-aiff audio/x-wave |
ARCHIVAL_MASTER | channel.n channel.n.wav |
|||
| PRODUCTION_MASTER | channel.n channel.n.wav |
|||||
| audio/x-pn-realaudio | DELIVERABLE | channel.n.ra | APPLICATION * | WAVEFORM_REDUCTION * | application/x-sonic-waveform-reduction * application/x-wavelab-waveform-reduction * |
NA | channel.1.r channel.gpk |
An asterisk (*) indicates new values added to DRS specifically for audio.
The following DRS relationships are specified:
An asterisk (*) indicates new values added to the DTD specifically for audio.
Source Directory Structure | Dmart tool ( Examples, Configuration, Download) | DRS Audio Deposit Package
Copyright 2008 by the President and Fellows of Harvard CollegeRev.: 2008/1/31 - SM