The DRS has experienced three separate problems over the course of the last week that impacted the DRS loader, delivery and management applications. As Vitaly reported earlier this morning, all of these applications are back to operating normally. I wanted to follow-up with an explanation of what went wrong and what we're doing as a result.
Early Monday morning the storage management software used to move content in or out of the DRS storage system temporarily stopped working. The result was that for the first half of Monday new content could not be loaded into the DRS, and content already in the DRS was very slow to appear in applications such as IDS, VIA and OLIVIA. By late Monday morning, OIS system administrators were able to get the software working properly again. This morning the problem briefly reoccurred and was fixed quickly. We continue to investigate the root cause of this problem together with the software vendor (Sun Microsystems).
We are also dealing with a second and unrelated problem that started last Thursday, Oct. 8. The software that continuously monitors the DRS content detected a file system error. This type of problem is resolved by repairing the file system and restoring any corrupt files from backup copies. The repair work began last night and is expected to take 2-3 days. The system is designed so that this work will not affect any DRS applications or processes, and new batches can continue to be loaded into the DRS.
Finally, a routine upgrade of software over the weekend inadvertently caused the DRS loader to send many redundant loader error emails. By Monday afternoon the problem was diagnosed and a technical solution was implemented to prevent this happening after future software upgrades.
We apologize for the trouble these problems caused. If you have questions or concerns about any of this please don't hesitate to contact me.
Thanks,
Andrea
–
Andrea Goethals
Digital Preservation and Repository Services Manager
HUL - Office for Information Systems