Abstract

A group of analysts were having trouble understanding the track followed by vessels in exercises from data collated from a number of sensors, They had a third party file converter for extracting data from Sonar S2087. When S2087’s file formats changed in 2014, the 3rd party converter became obsolete, and the analysts were stuck trying to manually convert rows upon rows of data using MS Excel. Each exercise required manually converting at least 5 sets of data, which required the whole afternoon.

To enable analysts to spend more time analysing and less time battling with MS Excel, thereby speeding up the analysis process, Ian (from Deep Blue C) developed a file importer for data in the Sonar S2087 data format.

The analysts no longer rely on a proprietary file converter over which they have no direct control – whenever they encounter the data, they can drag it right in. They can seamlessly load the data and start analysis, without frustrating experience of having to act as Data Technician as well as Analyst.

A wider body of analysts decided that they required an ongoing support contract – to ensure upgrades for such impromptu requirements were available .

Customer

DSTL

Problem

A group of analysts collate data from a range of sensors. Frequently it’s a challenge to get the path (track) followed by vessels in exercises and operations. So, the analysts get well versed in collecting vessel tracks from a range of sources, including Sonar S2087.

This body of analysts had a third-party file converter that was able to convert data extracted from Sonar S2087. But, format in which the S2087 data arrived had changed, and the third-party file converter no longer worked. Without a support contract for the third-party converter, they were left trying to load the data into MS Excel, then re-arrange/re-format the columns by hand to produce data that could be imported into Debrief. Plus, they had to do this for at least 5 data files before they could start to analyse the exercise – a task that would probably take all afternoon.

Whilst a typical science graduate should be able to fling data around in MS Excel quite quickly, an analyst with a typical background of years at sea is less able to do this.
This file import problem was going to cause a number of problems:

• there would be a delay before analysis could commence
• a number of analysts would be spending their time battling with MS Excel rather than delivering value to their government customer

Process

During an onsite support visit, an analyst explained the problem to Ian. Ian explained that there were three options:

The analysts could hire a technician to reformat the data in advance. This would be a costly solution long term, and wouldn’t really speed up the process.

Ian could spend a couple of hours giving the analysts some training in the necessary MS Excel skills, so that the analysts could attempt the transfer themselves in the future. Again, it still relied on the analysts still spending analysis time in getting data in the right order.

Ian could develop a file importer for data in the Sonar S2087 data format. This would allow the analysts to drag-drop such files directly into Debrief in the future. This would automate the data formatting process, letting the analysts get straight to work.

The analysts decided that Option 3 represented the greatest benefit to their ongoing analysis.

Solution

 

• We identified a snippet of sample data the analysts confirmed wasn’t sensitive, and took a copy of the file to work with.

• Back at the office we duplicated this line (in the Debrief text editor), and introduced sufficient variety in the data to validate the import

• We wrote the new drag/drop import functionality, and verified that the data would load.

• Next, the Debrief analysis tools were used to verify loaded track against sample data

• Once completed, a trial version of Debrief was delivered to the client site to verify on their network against real data.
• The new format was recorded in the reference section of the Debrief User Manual, in HTML form

And in printed PDF form.

• We learn’t that the file suffix on the sample had been amended, and returned later that day with updated importer
• The analysts verified file import against their series of data-files, and application update was accepted

Long Term

This body of users have recognised the benefits of Debrief being extended to support the file formats that they encounter. For S2087 they no longer rely on a proprietary file converter over which they have no direct control – whenever they encounter the data, they can drag it right in. They can seamlessly load the data and start analysis, without frustrating experience of having to act as Data Technician as well as Analyst.

Looking Forward

This project demonstrated how how valuable bespoke support is, while helping us realize that file formats and data conversion are tasks that eat up a lot of precious work time for analysts. File formats often get ignored because they aren’t thought much about, and because people get used to them after a while, not realizing how much time it’s costing them. Debrief support is now able to offer custom file format support for it’s users, and is happy to be able to improve analyst’s efficiency.