Data Processing Applications
What these are:
These applications are Perl scripts packaged as Windows executable files or Mac OS X applications. The underlying scripts in the Windows version are exactly the same as those used in the Mac OS X apps. Each app is a stand-alone program that does not need to be installed, you can simply run it from a USB drive or its current directory. You do not need to have administrative permission* or any other required programs installed for the scripts to run.
The Windows apps will install into the ‘Program Files’ / ‘Program Files (x86)’ folders by default. If you do not have permission to write to these directories, you can choose to install them into any folder you have permission to modify. On Mac OS X, the installer application attempts to write the apps to the Applications folder. This does require administrative permissions. However, on OS X or Windows, the programs themselves should run just fine if you simply copy them off of a USB drive into your Home folder rather than using the installer app. Hide Note
What these do:
The apps are designed to automate processing of data files generated by the Echo Medical Systems EchoMRI, Columbus Instruments calorimeter, MedAssociates open field activity systems, and Lafayette running wheels. Data from these systems are handled by the Echo2CSV, OxyMaxer, Spa2CSV/SPA Grouper, and Wheel2CSV apps, respectively.
How to use these:
Short answer: Drag a folder full of data files onto the program icon. The app will generate a folder with a new file containing all the processed data. The output file(s) can be opened directly in Excel to view or analyze the data. That’s it.
On Mac OS X, the apps open a small program window that will accept dropped files. On Windows, the program window has buttons to select and process files or folders. On both platforms, the program window will remain open until you quit the program.
Long answer: Any files or folders dropped on the app will be searched for matching data files. The Echo2CSV app looks for .tab files, the OxyMaxer app takes .csv files, and the Spa2CSV app accepts files ending in .Summary or .Zone (all of these are the default file types output by the behavioral testing equipment).
Matching files are scanned to extract data. For each app, all extracted data is processed to reorganize it, sorting alphanumerically by subject ID (and optionally, by treatment group - see below), and then processed data is written to an output file. The output file(s) will be placed in a folder named ‘Processed’, inside the original data folder. If you dropped more than one folder full of data on the app, each of these folders will get its own output file.
The output file(s) are saved as comma separated value (.csv) files. You can open these directly into Excel to view or analyze them. Output file for most apps is saved as ‘combined.csv’. If a file with this name exists, the new output file is named ‘combined-1.csv’, ‘combined-2.csv’, etc. to avoid overwriting existing files. I tried to organize output files such that data is in logical groups and is as easy to pass to a stats program as possible. Some long-standing issues with data files are hopefully addressed by these scripts (see below).
Specifics for each program:
Echo2CSV: All data is sorted into columns, with all data from one day in each column. Fat mass, lean mass, total and free water are arranged in separate blocks of data, each sorted by subject ID. If there was no data for one subject on a given day, no value is written into the output table (leaving a blank cell). Note: Weights entered into the Echo when scanning are not reported in the output file. (If this is an issue, it can be fixed - just ask and I will update the script.)
OxyMaxer: All data is sorted into rows by measurement type, each sorted by subject ID. All data is aligned by the light-dark measurement: If lights did not turn off, the data will be aligned by timestamp instead. This may or may not be as accurate, so be sure to check your data!. Animal weights and test chamber assignments (by test date) are printed in a block at the top of the sheet. The very first data block is the time, saved in Excel format: You will need to manually select this block of data and format it (Format Cells --> Time and Date --> choose desired time format, e.g. “HH:MM:SS”) to properly view the timestamps. I typically decide which column of data should be used to start and end analysis, and delete anything before or after this set of columns to reduce file size.
Spa2CSV: This program is slightly different from the others. You will get one output file per animal per test date. Files are named in the pattern ‘YYYY-MM-DD_SubjectID.csv’. Each file will have all data for that animal, along with start date, start time, and a few other important points. Data is arranged in rows by measurement type, with columns sorted by timestamp. Time Spent Moving is automatically calculated and placed in the first data row. All timestamps are either reported as minutes (in data blocks) or are converted to Excel format (in timestamps). Time in minutes is expressed as a decimal (i.e., 1 min 30 sec would be reported as 1.5): Time in data blocks thus does not need to be reformatted in Excel; data is reported as a decimal number and can be used as-is. For timestamps indicating the actual clock time each datapoint was acquired, you will need to format cells as described under OxyMaxer (above). Note: Because there may be times when you wish to exclude a specific subject, or combine files from different test days, the Spa2CSV app does not attempt to lump data into a single file. To organize your data into a single file, use the Spa Grouper app (below).
Spa Grouper: This app is used to organize individual output files from the Spa2CSV app into single combined files for analysis. You can either drag selected files from a folder, or entire folders full of files, to group some or all of your data as desired. Data will be arranged in blocks by measurement type, sorted into rows by subject ID. The output file (combined.csv) will be placed into the same folder as the original data files.
Wheel2CSV: This app is used to organize data from the Lafayette running wheels. Data as produced by the system is arranged into blocks by timestamp, so that each type of data only appears on every 5th line. This script rearranges the data into four blocks, sorted across by subject ID and down by time, with the date and timestamp as the first two columns. This script will should make it significantly easier for you to select the datapoints of interest.
Sorting data by treatment groups:
The Echo2CSV, OxyMaxer, Spa2CSV, Spa Grouper, and Wheel2CSV apps will sort data by treatment group if you include a file listing treatments. You do not have to do this if you do not want to do so, but it does save time during data analysis.
To work, the file must be named ‘key.txt’ and must be present in the same folder as the data files being processed. The key.txt file should contain a list of subject and group pairs, separated by a comma (like ‘SubjectID,Group’). Each subject:group pair should be on a separate line. You can create this file in any text editor (Notepad on Windows and TextEdit on Mac are the default programs for text files). The group can be any name you wish (so long as you do not use special characters [e.g. ', ", $, (, :, @, #, etc.] in the group name), but the subject ID must exactly match what was entered into the computer when performing the behavioral test or EchoMRI scan.
If the key.txt file is present during data analysis, group designations will be added to the output files. Spa2CSV will simply replace the subject ID with ‘Group@SubjectID’, and output files will use the naming pattern ‘YYYY-MM-DD_Group@SubjectID.csv’. For other apps, a new column (Echo2CSV, OxyMaxer, Spa Grouper) or row (Wheel2CSV) for group ID will be added to the output file(s). The output file(s) will be sorted alphanumerically by group, and then alphanumerically by subject within groups. Note that sorting may be case-sensitive. Non-alphanumeric characters will be sorted by ASCII code. Any animals for which a group assignment is not listed will be placed in a ‘null’ group in the output file(s).
The Spa2CSV app has been updated to incorporate group IDs in output files. Any key.txt file found is still copied into the output directory so it will be available for use with the Spa Grouper app, but this file will only be needed by Spa Grouper if Spa2CSV has not already added group assignments. This change was made to allow for repeated measure treatments. Prior to this change, Spa2CSV and Spa Grouper made analyzing repeated measures data somewhat difficult. All that is needed to speed this process is to include a different key.txt file for each set of Zone or Summary files analyzed by Spa2CSV (e.g. listing treatments for each subject on that specific test), then combine all sets of files into one folder and send to Spa Grouper.
A note on versions:
There are two versions of these apps: 32-bit versions for Windows XP / Mac OS 10.5 and below, and 64-bit versions for Windows 7 / Intel Macs running 10.6 or higher. On Windows, 32-bit versions should be named ‘AppName_32’, while 64-bit versions are ‘AppName’. If in doubt, try the 32-bit versions. 64-bit versions will not run if your system cannot support them, but 32-bit should run anywhere.
The Perl scripts underlying these apps were written by Joshua Nixon. The Spa2CSV app was based on a MedAssociates data processing script written by Claudio Perez-Leighton; portions of this code were re-used in other apps. Program icons were created by Joshua Nixon. Apps were packaged for Windows using Cava Packager, and for Mac OS X using Platypus. Installer script for OS X was written by Joshua Nixon and packaged using Platypus. Installer for Windows was built using NSIS.