The files contained in this folder are the basis for the human-automation relationship taxonomy (HART), which has been developed to predict human-automation interaction (HAI). HART attempts to model the human-automation relationship using a multi-dimensional description of the automation itself, the task for which the automation is used, the context in which the automation is used, and the operator. Therefore, HART assumes that HAI is a function of automation, the task being performed, the operator characteristics, and the context in which the operators are performing the automation related task. HART models HAI as a set of six consequence: mental workload, task management effort, automation mode awareness, situational awareness, amount of automation crosschecking performed, and likelihood of automation-related error. This work demonstrated that HART could be linked to HAI consequences using a backpropagation neural network approach. For a more thorough discussion of this approach, please see the following publications: K.B. Sullivan, K.M. Feigh, R. Mappus IV, F.T. Durso, U. Fischer, V. Pop, K.L. Mosier, D.G. Morrow, Using neural networks to assess flight deck human–automation interaction, Reliability Engineering & System Safety, Volume 114, December 2012, Pages 26-35, http://dx.doi.org/10.1016/j.ress.2012.12.005; and K.L. Mosier, U. Fischer, D. Morrow, K.M. Feigh, F.T. Durso, K.B. Sullivan, and V. Pop. Automation, Task, and Context Features: Impacts on Pilots’ Judgments of Human–Automation Interaction. Journal of Cognitive Engineering and Decision Making, May 2013, http://dx.doi.org/10.1177/1555343413487178. The "ARFF Files and Results" parent folder contains 4 sub-folders, which represent the four different sets of conditions under which the neural network were tested - "2x2 LAS · LAS Runs", "2x2 LAS · LAX Runs", "8x8 LAS · LAS Runs", and "8x8 LAS · LAX Runs". “2x2”and “8x8” refer to the size of the confusion matrix used in the analyses and whether the data are training or testing data (LAS · LAS refers to training data, and LAS · LAX refers to testing data), so two simulations were run with a 2x2 confusion matrix and two were run with an 8x8 confusion matrix (for those files whose names include “7x7,” the confusion matrix was only 7x7) and two were testing data and two were training data. Each of these four sub-folders folders contain four additional sub-sub-folders, which represent the four different input models tested, "Triage", "Triage+Changing", "Triage+Context", and Triage+Context+Changing". The one exception is the “8x8 LAS · LAX Runs” set of conditions, which was only run for Triage and Triage+Context+Changing. The “Triage” input model included only the triage parameters, which are the summative, high-level parameters of the automation features of HAI. The “Triage+Context” input model includes the triage questions along with the context parameters, which varied between scenarios. The “Triage+Changing” input model included the input parameter values of the triage parameters in conjunction with the automation input parameters that were rated differently throughout the scenario variants. The additional parameters are called the changing parameters, and they were selected because only parameters that varied between scenario variants were able to teach the neural network about relationships between input and output parameters. The final model, Triage/Context/Changing, included all three sets of input parameters. The file tree for the zipped folder is below: HART_dataset | runlaslax | runnn | +---ARFF Files and Results | +---2x2 LAS?LAS Runs | | +---Triage | | | 2x2_triage_autorelerr_results.txt | | | 2x2_triage_awareauto_results.txt | | | 2x2_triage_crosschk_results.txt | | | 2x2_triage_sa_results.txt | | | 2x2_triage_taskmang_results.txt | | | 2x2_triage_workload_results.txt | | | las_2x2_triage_autorelerr.txt | | | las_2x2_triage_awareauto.txt | | | las_2x2_triage_crosschk.txt | | | las_2x2_triage_sa.txt | | | las_2x2_triage_taskmang.txt | | | las_2x2_triage_workload.txt | | | runlaslax | | | runnn | | | | | +---Triage+Changing | | | 2x2_triagechanging_autorelerr.txt | | | 2x2_triagechanging_awareauto.txt | | | 2x2_triagechanging_crosschk.txt | | | 2x2_triagechanging_sa.txt | | | 2x2_triagechanging_taskmang.txt | | | 2x2_triagechanging_workload.txt | | | las_2x2_triagechanging_autorelerr.txt | | | las_2x2_triagechanging_awareauto.txt | | | las_2x2_triagechanging_crosschk.txt | | | las_2x2_triagechanging_sa.txt | | | las_2x2_triagechanging_taskmang.txt | | | las_2x2_triagechanging_workload.txt | | | runlaslax | | | runnn | | | | | +---Triage+Context | | | 2x2_triagecontext_autorelerr.txt | | | 2x2_triagecontext_awareauto.txt | | | 2x2_triagecontext_crosschk.txt | | | 2x2_triagecontext_sa.txt | | | 2x2_triagecontext_taskmang.txt | | | 2x2_triagecontext_workload.txt | | | las_2x2_triagecontext_autorelerr.txt | | | las_2x2_triagecontext_awareauto.txt | | | las_2x2_triagecontext_crosschk.txt | | | las_2x2_triagecontext_sa.txt | | | las_2x2_triagecontext_taskmang.txt | | | las_2x2_triagecontext_workload.txt | | | runlaslax | | | runnn | | | | | \---Triage+Context+Changing | | 2x2_all_autorelerr.txt | | 2x2_all_awareauto.txt | | 2x2_all_crosschk.txt | | 2x2_all_sa.txt | | 2x2_all_taskmang.txt | | 2x2_all_workload.txt | | las_2x2_all_autorelerr.txt | | las_2x2_all_awareauto.txt | | las_2x2_all_crosschk.txt | | las_2x2_all_sa.txt | | las_2x2_all_taskmang.txt | | las_2x2_all_workload.txt | | runlaslax | | runnn | | | +---2x2 LAS?LAX Runs | | +---Triage | | | laslax_2x2_triage_autorelerr.txt | | | laslax_2x2_triage_awareauto.txt | | | laslax_2x2_triage_crosschk.txt | | | laslax_2x2_triage_sa.txt | | | laslax_2x2_triage_taskmang.txt | | | laslax_2x2_triage_workload.txt | | | las_2x2_triage_autorelerr.txt | | | las_2x2_triage_awareauto.txt | | | las_2x2_triage_crosschk.txt | | | las_2x2_triage_sa.txt | | | las_2x2_triage_taskmang.txt | | | las_2x2_triage_workload.txt | | | lax_2x2_triage_autorelerr.txt | | | lax_2x2_triage_awareauto.txt | | | lax_2x2_triage_crosschk.txt | | | lax_2x2_triage_sa.txt | | | lax_2x2_triage_taskmang.txt | | | lax_2x2_triage_workload.txt | | | runlaslax | | | runnn | | | | | +---Triage+Changing | | | laslax_2x2_triagechanging_autorelerr.txt | | | laslax_2x2_triagechanging_awareauto.txt | | | laslax_2x2_triagechanging_crosschk.txt | | | laslax_2x2_triagechanging_sa.txt | | | laslax_2x2_triagechanging_taskmang.txt | | | laslax_2x2_triagechanging_workload.txt | | | las_2x2_triagechanging_autorelerr.txt | | | las_2x2_triagechanging_awareauto.txt | | | las_2x2_triagechanging_crosschk.txt | | | las_2x2_triagechanging_sa.txt | | | las_2x2_triagechanging_taskmang.txt | | | las_2x2_triagechanging_workload.txt | | | lax_2x2_triagechanging_autorelerr.txt | | | lax_2x2_triagechanging_awareauto.txt | | | lax_2x2_triagechanging_crosschk.txt | | | lax_2x2_triagechanging_sa.txt | | | lax_2x2_triagechanging_taskmang.txt | | | lax_2x2_triagechanging_workload.txt | | | runlaslax | | | runnn | | | | | +---Triage+Context | | | laslax_2x2_triagecontext_autorelerr.txt | | | laslax_2x2_triagecontext_awareauto.txt | | | laslax_2x2_triagecontext_crosschk.txt | | | laslax_2x2_triagecontext_sa.txt | | | laslax_2x2_triagecontext_taskmang.txt | | | laslax_2x2_triagecontext_workload.txt | | | las_2x2_triagecontext_autorelerr.txt | | | las_2x2_triagecontext_awareauto.txt | | | las_2x2_triagecontext_crosschk.txt | | | las_2x2_triagecontext_sa.txt | | | las_2x2_triagecontext_taskmang.txt | | | las_2x2_triagecontext_workload.txt | | | lax_2x2_triagecontext_autorelerr.txt | | | lax_2x2_triagecontext_awareauto.txt | | | lax_2x2_triagecontext_crosschk.txt | | | lax_2x2_triagecontext_sa.txt | | | lax_2x2_triagecontext_taskmang.txt | | | lax_2x2_triagecontext_workload.txt | | | runlaslax | | | runnn | | | | | \---Triage+Context+Changing | | laslax_2x2_all_autorelerr.txt | | laslax_2x2_all_awareauto.txt | | laslax_2x2_all_crosschk.txt | | laslax_2x2_all_sa.txt | | laslax_2x2_all_taskmang.txt | | laslax_2x2_all_workload.txt | | las_2x2_all_autorelerr.txt | | las_2x2_all_awareauto.txt | | las_2x2_all_crosschk.txt | | las_2x2_all_sa.txt | | las_2x2_all_taskmang.txt | | las_2x2_all_workload.txt | | lax_2x2_all_autorelerr.txt | | lax_2x2_all_awareauto.txt | | lax_2x2_all_crosschk.txt | | lax_2x2_all_sa.txt | | lax_2x2_all_taskmang.txt | | lax_2x2_all_workload.txt | | runlaslax | | runnn | | | +---8x8 LAS?LAS Runs | | +---Triage | | | 7x7_triage_autorelerr_results.txt | | | 7x7_triage_awareauto_results.txt | | | 7x7_triage_crosschk_results.txt | | | 7x7_triage_sa_results.txt | | | 7x7_triage_taskmang_results.txt | | | 7x7_triage_workload_results.txt | | | las_7x7_triage_autorelerr.txt | | | las_7x7_triage_awareauto.txt | | | las_7x7_triage_crosschk.txt | | | las_7x7_triage_sa.txt | | | las_7x7_triage_taskmang.txt | | | las_7x7_triage_workload.txt | | | runlaslax | | | runnn | | | | | +---Triage+Changing | | | las_8x8_triagechanging_autorelerr.txt | | | las_8x8_triagechanging_autorelerr_results.txt | | | las_8x8_triagechanging_awareauto.txt | | | las_8x8_triagechanging_awareauto_results.txt | | | las_8x8_triagechanging_crosschk.txt | | | las_8x8_triagechanging_crosschk_results.txt | | | las_8x8_triagechanging_sa.txt | | | las_8x8_triagechanging_sa_results.txt | | | las_8x8_triagechanging_taskmang.txt | | | las_8x8_triagechanging_taskmang_results.txt | | | las_8x8_triagechanging_workload.txt | | | las_8x8_triagechanging_workload_results.txt | | | runnn | | | | | +---Triage+Context | | | las_8x8_triagecontext_autorelerr.txt | | | las_8x8_triagecontext_autorelerror_results.txt | | | las_8x8_triagecontext_awareauto.txt | | | las_8x8_triagecontext_awareauto_results.txt | | | las_8x8_triagecontext_crosschk.txt | | | las_8x8_triagecontext_crosschk_results.txt | | | las_8x8_triagecontext_sa.txt | | | las_8x8_triagecontext_sa_results.txt | | | las_8x8_triagecontext_taskmang.txt | | | las_8x8_triagecontext_taskmang_results.txt | | | las_8x8_triagecontext_workload.txt | | | las_8x8_triagecontext_workload_results.txt | | | runnn | | | | | \---Triage+Context+Changing | | 7x7_changing_autorelerr_result.txt | | 7x7_changing_awareauto_results.txt | | 7x7_changing_crosschk_results.txt | | 7x7_changing_sa_results.txt | | 7x7_changing_taskmang_result.txt | | 7x7_changing_workload_results.txt | | las_7x7_changing_autorelerr.txt | | las_7x7_changing_awareauto.txt | | las_7x7_changing_crosschk.txt | | las_7x7_changing_sa.txt | | las_7x7_changing_taskmang.txt | | las_7x7_changing_workload.txt | | runlaslax | | runnn | | | \---8x8 LAS?LAX Runs | +---Triage | | laslax_7x7_triage_autorelerr.txt | | laslax_7x7_triage_awareauto.txt | | laslax_7x7_triage_crosschk.txt | | laslax_7x7_triage_sa.txt | | laslax_7x7_triage_taskmang.txt | | laslax_7x7_triage_workload.txt | | las_7x7_triage_autorelerr.txt | | las_7x7_triage_awareauto.txt | | las_7x7_triage_crosschk.txt | | las_7x7_triage_sa.txt | | las_7x7_triage_taskmang.txt | | las_7x7_triage_workload.txt | | lax_7x7_triage_autorelerr.txt | | lax_7x7_triage_awareauto.txt | | lax_7x7_triage_crosschk.txt | | lax_7x7_triage_sa.txt | | lax_7x7_triage_taskmang.txt | | lax_7x7_triage_workload.txt | | runlaslax | | runnn | | | \---Triage+Context+Changing | laslax_7x7_changing_autorelerr.txt | laslax_7x7_changing_awareauto.txt | laslax_7x7_changing_crosschk.txt | laslax_7x7_changing_sa.txt | laslax_7x7_changing_taskmang.txt | laslax_7x7_changing_workload.txt | las_7x7_changing_autorelerr.txt | las_7x7_changing_awareauto.txt | las_7x7_changing_crosschk.txt | las_7x7_changing_sa.txt | las_7x7_changing_taskmang.txt | las_7x7_changing_workload.txt | lax_7x7_changing_autorelerr.txt | lax_7x7_changing_awareauto.txt | lax_7x7_changing_crosschk.txt | lax_7x7_changing_sa.txt | lax_7x7_changing_taskmang.txt | lax_7x7_changing_workload.txt | runlaslax | runnn | \---MATLAB Files to Create ARFFs #filelist.org# attributeHeader.m filelist.org inputVector.m inputVectorMonolith.m LAXmonolith.xls makebigarffmonolithf.m monolith_wcontext.xls runmakebigarffmonolithf.m setupfiles.m tlaslax.xls In total, there are 246 files. Software used in this project: Matlab, Microsoft Excel, Weka The files contained in this dataset were last modified in August of 2013 in preparation for data archiving, but the data themselves were last modified in February 2012. Additional information regarding the use of ARFF (Attribute-Relation File Format) files is below: How to create the ARFF files 1. Open the latest version of MATLAB 2. Make sure that you are in the correct directory a. If using the DVDs—you need to be in the folder entitled “MATLAB files to create ARFFs” 3. Open file runmakebigarffs from the folder 4. Click run How to run the ARFF files using the Goddard computer cluster ***Note: These directions are assuming you are using a Mac computer or Linux box and have access to the computer servers at ISYE at Georgia Tech. For the purposes of these directions, all direct code entries will be written in italic font. Also, “username” should be substituted with your actual ISYE username and your password will be your ISYE password. You will need to first download the most current version of the Weka software, which can be found here: http://www.cs.waikato.ac.nz/ml/weka/. Once the software has been installed on your computer, you can follow the steps below to create and train neural networks based on the data that currently exists. 1. Open a terminal window 2. Compress all of the files you need a. tar zcvf filename.tar.gz directory-name-containing-all-the-arff-folders 3. Log onto the Georgia Institute of Technology computer cluster a. ssh username@castle.isye.gatech.edu b. type your password when prompted and press enter c. ssh wren d. type your password when prompted and press enter 4. Get all of the files to the right server a. scp .tar.gz @castle.isye.gatech.edu:~/weka b. Type in password and press enter c. As confirmation, you will see the files uploading 5. Unpack all of the tar files a. cd weka b. tar zxf .tar.gz 6. Open a screen for the test you wish to run. Feel free to use Goddard1 and Goddard2 interchangeably. a. Screen b. ssh goddard1 (or goddard2) c. type your password when prompted and press enter 7. Run the test (if 2x2, this will take an hour—if 8x8, it might take an afternoon) a. LAS/LAS (do this process once for each HAI consequence) i. cd weka ii. ./runnn iii. press enter iv. hold CTL and press “a” and “d” v. exit b. LAS/LAX (do this process once for each HAI consequence) i. cd weka ii. ./runlaslax iii. press enter iv. hold CTL and press “a” and “d” v. exit 8. Download results a. Using Windows explorer, create a folder where you want the files to be stored called Results (Documents>HART>Results) b. Open a terminal window c. cd Documents d. cd HART e. cd Results f. scp @castle.isye.gatech.edu:~/weka/*.txt . g. For confirmation, you should see the text files downloading 9. Read results a. Using Windows explorer, open one text file (i.e. 2x2_las_autorelerr_results.txt) b. Scroll down to the bottom (note that you are seeing the weights of all connections in the network as you scroll) c. The last confusion matrix will show you the success rate of the network, along with a percentage rate of success (i.e. 74%) i. Double check that this is indeed successful by examining the confusion matrix 1. If all non-zero entries are in one column, the neural network used majority classification, and the percentage rate of success is null and void 2. If there are non-zero entries in multiple columns, the rate of success is correct For preservation and reuse purposes, the ARFF files are stored in .txt files. Information about the ARFF format is included below: ARFF files have two distinct sections. The first section is the Header information, which is followed the Data information. The Header of the ARFF file contains the name of the relation, a list of the attributes (the columns in the data), and their types. The relation name is defined as the first line in the ARFF file. The format is: @relation , where  is a string. The string must be quoted if the name includes spaces. Attribute declarations take the form of an ordered sequence of @attribute statements. Each attribute in the data set has its own @attribute statement which uniquely defines the name of that attribute and its data type. The order the attributes are declared indicates the column position in the data section of the file. For example, if an attribute is the third one declared then Weka expects that all that attributes values will be found in the third comma delimited column. The format for the @attribute statement is: @attribute , where the  must start with an alphabetic character. If spaces are to be included in the name then the entire name must be quoted. The ARFF data section of the file contains the data declaration line and the actual instance lines. The @data declaration is a single line denoting the start of the data segment in the file. The format is: @data. Each instance is represented on a single line, with carriage returns denoting the end of the instance. Attribute values for each instance are delimited by commas. They must appear in the order that they were declared in the header section (i.e. the data corresponding to the nth @attribute declaration is always the nth field of the attribute). Missing values are represented by a single question mark, as in: @data 4.4,?,1.5,?,Iris-setosa Values of string and nominal attributes are case sensitive, and any that contain space or the comment-delimiter character % must be quoted, as in the following example: @relation LCCvsLCSH @attribute LCC string @attribute LCSH string @data AG5, 'Encyclopedias and dictionaries.;Twentieth century.' AS262, 'Science -- Soviet Union -- History.' AE5, 'Encyclopedias and dictionaries.' AS281, 'Astronomy, Assyro-Babylonian.;Moon -- Phases.' AS281, 'Astronomy, Assyro-Babylonian.;Moon -- Tables.' Dates must be specified in the data section using the string representation specified in the attribute declaration. For example: @RELATION Timestamps @ATTRIBUTE timestamp DATE "yyyy-MM-dd HH:mm:ss" @DATA "2001-04-03 12:12:12" "2001-05-03 12:59:55" Additional information on the full format specification can be found at http://weka.wikispaces.com/ARFF+%28stable+version%29.