Author: Jacolien van Rij & Dorothée Hoppe

# EDL

#### Jacolien van Rij & Dorothée Hoppe

#### 10/8/2018

Here we provide astep-by-step overview of the core functions of the package using an example data set.

## Loading library

    library(edl)

## Example data

Load the example data from the package `edl`:

    data(dat)
    head(dat)

    ##     Shape Color Category Frequency2 Frequency1
    ## 1     cat brown   animal         88          6
    ## 2  rabbit brown   animal         88          1
    ## 3  flower brown    plant          6         16
    ## 4    tree brown    plant          6         56
    ## 5     car brown  vehicle          1         88
    ## 6 bicycle brown  vehicle         56         32

## step 1: Prepare data

This data set lists all unique learning events (i.e., the types) and their associated frequencies. However, for a data set to function as input for the learning functions, the dataframe must include the columns `Cues` and `Outcomes`, and optionally `Frequency`. Note that if `Frequency` is not included, the frequency of each learning event is assumed to be 1.

First, we construct the columns `Cues` and `Outcomes` for this example simulation. Here we will simulate how two features `Color` and `Shape` may function as cues for their category `Category`. We will add a background cue “BG” to represent the learner. The different cues and outcomes are separated using an underscore (i.e., "_"). It is possible to another symbol, but then we will need to indicate this in the various functions (i.e., `RWlearning`) with the argument `split`.

    dat$Cues      <- paste("BG", dat$Shape, dat$Color, sep="_")
    dat$Outcomes  <- paste(dat$Category)
    dat$Frequency <- dat$Frequency2
    # remove remaining columns to simplify this example:
    dat <- dat[, c("Cues", "Outcomes", "Frequency")]
    # add ID for learning events:
    dat$ID <- 1:nrow(dat)
    head(dat)

    ##               Cues Outcomes Frequency ID
    ## 1     BG_cat_brown   animal        88  1
    ## 2  BG_rabbit_brown   animal        88  2
    ## 3  BG_flower_brown    plant         6  3
    ## 4    BG_tree_brown    plant         6  4
    ## 5     BG_car_brown  vehicle         1  5
    ## 6 BG_bicycle_brown  vehicle        56  6

Now the data `dat` defines 36 unique learning events with their associated frequencies.

    table(dat$Outcomes)

    ##
    ##  animal   plant vehicle
    ##      12      12      12

## step 2: Create training data

The training data lists all learning events (i.e., the tokens) in their order of presentation, one learning event per row. If no column with frequencies is specificied or all frequencies are 1, the training data mirrors the data set of learning events. The training data also determines the order in which the learning events are presented to the learning network.

    # by default 1 run, with tokens randomized:
    train <- createTrainingData(dat)
    head(train)

    ##              Cues Outcomes Frequency ID    Item Trial
    ## 1   BG_car_yellow  vehicle         1 23 item_23     1
    ## 2     BG_car_blue  vehicle         1 35 item_35     2
    ## 3  BG_flower_blue    plant         1 33 item_33     3
    ## 4 BG_bicycle_gray  vehicle         1 12 item_12     4
    ## 5 BG_bicycle_gray  vehicle         1 12 item_12     5
    ## 6 BG_bicycle_gray  vehicle         1 12 item_12     6

    # Frequency is always 1:
    unique(train$Frequency)

    ## [1] 1

    # total counts per outcome match original frequencies:
    table(train$Outcomes)

    ##
    ##  animal   plant vehicle
    ##     398     398     398

    table(train$ID)

    ##
    ##  1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26
    ## 88 88  6  6  1 56 56 56  1  1 88 88 32 32 56 56 56  1 16 16 88 88  6  6  6  6
    ## 27 28 29 30 31 32 33 34 35 36
    ## 32 32 32 32  1  1 16 16 16 16

Note that the function `createTrainingData` could also be used to train the network on multiple (blocked or randomized) runs. We refer to the examples in the function helpfile.

## step 3: Learning

The function `RWlearning` trains the error-driven learning network. The output `wm` is a list with weight matrices, a weight matrix for each learning event (a learning event is basically a row in the data frame `dat`). The last weight matrix shows the connections after processing all data.

    wm <- RWlearning(train)

Inspection:

    length(wm)

    ## [1] 1194

    # ... which is the same as the number of rows in the training data:
    nrow(train)

    ## [1] 1194

## step 4: Inspection

We can now inspect the changes in the weights for after each learning event. The last weight matrix shows the connections after processing all data:

    # after the first learning event:
    wm[[1]]

    ##        vehicle
    ## BG        0.01
    ## car       0.01
    ## yellow    0.01

    # the final state of the network:
    wm[[length(wm)]]

    ##             vehicle       plant      animal
    ## BG       0.25183879  0.26023902  0.23234787
    ## car      0.57869696 -0.22138222 -0.23654574
    ## yellow  -0.05222424  0.19551856 -0.01538718
    ## blue     0.05640855  0.06106817 -0.02745376
    ## flower  -0.20567025  0.53628639 -0.19586861
    ## bicycle  0.58427796 -0.18397074 -0.27369648
    ## gray     0.13856554 -0.08443580  0.08633106
    ## tree    -0.20180690  0.52949356 -0.19595374
    ## white    0.01483705  0.09271535  0.02837224
    ## brown    0.02539575 -0.07375082  0.18870029
    ## rabbit  -0.25318191 -0.19939852  0.56773558
    ## cat     -0.25047708 -0.20078945  0.56667685
    ## red      0.06885614  0.06912355 -0.02821478

The function `getWM` retrieves the weight matrix after a specific event, and adds zero-weight connections for the not yet encountered cues and outcomes.

    # after the first learning event:
    getWM(wm,1)

    ##         vehicle plant animal
    ## BG         0.01     0      0
    ## car        0.01     0      0
    ## yellow     0.01     0      0
    ## blue       0.00     0      0
    ## flower     0.00     0      0
    ## bicycle    0.00     0      0
    ## gray       0.00     0      0
    ## tree       0.00     0      0
    ## white      0.00     0      0
    ## brown      0.00     0      0
    ## rabbit     0.00     0      0
    ## cat        0.00     0      0
    ## red        0.00     0      0

We can use the functions `sapply` and `getWM` to add zero-weight connections to all states of the network:

    wm2 <- sapply(1:length(wm), function(x){getWM(wm,x)}, simplify = FALSE)
    # inspect the list of states:
    length(wm2)

    ## [1] 1194

    wm2[[1]]

    ##         vehicle plant animal
    ## BG         0.01     0      0
    ## car        0.01     0      0
    ## yellow     0.01     0      0
    ## blue       0.00     0      0
    ## flower     0.00     0      0
    ## bicycle    0.00     0      0
    ## gray       0.00     0      0
    ## tree       0.00     0      0
    ## white      0.00     0      0
    ## brown      0.00     0      0
    ## rabbit     0.00     0      0
    ## cat        0.00     0      0
    ## red        0.00     0      0

The functions `getWeightsByCue` and `getWeightsByOutcome` could be used to extract the weights per cue or per outcome.

    # weights for outcome "plant"
    weights <- getWeightsByOutcome(wm, outcome="plant")
    head(weights)

    ##           BG     bicycle blue brown car cat flower        gray rabbit red tree
    ## 1 0.00000000  0.00000000 0.00     0   0   0   0.00  0.00000000      0   0    0
    ## 2 0.00000000  0.00000000 0.00     0   0   0   0.00  0.00000000      0   0    0
    ## 3 0.01000000  0.00000000 0.01     0   0   0   0.01  0.00000000      0   0    0
    ## 4 0.00990000 -0.00010000 0.01     0   0   0   0.01 -0.00010000      0   0    0
    ## 5 0.00980300 -0.00019700 0.01     0   0   0   0.01 -0.00019700      0   0    0
    ## 6 0.00970891 -0.00029109 0.01     0   0   0   0.01 -0.00029109      0   0    0
    ##   white yellow
    ## 1     0      0
    ## 2     0      0
    ## 3     0      0
    ## 4     0      0
    ## 5     0      0
    ## 6     0      0

    tail(weights)

    ##             BG    bicycle       blue       brown        car        cat
    ## 1189 0.2613355 -0.1839707 0.06106817 -0.07403495 -0.2213822 -0.1983300
    ## 1190 0.2587250 -0.1839707 0.06106817 -0.07403495 -0.2213822 -0.2009405
    ## 1191 0.2598715 -0.1839707 0.06106817 -0.07403495 -0.2213822 -0.2009405
    ## 1192 0.2600225 -0.1839707 0.06106817 -0.07388391 -0.2213822 -0.2007894
    ## 1193 0.2601059 -0.1839707 0.06106817 -0.07388391 -0.2213822 -0.2007894
    ## 1194 0.2602390 -0.1839707 0.06106817 -0.07375082 -0.2213822 -0.2007894
    ##         flower       gray     rabbit        red      tree      white    yellow
    ## 1189 0.5350565 -0.0844358 -0.1995316 0.06912355 0.5294936 0.09156885 0.1980457
    ## 1190 0.5350565 -0.0844358 -0.1995316 0.06912355 0.5294936 0.09156885 0.1954352
    ## 1191 0.5362030 -0.0844358 -0.1995316 0.06912355 0.5294936 0.09271535 0.1954352
    ## 1192 0.5362030 -0.0844358 -0.1995316 0.06912355 0.5294936 0.09271535 0.1954352
    ## 1193 0.5362864 -0.0844358 -0.1995316 0.06912355 0.5294936 0.09271535 0.1955186
    ## 1194 0.5362864 -0.0844358 -0.1993985 0.06912355 0.5294936 0.09271535 0.1955186

    # weights for cue "red"
    weights <- getWeightsByCue(wm, cue="red")
    head(weights)

    ##   animal plant vehicle
    ## 1      0     0       0
    ## 2      0     0       0
    ## 3      0     0       0
    ## 4      0     0       0
    ## 5      0     0       0
    ## 6      0     0       0

    tail(weights)

    ##           animal      plant    vehicle
    ## 1189 -0.02821478 0.06912355 0.06885614
    ## 1190 -0.02821478 0.06912355 0.06885614
    ## 1191 -0.02821478 0.06912355 0.06885614
    ## 1192 -0.02821478 0.06912355 0.06885614
    ## 1193 -0.02821478 0.06912355 0.06885614
    ## 1194 -0.02821478 0.06912355 0.06885614

In addition, there are various functions to extract the activations for each learning event. The function `getActivations` is a wrapper that captures most common calculations, but other, more specialistic functions are described below.

    act <- getActivations(wm, data=train)
    head(act)

    ##              Cues Outcomes Activation
    ## 1   BG_car_yellow  vehicle 0.03000000
    ## 2     BG_car_blue  vehicle 0.04940000
    ## 3  BG_flower_blue    plant 0.03000000
    ## 4 BG_bicycle_gray  vehicle 0.04891888
    ## 5 BG_bicycle_gray  vehicle 0.07745131
    ## 6 BG_bicycle_gray  vehicle 0.10512777

Alternatively,the function `getActivations` can output all possible outcomes per learning event.

    act <- getActivations(wm, data=train, select.outcomes = TRUE)
    head(act)

    ##              Cues Outcomes animal      plant    vehicle
    ## 1   BG_car_yellow  vehicle      0 0.00000000 0.03000000
    ## 2     BG_car_blue  vehicle      0 0.00000000 0.04940000
    ## 3  BG_flower_blue    plant      0 0.03000000 0.02871200
    ## 4 BG_bicycle_gray  vehicle      0 0.00970000 0.04891888
    ## 5 BG_bicycle_gray  vehicle      0 0.00940900 0.07745131
    ## 6 BG_bicycle_gray  vehicle      0 0.00912673 0.10512777

We may want to add the activation of observed outcome in separate column:

    act$Activation <- apply(act, 1, function(x){
      out <- x['Outcomes']
      return(as.numeric(x[out]))
    })
    head(act)

    ##              Cues Outcomes animal      plant    vehicle Activation
    ## 1   BG_car_yellow  vehicle      0 0.00000000 0.03000000 0.03000000
    ## 2     BG_car_blue  vehicle      0 0.00000000 0.04940000 0.04940000
    ## 3  BG_flower_blue    plant      0 0.03000000 0.02871200 0.03000000
    ## 4 BG_bicycle_gray  vehicle      0 0.00970000 0.04891888 0.04891888
    ## 5 BG_bicycle_gray  vehicle      0 0.00940900 0.07745131 0.07745131
    ## 6 BG_bicycle_gray  vehicle      0 0.00912673 0.10512777 0.10512780

Note that `getActivations` only works for a single outcome in each learning event. With multiple outcomes, please use one of the other activation functions.

## step 5: Visualization

Visualizing the change in weights between cues and outcomes is facilitated by two functions: `plotCueWeights` and `plotOutcomeWeights`. The first function retrieves the weights between a specific cue and all outcomes (or a selection of outcomes) for each learning event. The second function retrieves the weights between a specific outcome and all cues (or a selection of cues) for each learning.

    oldpar <- par(mfrow=c(1,2), cex=1.1)

    # plot left:
    plotCueWeights(wm, cue="brown")

    # plot right:
    plotOutcomeWeights(wm, outcome="animal")

    par(oldpar)

Both plot functions have a range of arguments that can be used to change the layout, as illustrated for the same two plots below:

    oldpar <- par(mfrow=c(1,2), cex=1.1)

    # plot left:
    # 1. get outcome values:
    out <- getValues(train$Outcomes, unique=TRUE)
    out <- out[out != "animal"]
    # 2. plot all outcomes, except 'plural':
    lab <- plotCueWeights(wm, cue="brown", select.outcomes = out,
                          col=1, add.labels=FALSE, xlab='', ylim=range(getWM(wm)))
    # 3. add plural:
    lab2 <- plotCueWeights(wm, cue="brown", select.outcomes = "animal", col=2, lwd=2, adj=0, add=TRUE, font=2)
    # 4. add legend:
    legend_margin('bottom', ncol=4,
                  legend=c(lab2$labels, lab$labels),
                  col=c(lab2$col, lab$col), lty=c(lab2$lty, lab$lty),
                  lwd=c(lab2$lwd, lab$lwd), bty='n', cex=.85)

    # plot right, different layout variant:
    out <- getValues(dat$Cues, unique=TRUE)
    out <- out[out != "animal"]
    lab <- plotOutcomeWeights(wm, outcome="animal", select.cues = out,
                              col=alpha(1, f=.25), lty=1, pos=4, ylim=c(-.02,.2), font=2, ylim=range(getWM(wm)))
    lab2 <- plotOutcomeWeights(wm, outcome="animal", select.cues = "brown", col='red', lwd=2, pos=4, add=TRUE, font=2)

    par(oldpar)

Both plotfunctions are a wrapper around the functions `getWeightsByCue` and `getWeightsByOutcome`. These values could be used to extract the weights per cue or per outcome.

    weights <- getWeightsByCue(wm, cue="brown")
    head(weights)

    ##   animal plant vehicle
    ## 1      0     0       0
    ## 2      0     0       0
    ## 3      0     0       0
    ## 4      0     0       0
    ## 5      0     0       0
    ## 6      0     0       0

Similarly, we can visualize the change in activations using the function `plotActivations`, which is a wrapper around the function `getActivations`.

    oldpar <- par(mfrow=c(1,2), cex=1.1)

    # an observed cueset:
    plotActivations(wm, cueset="BG_cat_brown")
    # an un-observed cueset:
    plotActivations(wm, cueset="BG_cat_yellow")

    par(oldpar)

## Extra: Continue training

Another possibility worth mentioning is the possibility to continue training from an existing weight matrix.

    # create a second data set with different frequencies:
    data(dat)
    head(dat)

    ##     Shape Color Category Frequency2 Frequency1
    ## 1     cat brown   animal         88          6
    ## 2  rabbit brown   animal         88          1
    ## 3  flower brown    plant          6         16
    ## 4    tree brown    plant          6         56
    ## 5     car brown  vehicle          1         88
    ## 6 bicycle brown  vehicle         56         32

We used the column `Frequency2`, and now we continue training with column `Frequency1`. Note that in the new data (rows 1 and 2, column `Frequency1`), there are much fewer brown animals than in the earlier training data (column `Frequency2`).

    dat$Cues      <- paste("BG", dat$Shape, dat$Color, sep="_")
    dat$Outcomes  <- paste(dat$Category)
    dat$Frequency <- dat$Frequency1
    # remove remaining columns to simplify this example:
    dat <- dat[, c("Cues", "Outcomes", "Frequency")]
    # add ID for learning events:
    dat$ID <- 1:nrow(dat)
    head(dat)

    ##               Cues Outcomes Frequency ID
    ## 1     BG_cat_brown   animal         6  1
    ## 2  BG_rabbit_brown   animal         1  2
    ## 3  BG_flower_brown    plant        16  3
    ## 4    BG_tree_brown    plant        56  4
    ## 5     BG_car_brown  vehicle        88  5
    ## 6 BG_bicycle_brown  vehicle        32  6

    # create training data:
    train2 <- createTrainingData(dat)

After creating the training data (one event per row), we continue training. We will use the end state of the previous training as starting point for the new training. Two methods are illustrated in the code block below:

    # continue learning from last weight matrix:
    wm2 <- RWlearning(train2, wm=getWM(wm), progress = FALSE)
    # number of learned event matches rows in dat2:
    nrow(train2)

    ## [1] 1194

    length(wm2)

    ## [1] 1194

    # Alternatively, add the learning events to the existing output list wm1:
    wm3 <- RWlearning(train2, wm=wm, progress = FALSE)
    # number of learned event are now added to wm1:
    length(wm3)

    ## [1] 2388

Now we can visualize how changing the input frequencies changes the connection weights. The red line in the plot visualizes the change in how predictable the color “brown” is for an animal.

    out <- getValues(dat$Cues, unique=TRUE)
    out <- out[out != "animal"]
    lab <- plotOutcomeWeights(wm3, outcome="animal",
                              select.cues = out,
                              col=alpha(1, f=.25), lty=1, pos=4,
                              ylim=c(-.02,.2), font=2, ylim=range(getWM(wm3)),
                              xmark=TRUE, ymark=TRUE, las=1)
    lab2 <- plotOutcomeWeights(wm3, outcome="animal",
                               select.cues = "brown", col='red',
                               lwd=2, pos=4, add=TRUE, font=2)
    abline(v=length(wm), lty=3)

## step 6: Activations

The activation of outcomes reflect the learner’s expectation that this outcome will occur, based on the present cues.

The `edl` package includes a series different functions to calculate the activativations for outcomes:

- `activationsMatrix`: Calculate activation based on a static state of the network. The basic activation function that is called by the other activation functions.

    # select weight matrix:
    mat <- getWM(wm)
    # for a cueset:
    activationsMatrix(mat,cues="BG_cat_brown")

    ##     animal       plant    vehicle
    ## 1 0.987725 -0.01430125 0.02675746

    # for a specific outcome:
    activationsMatrix(mat,cues="BG_cat_brown", select.outcomes = "animal")

    ##     animal
    ## 1 0.987725

    # for a group of cuesets (all connection weights will be added):
    activationsMatrix(mat,cues=c("BG_cat_brown", "BG_cat_blue"))

    ##     animal     plant    vehicle
    ## 1 1.759296 0.1062165 0.08452773

- `activationsEvents`: Calculate activations for each learning event in the training data.

    # new dummy data:
    dat <- data.frame(Cues = c("noise", "noise", "light"),
                      Outcomes = c("food", "other", "food_other"),
                      Frequency = c(5, 10, 15) )
    dat$Cues <- paste("BG", dat$Cues, sep="_")
    train <- createTrainingData(dat)
    wm <- RWlearning(train, progress = FALSE)

    # list with activations for observed outcomes:
    act <- activationsEvents(wm, data=train)

    ## Warning in activationsEvents(wm, data = train): Function will return list of
    ## activations, because in some events multiple outcomes occurred.

    head(act)

    ## [[1]]
    ##   food other
    ## 1 0.02  0.02
    ##
    ## [[2]]
    ##     food  other
    ## 1 0.0396 0.0396
    ##
    ## [[3]]
    ##       food
    ## 1 0.039404
    ##
    ## [[4]]
    ##        other
    ## 1 0.03901592
    ##
    ## [[5]]
    ##       other
    ## 1 0.0582356
    ##
    ## [[6]]
    ##         food      other
    ## 1 0.06764936 0.07764144

    # calculate max activation:
    maxact <- lapply(act, function(x){ return(max(x, na.rm=TRUE)) })
    unlist(maxact)

    ##  [1] 0.02000000 0.03960000 0.03940400 0.03901592 0.05823560 0.07764144
    ##  [7] 0.09608862 0.09533359 0.11342692 0.13115838 0.14076765 0.15795230
    ## [13] 0.17479326 0.19129739 0.20747144 0.18981239 0.20601614 0.11159833
    ## [19] 0.23734477 0.25259787 0.14528315 0.16237749 0.22452532 0.17594734
    ## [25] 0.26888374 0.28350606 0.29783594 0.25713186 0.31930790 0.27879615

    # Using argument 'fun':
    act <- activationsEvents(wm, data=train, fun="max")
    head(act)

    ## [1] 0.02000000 0.03960000 0.03940400 0.03901592 0.05823560 0.07764144

- `activationsCueSet`: Calculate activations for one or multiple cuesets.

    # list with activations for observed outcomes:
    act <- activationsCueSet(wm, cueset=c("BG_noise", "BG_light", "BG_somethingelse"))
    names(act)

    ## [1] "BG_noise"         "BG_light"         "BG_somethingelse"

    head(act[[1]])

    ##         food      other
    ## 1 0.01000000 0.01000000
    ## 2 0.01980000 0.01980000
    ## 3 0.03940400 0.01940400
    ## 4 0.03861592 0.03901592
    ## 5 0.03784360 0.05823560
    ## 6 0.04735738 0.06764742

    # also activations for non-trained connections:
    head(act[[3]])

    ##         food      other
    ## 1 0.01000000 0.01000000
    ## 2 0.01980000 0.01980000
    ## 3 0.02960200 0.01960200
    ## 4 0.02920796 0.02940796
    ## 5 0.02882180 0.03901780
    ## 6 0.03833558 0.04842962

- `activationsOutcomes`: Calculate activations per learning event for all outcomes in the data.

    # list with activations for observed outcomes:
    act <- activationsOutcomes(wm, data=train)
    head(act)

    ##         food      other
    ## 1 0.02000000 0.02000000
    ## 2 0.03960000 0.03960000
    ## 3 0.03940400 0.01940400
    ## 4 0.03861592 0.03901592
    ## 5 0.03784360 0.05823560
    ## 6 0.06764936 0.07764144