Measures of Awareness and of Sequence Knowledge

	Lynne Reder's book Implicit Memory and Metacognition may be purchased from Amazon.Com

Measures of Awareness and of Sequence Knowledge

Luis Jimenez,	Castor Mendez	&	Axel Cleeremans
Facultad de Psicologia Universidad de Santiago 15706, Santiago SPAIN	Facultad de Psicologia Universidad de Santiago 15706, Santiago SPAIN		Department of Psychology Universite Libre de Bruxelles BELGIUM
jimenez@usc.es

PSYCHE, 2(33), July 1996
http://psyche.cs.monash.edu.au/v2/psyche-2-33-jimenez.html

KEYWORDS: Sequence learning, Implicit Learning, Awareness, Methods.

COMMENTARY ON: Jackson, G. M. & Jackson, S. R. (1996) Do Measures of Explicit Learning Actually Measure What is Being Learnt in the Serial Reaction Time Task? A Critique of Current Methods PSYCHE, 2(20).

ABSTRACT: Jackson and Jackson (1995) argue that most current tests used to assess awareness of sequential material are flawed because of their emphasis on accuracy. They propose to distinguish two forms of sequence knowledge: Serial knowledge, that is, knowledge about the specific sequence that stimuli follow, which involves information about the statistical relationship between many sequence elements, and statistical knowledge, or knowledge about the probability of different transitions between adjacent sequence elements. Further, they suggest a new method to analyze generation performance, which involves considering the correlation between subjects' responses and the distribution of transition probabilities, regardless of the accuracy of generation performance. In this comment, we first suggest that the distinction between serial and statistical knowledge is unwarranted except in one case which is not addressed by Jackson and Jackson. We propose instead that all sequence knowledge is essentially statistical in nature. Second, we suggest that using probabilistic instead of deterministic sequences is a better way to approach the assessment of explicit knowledge, and illustrate this contention with empirical and simulated examples based on previous and current research (Cleeremans, 1993; Cleeremans and McClelland, 1991; Jimenez, Mendez and Cleeremans, 1996).

1. Introduction

In a recent paper, Jackson and Jackson (1995) discuss the methods used in some implicit learning paradigms to demonstrate that the information acquired by participants in such experiments can accurately be characterized as unconscious. In accordance with most standard definitions of implicit learning, Jackson and Jackson characterize it as a process by which task relevant information is acquired automatically and without conscious awareness of what is being learnt. Demonstrating that a given learning situation has resulted in the acquisition of some implicit knowledge therefore involves observing a dissociation between some measure of performance (taken to reflect the knowledge acquired during the task) and some measure of awareness, which should be sensitive exclusively and exhaustively to the conscious knowledge that may have affected performance (see Shanks and St.John, 1994, for a complete discussion of these constraints).

Jackson and Jackson (1995) focus their discussion on the Serial Reaction Time (SRT) task, a paradigm initially developed by Nissen and Bullemer (1987) and which has received considerable interest since its inception (e.g., Cohen, Ivry and Keele, 1990; Curran and Keele,1993; Jackson and Jackson, 1992; Perruchet and Amorim, 1992; Reed and Johnson, 1994; Willingham, Nissen and Bullemer, 1989). In the SRT task, participants are exposed to a target stimulus that can appear at one of several locations (usually, four) on a computer display. Participants respond by pressing on the corresponding key, just as in a simple choice reaction task. Unknown to them, however, the series of locations that the stimulus appears at over the entire experiment follows a repeating deterministic sequence. Participants can be shown to acquire knowledge about the sequential structure of the material because their performance exhibits a significant decrease in reaction times in response to such sequential material, as compared to their performance with material that follows a random sequence. This improvement in performance has been widely taken as reflecting implicit learning because participants are often unable to report on the acquired knowledge in a variety of tests of explicit knowledge, which can range from free verbal reports to recognition of the presented sequence of locations (e.g., Willingham, Greenley and Bardone, 1993; Willingham et al., 1989, but see also Perruchet and Amorim, 1992, for a critical discussion).

The main goal of Jackson and Jackson (1995) is to call attention to the limitations of current methods for assessing explicit knowledge. They concur with a number of other authors (e.g. Perruchet and Amorim, 1992; Shanks and St. John, 1994) in pointing out that both free verbal reports and structured questionnaires are weak methods of assessing explicit knowledge. Indeed, verbal reports and questionnaires are relatively insensitive and contextually very different from the SRT task that they are compared with. Jackson and Jackson therefore focus on the analysis of several cued recall methods that are usually referred to as "generation tasks".

In the standard generation task, as first employed by Nissen and Bullemer (1987), participants are presented with each element of the sequence they had been exposed to during the SRT task, but instead of merely identifying the current stimulus, they are now asked to predict where the target stimulus will appear on the next trial. Further, participants are required to keep guessing until they produce a correct prediction; at which point the next stimulus is presented and the next prediction trial is initiated. Several authors (Jimenez, Mendez and Cleeremans, 1996; Perruchet and Amorim, 1992) have suggested that this multiple guessing procedure may be problematic because the different responses that participants are required to produce on each trial may interfere with their memory of the previous elements. Some authors have therefore argued that far better measures of explicit knowledge may be obtained by using different, but related tasks.

Perruchet and Amorim (1992) proposed to use a "free" generation task, in which participants are told to generate complete sequences of trials without any feedback. Perruchet and Amorim also proposed a recognition task (see also Willingham et al., 1993), in which participants are presented with a sequence of several elements and have to judge the likelihood that it may have appeared during the SRT task. Finally, several authors have used a "continuous" version of the generation task that differs from the standard version in that presentation of the next element is not dependent on whether a correct prediction had been produced (e.g., Cohen et al., 1990; Cleeremans and McClelland, 1991; Jackson and Jackson, 1992; Jimenez et al., 1996). Elsewhere, (see Jimenez et al., 1996) we have argued that the continuous generation task is a better measure of explicit knowledge than either free generation and recognition tasks because it is contextually most similar to the SRT task. Indeed, the only differences between continuous generation and the SRT task are (1) that the former eliminates speed pressure and, (2) that it requires participants to anticipate the location of next stimulus instead of identifying it. By contrast, both free generation and recognition place additional demands on participants and put them in different task contexts.

Jackson and Jackson (1995) do not directly address the issue of evaluating the relative merits of each of these measures as tools to assess explicit knowledge, but argue instead that they may all be flawed because of their emphasis on accuracy. To this effect, Jackson and Jackson introduce a distinction between serial knowledge and statistical knowledge, and argue that accuracy measures may selectively be influenced by serial knowledge, whereas responding to the SRT task can be influenced by both serial knowledge and statistical knowledge of the stimulus material.

Serial knowledge, according to the authors, is knowledge about the specific sequence that stimuli follow, thus involving information about the statistical relationship between many sequence elements. Statistical knowledge, on the other hand, denotes knowledge about the probability of different transitions, or relationship between two sequentially adjacent sequence elements. For instance, consider the sequence "ABCACB" and assume that participants are asked to produce this sequence explicitly in one of the standard tests of explicit knowledge used in this paradigm, such as the generation task. Performance is typically assessed, according to Jackson and Jackson, by measuring the accuracy with which participants are capable of reproducing the sequence. Thus for instance, predicting that 'C' will occur after the first 'A' token in the sequence above would be considered as erroneous because according to the sequence, 'B' is the successor to the first 'A'. Jackson and Jackson's main point is to suggest that responding with 'C', if incorrect from a serial-order point of view, is in fact correct from a statistical point of view, because 'A' is sometimes followed by 'B' and sometimes by 'C' when the transitions that may occur over the entire sequence are taken into account.

This statistical knowledge about the relative likelihood of each stimulus given its context is not accurately reflected in prediction accuracy measures such as the percentage of correct responding in any version of the generation task. Jackson and Jackson support their argument through new analyses of published data (e.g. Jackson and Jackson, 1992; Jackson, Jackson, Harrison, Henderson and Kennard, 1995). The new analyses indicate that these studies may have misclassified participants along the "awareness" dimension because of their exclusive reliance on measures of generation accuracy. For instance, some participants may have been classified as lacking explicit knowledge of the sequence, but further analysis of the correlation between their generation performance and the statistical structure of the material demonstrates the existence of a considerable amount of explicit learning.

If we agree wholeheartedly with the spirit of Jackson and Jackson's analyses, we would like to address two issues that we take as problematic in their treatment. First, we would like to clarify the distinction coined by Jackson and Jackson (1995) between serial and statistical knowledge. It appears to us that this terminology tends to ascribe a qualitative character to a difference that we believe is better understood as purely quantitative. Second, we would like to suggest that the main issues raised by Jackson and Jackson are problems only because the sequences that are typically used in this paradigm are deterministic. Other researchers have used probabilistic sequences based on finite-state grammars (e.g., Cleeremans and McClelland, 1991; Cleeremans, 1993a; Jimenez et al., 1996), and we think that using probabilistic sequences rather than deterministic sequences offers important advantages with respect to the kind of data analysis that Jackson and Jackson (1995) propose to carry out. Indeed, as we describe below, probabilistic sequences produce more data points to compute correlations with, and enable the experimenter to increase the number of generation trials without producing new learning, therefore again improving the reliability of the indices upon which correlational analyses are computed. We illustrate these points in the following sections, by discussing the results obtained by Jimenez et al. (1996).

2. Serial vs. Statistical Knowledge: A Substantive Difference?

As proposed by Jackson and Jackson (1995), the main difference between serial and statistical knowledge is only that the former designates knowledge about the set of sequential dependencies that guarantee that participants would make a correct prediction of the next item, thus involving "knowledge of the statistical relationship between many sequence elements" (Jackson and Jackson, 1995), whereas the latter refers to relationship between sequentially adjacent elements of the sequence.

We think that this characterization is misleading. First, if the distinction refers only to differences in processing adjacent sequence elements vs. processing more complex contingencies, we see no good reason to forge a qualitative difference between them. As illustrated below, learning of the contingencies that exist between adjacent elements and learning of more complex contingencies (involving three or more elements) can all be characterized as statistical in nature, and can all be learnt based on the same mechanisms. On the other hand, if serial knowledge is meant to refer to the acquisition of a specific memory of the entire sequence, then it should indeed be considered as a qualitatively different form of knowledge (see Cleeremans, 1993a). However, if serial knowledge means memory for the sequence, then we fail to see how the approach proposed by Jackson and Jackson help in distinguishing between serial knowledge and statistical knowledge. We discuss both issues in the following paragraphs.

To illustrate our first contention - that knowledge about the contingencies which exist between adjacent elements can be acquired in the same way as knowledge about more complex relationships, consider the way in which successful models of sequence processing such as the Simple Recurrent Network (SRN; see Cleeremans, 1993a) process sequential information. SRNs are typically trained to predict each element of sequences presented to the network one element at a time, and have been used extensively (Altman, Dienes and Goode, 1995; Cleeremans, 1993a; Jimenez et al., 1996) as models of implicit sequence learning. To model human performance, the network is presented on each trial with element t of the sequence, and is required to predict element t+1. Its response is then compared to the actual successor of element t as prescribed by the sequence it is trained on, and the error information is backpropagated to modify the connection weights. To enable the network to use the temporal context, its architecture includes recurrent connections on the hidden units.

When training starts, the network is only sensitive to the frequency of each sequence element regardless of the temporal context in which these elements occur, and becomes only gradually sensitive to the constraints set by increasingly longer sequences of previous elements as training progresses. For instance, given a four-element repeating sequence such "ABAC ABAC ...", the network will first become sensitive to the fact that 'A' is more frequent than either 'B' or 'C'. Its prediction responses will reflect this sensitivity in that the output unit associated to 'A' will tend to be more activated than the others in all sequential contexts. Soon, however, the network will predict that either 'B' or 'C' are equally likely to occur only in the context of an 'A', and that 'A' is always the only possible successor to either 'B' or 'C'. At this point, the network is sensitive to first-order sequential constraints. Later in training, the network will further refine its prediction responses and will start to differentiate the two occurrences of 'A' by incorporating information about the predecessor of each 'A' token in their representation. For instance, 'A' in the context of 'B' will now elicit a different internal representation than 'A' in the context of 'C'. This differentiation will in turn enable the network to correctly predict that 'B' is the only possible successor to the first 'A' token, which is always preceded by 'C', and that 'C' is the only possible successor to the second 'A' token, which always occurs in the context of 'B'. At this point, the network is thus sensitive to second-order sequential constraints, because its predictions are now based on the contingencies between subsequences of two elements and their possible successors. Throughout training, the network's responses thus reflect almost exactly the conditional probabilities of occurrence of each stimulus given an increasingly large temporal context set by previous sequences of elements. This smooth integration of information about increasingly large temporal contexts matches participants's performance extremely well (see Cleeeremans, 1993a, for detailed comparisons), and suggests that there is no qualitative difference between sensitivity to first-order temporal constraints and higher- order temporal constraints. Indeed, the mechanisms and representations involved in processing first-order sequential information are exactly the same as those involved in proces sing higher-order sequential information.

The second point is that there is another sense in which the serial vs. statistical distinction may in fact be valid and useful. When presented with a short repeating deterministic sequence, people are likely to be able to memorize the corresponding sequence of movements after a few exposures. As training progresses, they would thus increasingly be able to retrieve information about where the stimulus will appear next based on their memory of the entire sequence. This process of memory retrieval is rather different from the kind of prediction process instantiated by the SRN. To see the difference,consider what happens when the sequence, rather than being deterministic, is probabilistic in nature. Such a sequence is impossible to memorize exactly because it never repeats exactly, and because there is always uncertainty associated with each element. Thus, this kind of probabilistic sequences may be used as controls to eliminate the role of serial knowledge in sequence learning and performance. Probabilistic sequences can be generated based on a finite-state grammar such as those used by Cleeremans and McClelland (1991) or by Jimenez et al. (1996). In these experiments, successive stimuli were not read off a repeating deterministic sequence but were instead generated on a trial-by-trial basis by following a probabilistic path in a finite-state automaton. Finite-state grammars consist of a given number of nodes connected by arcs. Arcs correspond to one of the screen locations at which the stimulus may appear. Several arcs can emanate from a given node, and different arcs may correspond to identical screen locations. Stimulus generation involves (1) randomly selecting an arc emanating from the current node, (2) recording the corresponding screen location, and (3) setting the current node to be the node that arc selected in step 1 points to. With the further assumption that the grammar is re-entrant, that is, that the first and last nodes are identical, a probabilistic sequence that is not limited in length and that never repeats exactly over great lengths can easily be generated. To assess learning with this kind of material, one can substitute random screen locations to the screen locations that had been selected based on the generation procedure described above in a given proportion of the trials (typically, 15 to 20%). Learning is then assessed by comparing performance on the grammatical trials with performance on the random trials interspersed throughout training.

This procedure makes it impossible for participants to memorize more than a few very frequent sequential patterns. Hence, they cannot develop full knowledge of which specific elements can follow other elements. This contrasts strongly with what happens with the deterministic sequences used in most implicit sequence learning research. Yet, participants can learn about both kinds of material. This suggests that even when it is impossible to memorize the sequence, participants can still become sensitive to the statistical relationships between many sequence elements, in a way that we believe is best exemplified by the processes instantiated by the Simple Recurrent Network model.

Is there any evidence that the effects of memorizing the sequence can be distinguished from the effects of learning about the statistical structure of the material? Curran and Keele (1993) explored precisely this issue by asking some participants of their experiments to memorize a sequence and by telling them to use this knowledge during the SRT task that followed. The results indicated large effects of this intentional orientation. Participants who knew the sequence were systematically and considerably faster than "incidental" participants, at least when both groups performed the SRT task under conditions of undivided attention. Both groups, however, were able to learn about the sequence and exhibited reliable differences between performance on sequential blocks and performance on random blocks. The question about whether incidental participants merely learn to memorize the sequence during the task, or whether additional and distinct processes are involved is still an open question. However, based on these and other results with probabilistic sequences, it appears fair to say that memory of the sequence is not the same thing as knowledge of the statistical regularities present in the stimulus material (see Cleeremans, 1993b, for a computational account of this distinction applied to the Curran and Keele studies). In this sense, the distinction between serial and statistical knowledge appears to capture an important qualitative difference between the various processes involved in SRT situations.

3. Probabilistic vs. Deterministic Sequences and Assessment of Explicit Knowledge

Jackson and Jackson (1995) proposed the use of correlations between the transition structure of the sequence and the pattern of responding to the generation task as a better procedure to assess explicit knowledge than are the typical indices based on generation accuracy. In this section, we propose that using probabilistic rather than deterministic sequences is a better way to address the thorny issue of exploring the relationship between implicit and explicit knowledge. Obtaining valid and reliable correlations between the structure of the sequence (as defined by contingencies of any given length) and the pattern of responding to either SRT or generation tasks is difficult when the sequence is deterministic for two reasons. First, by virtue of being deterministic, these sequences fail to contain many combinations of sequence elements. Because of this, the number of relevant data points with which to compute correlations is necessarily low.

Second, and most importantly, using a repeating deterministic sequence as the stimulus material seriously limits the possible duration of the generation task, because extended practice at this task when the sequence is deterministic offers a new learning opportunity for participants. This is problematic because generation performance can no longer be considered as a valid measure of the explicit knowledge acquired during the SRT task if participants are allowed to learn within the generation task. The traditional way to address this issue has been to severely limit the number of generation trials to which participants are exposed. For instance, some authors have suggested that only responses to the first one or two repetitions of the sequence should be considered as relevant to the analyses (e.g., Willingham et al., 1989). However, restricting the number of analyzable generation trials in this way again results in a very low (typically, 2 to 6 trials for each data point) and often insufficient total number of observations with which to conduct further analyses.

What can be done to increase the number of data points while retaining the generation task as a measure of explicit knowledge? Based on previous (e.g., Cleeremans and McClelland, 1991) and recent research (Jimenez et al., 1996), we suggest that using probabilistic rather than deterministic sequences as the stimulus material can address the two problematic issues described in the last paragraph.

There are at least three advantages of using a probabilistic stimulus generation procedure such as the one described in the previous section. First, because random stimuli are randomly interspersed with the sequence elements generated based on a probabilistic finite-state automaton, learning can be assessed continuously throughout training and without disrupting participant's representations of the task by abruptly switching from structured to random blocks.

Second, because the sequence is probabilistic, learning is much slower, both in the SRT task or in the generation task. This allows for a large number of observations to be collected during generation, for instance, while at the same time ensuring that participants do not acquire new knowledge during generation. For instance, participants in the Cleeremans and McClelland (1991, Exp. 2) were exposed to 450 trials of the continuous generation task, but nevertheless failed to improve their prediction accuracy with practice. Note that the task is not impossibly hard, as participants did exhibit a significantly better ability to predict grammatical trials over random trials.

Finally, a third advantage of this procedure is that it allows for a far greater number of different combinations of sequence elements to be represented through the sequence than when using deterministic sequences. Indeed, because of the substitution procedure, any stimulus is likely to occur at least a few times in the context of any sequence of other stimuli. Analyzing this kind of data therefore enables us to explore participants' sensitivity to the sequential constraints contained in the material with great detail.

These three features of probabilistic material make it particularly suited to implement the kind of statistical analyses proposed by Jackson and Jackson (1995), and to further extend these analyses by exploring not only the relationship between generation responding and the first-order structure of the material, but also its relationship with the contingencies defined by contexts of length 2, 3, and so on. Detailed examples of such analyses, as well as a generalization of this correlational approach to the analysis of SRT performance as it relates to the statistical structure of the material (as defined by the conditional probabilities associated to the appearance of each stimulus in the context of arbitrary sequence fragments), can be found in Jimenez et al. (1996).

To briefly recap the arguments and results, we have suggested that both SRT and generation performance should be assessed (1) by analyzing whether grammatical successors of different contexts are treated more efficiently than non-grammatical ones, and (2) by analyzing whether each successor is responded to (i.e., identified or generated) according to its theoretical conditional probability of appearance after any given context of any given length (Jimenez et al., 1996). The basic assumption that underpins this reasoning is that the best any system can do from the perspective of preparing itself optimally to the appearance of the next stimulus is to be sensitive to its conditional probability given the largest temporal context it can encode. If this is the case, we would expect the distribution of reaction times to different stimuli in different contexts of a given length for instance, to be strongly correlated with the corresponding distribution of conditional probabilities as observed in the stimulus material. There is ample evidence that this correspondence exists, and also that it develops over time in a way that is similar to the way it develops in the Simple Recurrent Network model (see Cleeremans and McClelland, 1991).

To illustrate how this kind of correlational approach can improve our understanding of the relation between direct and indirect measures of sequence learning, Jimenez et al. (1996) analyzed the results of a probabilistic sequence learning task by computing the average reaction times and generation probabilities of each sequence element after contexts of several different lengths, and by correlating these data with the corresponding distributions of conditional probabilities. The average correlations between generation performance and the distribution of conditional probabilities were .47 for contexts of length 1; .22 for contexts of length 2; and -.01 for contexts of length 3. Similar analyses for SRT performance (late in training) demonstrated that the corresponding correlations were -.60; -.41 and -.03 respectively. Finally, partial correlational analyses between the distribution of reaction times and the distribution of conditional probabilities (controlling for knowledge also expressed through the generation task) showed that some knowledge tended to be exclusively expressed through the SRT task. Indeed, the partial correlations between reaction times and the corresponding conditional probabilities explained approximately 20% of the variance for contexts of length 1, and over 10% of the variance for contexts of length 2 (see Jimenez et al., 1996). We concluded that these results were suggestive of a dissociation between SRT and generation performance.

4. A Final Point About Participants vs. Knowledge as the Unit of Analysis

The conclusion that such a dissociation between otherwise similar direct and indirect measures of sequence learning can safely be interpreted as a demonstration of the acquisition of some implicit learning during the SRT task has been extensively discussed elsewhere (see Jimenez et al., 1996). These arguments will not be repeated here. Instead, we would like to end this comment by explaining why we think that separating the knowledge that each subject reveals through either the generation task or through the SRT task seems to be a better procedure than the one advocated by Jackson and Jackson (1995), which essentially consists of considering participants as the unit of analysis, and of categorizing them as "aware" or "unaware" based on generation performance.

As we see it, the main problem is that using a criterion based on generation performance is intrinsically arbitrary because the criterion can always be changed depending on the author's specific hypotheses. To be fully valid, it seems that the only valid criterion of this kind is the total absence of explicit knowledge - a result that could hardly obtain with the typical active, hypothesis-testing students who participate in implicit learning research. Moreover, classifying some participants as "aware" or "unaware" based on any other criterion could be misleading. For instance, if some participants manage to recall or reproduce a sequence consisting of four or more consecutive trials (the "awareness" criterion of Willingham et al., 1989), it would seem much more informative to look at their performance on the part of the sequence that they have failed to report about rather than to eliminate such data from further analysis. By the same token, the fact that some other participants failed to produce a series of more than three consecutive trials does not rule out the possibility that their responding to the SRT task could be based, for instance, on their explicit knowledge about the association observed between pairs of trials. As stated by Jackson and Jackson (1995), knowledge of a small set of the most probable transitions could be enough to account for fast reaction times to sequentially structured elements. Therefore, instead of categorizing participants as "aware" or "unaware", we think that the most productive strategy consists of considering knowledge as the unit of analysis. This approach involves two steps: (1) determining which knowledge each participant exhibits through generation performance, and (2) comparing this generation knowledge with the knowledge that the same participant exhibits through the SRT task. We believe that this strategy is now beginning to produce results that may help clarify some of the central issues involved in implicit learning research.

Acknowledgements

This work was supported by grant XUGA21102B93 from the Conselleria de Educacion e Ordenacion Universitaria da Xunta de Galicia (Spain)to the first two authors. Axel Cleeremans is a research associate of the National Fund for Scientific Research (Belgium).

References

Altman, G.T.M., Dienes, Z. and Goode, A. (1995). Modality independence of implicitly learned grammatical knowledge. Journal of Experimental Psychology: Learning, Memory and Cognition;21, 899-912.

Cleeremans, A. (1993a). Mechanisms of implicit learning: Connectionist models of sequence processing. Cambridge, MA: MIT press.

Cleeremans, A. (1993b). Attention and awareness in sequence learning. Proceedings of the 15th. Annual Conference of the Cognitive Science Society. Hillsdale, NJ: Erlbaum.

Cleeremans, A. and McClelland, J.L. (1991). Learning the structure of sequence events. Journal of Experimental Psycology: General, 120, 235-253.

Cohen, A., Ivry, R.I. and Keele, S.W. (1990). Attention and structure in sequence learning. Journal of Experimental Psychology: Learning, Memory and Cognition;16, 17-30.

Curran, T. and Keele, S.W. (1993). Attentional and non attentional forms of sequence learning. Journal of Experimental Psychology: Learning, Memory and Cognition;19, 189-202.

Jackson, G. and Jackson, S.R. (1992). Sequence structure and sequential learning:The evidence from ageing reconsidered. Technical Report No. 92-9,Institute of Cognitive and Decision Sciences, University of Oregon.

Jackson, G.M. and Jackson, S.R. (1995). Do measures of explicit learning actually measure what is being learnt in the serial reaction time task? Psyche, 2(20).

Jackson, G.M., Jackson, S.R, Harrison, J., Henderson, L. and Kennard, C. (1995). Serial reaction time learning and Parkinson's diesease: Evidence for a procedural learning deficit. *Neuropsychologia, 33(5), 577-593.

Jimenez, L., Mendez, C. and Cleeremans, A. (1996). Comparing direct and indirect measures of sequence learning. Journal of Experimental Psychology: Learning, Memory and Cognition, 22, 948-969.

Nissen, M.J. and Bullemer, P.T. (1987). Attentional requirements for learning: Evidence from performance measures. Cognitive Psychology, 19, 1-32.

Perruchet, P. and Amorim, M. (1992). Conscious knowledge and changes in perfromance in sequence learning: Evidence against dissociation. Journal of Experimental Psychology: Learning, Memory and Cognition, 18, 785-800.

Reed J. and Johnson, P. (1994). Assessing implicit learning with indirect tests: Determining what is learnt about sequence structure. Journal of Experimental Psychology: Learning, Memory and Cognition, 20, 585-94.

Shanks, D.R. and St. John, M.F. (1994). Characteristics of dissociable human learning systems. Behavioral and Brain Sciences, 17, 367-447.

Willingham, D.B., Greenley, D. B., and Bardone, A. M. (1993). Dissociation in a serial response time task using a recognition measure: Comment on Perruchet and Amorim (1992). Journal of Experimental Psychology: Learning, Memory and Cognition, 19, 1424-1430.

Willingham, D.B., Nissen, M.J. and Bullemer, P. T. (1989). On the development of procedural knowledge. Journal of Experimental Psychology: Learning, Memory and Cognition, 15, 1047-1060.