An examination of Nation's (1990) Vocabulary Levels Test

 

Kamimoto Tadamitsu

Kumamoto Gakuen University

 

Introduction

Nation's (1990) Vocabulary Levels Test (VLT) is widely used as a vocabulary size test. It consists of five word levels: 2000, 3000, 5000, UWL (University Word List), and 10000 and has four "equivalent" forms. Beglar and Hunt (1999) examined the equivalence and validity between the four forms of the 2000 and UWL levels by means of the classical item analysis and then proposed two revised forms each consisting of 27 items. They also validated those revised forms by Rasch item analyses. As a result, they claimed that the two revised forms have greater content validity than the original forms and that they are adequately equivalent.

Beglar and Hunt (1999) made significant contributions to our knowledge about the measurement of learners' vocabulary size, but to make their versions more reliable and valid, their study needs to be reviewed from the following points.

1. Combination of the 2000 and UWL levels

2. Subjects used (TOEFL: roughly from 260 to 550) and the purpose of the VLT (i.e., NRT)

3. Unsatisfactory items still in the revised forms (esp. loanwords such as "motor")

The purpose of this study is to replicate with subjects of more homogeneous L2 proficiency and examine the VLT from different angles. I will also propose some considerations we need to make in developing a reliable and valid vocabulary levels test for the EFL Japanese students.

 

Method

A group of 202 EFL Japanese university students took Forms A and B of the 2000, 3000, and 5000 word levels at over a 2-week interval. Due to absence on one testing occasion, eight students were dropped from the study, leaving a total of 196.

Results

 

1. Equivalence

Repeated measures ANOVA tests showed that Forms A and B of the 2000 and 3000 levels were statistically different, F (1, 195) = 8.95, p = 0.0031 and F (1, 195) = 30.35, p < 0.0001, while those of the 5000 level were not at p < 0.01, (F (1, 195) = 6.05, p = 0.0147). Thus, Forms A and B of the 2000 and 3000 levels are not equivalent with these groups.

 

2. Comparison of Forms A and B

It is generally assumed that students score more at the 2000 level than at the 3000 level. However, Table 2 shows that over 40% of the subjects scored less at the 2000 level than at the 3000 level on Form A and so did about 28% on Form B. Concerning those subjects, a wider range is also shown on Form A than on Form B.

 

 

 

 

Table 1. Descriptive Statistics

 

 

Form A

 

 

 

Form B

 

Level

2000

3000

5000

 

2000

3000

5000

N

196.00

196.00

196.00

 

196.00

196.00

196.00

k

18.00

18.00

18.00

 

18.00

18.00

18.00

Mean

13.96

12.73

8.04

 

13.30

11.34

7.36

Mode

16.00

14.00

8.00

 

14.00

11.00

7.00

Midpoint

14.00

13.00

8.00

 

14.00

11.00

7.00

SD

2.65

2.90

2.89

 

2.62

3.06

3.28

SEM

0.19

0.21

0.21

 

0.19

0.22

0.20

Low-High

5-18

2-18

0-16

 

4-18

1-18

1-17

Range

13.00

16.00

16.00

 

14.00

17.00

16.00

KR21

0.59

0.59

0.50

 

0.52

0.59

0.63


 

       Table 2. Frequency of subjects who scored less at the 2000 level than at 3000 level (N = 196)

 

 

Form A

 

 

Form B

Diffs

Frequency

Cumulative frequency

 

Frequency

Cumulative frequency

-6

0

0

 

0

0

-5

5

05 ( 2.56%)

 

0

0

-4

6

11 ( 5.61%)

 

0

0

-3

4

15 ( 7.65%)

 

3

3 ( 1.53%)

-2

18

33 (16.84%)

 

13

16 ( 8.16%)

-1

25

58 (29.59%)

 

15

31 (15.81%)

0

26

84 (42.86%)

 

23

54 (27.55%)

 

3. Comparison of Item Facility (IF) between the 2000 level and the 3000 level

As can be seen from Table 3, nine loanwords (with an asterisk) on the 2000 level are dispersed throughout the list, but eight loanwords at the 3000 level accounted exclusively for the easier half of the list, with an IF mean of the levels being 85.19% and 93.76% respectively.

 

When the word level to which an item belongs and its BNC frequency listing are compared across the whole list, some words do not belong to the designated word level in the VLT. However, the mean of the BNC frequency figures of all the 2000 level words is larger than that of all the words on the 3000 level (58.78 vs. 36.44).

 

Table 3. Increasing order of difficulty of items on 2000 and 3000 word levels of Form A

 

2000 Word

Level

 

 

 

3000 Word

Level

 

Order

Item

IF

BNC

 

Order

Item

IF

BNC

1.5

salary*

98.81

29

 

1.5

bench*

100

26

1.5

birth

98.81

59

 

1.5

charity*

100

45

3

victory*

95.24

63

 

3

slice*

96.43

15

4

pride*

88.10

27

 

4.5

angel*

95.24

37

5

total*

83.33

122

 

4.5

darling*

95.24

23

6

hide

82.14

64

 

6

thrill*

89.29

0

7.5

elect

79.76

52

 

7

toss*

88.10

13

7.5

temperature

79.76

58

 

8

illustrate

86.90

56

9

sport*

77.38

89

 

9

echo*

85.71

16

10

manufacture*

73.81

14

 

10

annual

78.57

81

11

melt

72.62

14

 

11

administration

75.00

69

12

private*

71.43

173

 

12

encounter

71.43

27

13

debt

66.67

73

 

13

savage

69.05

0

14

roar

60.71

palm

65.48

19

15

original*

53.57

108

 

14.5

scheme

65.48

170

16

invite

39.29

63

 

16

province

61.90

33

17

flesh

36.90

25

 

17

definite

53.57

16

18

spoil*

26.19

15

 

18

herd

52.38

10

Notes: Asterisks are used to mark that the words are listed as a Japanized loanword in Kojien by Shinmura (Ed.). (1999). The BNC (Leech, et al., 2001) column indicates that figures given are occurrences each per million words.

 

 

4. Item analysis with the VLT being used as a CRT

 To seek a wider application of the VLT, the B-index is used to analyse the items from a CRT viewpoint. The B-index is an item statistic that compares the IFs of those students who passed a test with the IFs of those who failed it (Brown, 1996) and those students are respectively called "masters" and "non-masters". Table 4 indicates that the majority of the loanwords do not differentiate "masters" and "non-masters" very well especially on the 3000 level test and also that some loanwords are very well known whereas others are not, as shown by the 2000 word level.

 

Table 4. Increasing order of B-index concerning items on 2000 and 3000 word levels of Form A

 

2000

Word

Level

 

 

3000

Word

Level

 

No

Item

IF-pass

IF-fail

B-index

 

Item

IF-pass

IF-fail

B-index

1

salary*

1.00

0.98

0.02

 

bench*

1.00

0.98

0.02

2

birth

1.00

0.97

0.03

 

slice*

1.00

0.94

0.06

3

victory*

1.00

0.96

0.04

 

angel*

0.94

0.88

0.06

4

pride*

0.98

0.88

0.10

 

charity*

1.00

0.93

0.07

5

sport*

0.95

0.82

0.14

 

echo*

0.91

0.82

0.09

6

total*

0.94

0.78

darling*

1.00

0.90

0.10

7

temperature

0.98

0.80

0.18

 

thrill*

0.97

0.77

0.20

8

hide

0.97

0.78

0.19

 

illustrate

0.97

0.77

0.20

9

manufacture*

0.97

0.75

0.22

 

toss*

1.00

0.79

0.21

10

elect

0.97

0.74

0.23

 

encounter

0.88

0.58

0.30

11

flesh

0.61

0.37

0.24

 

province

0.79

0.42

0.37

12

private*

0.95

0.68

0.27

 

scheme

0.85

0.47

0.38

13

melt

0.97

0.69

0.28

 

palm

0.82

0.42

0.39

14

roar

0.91

0.58

0.32

 

administration

0.97

0.56

0.41

15

original*

0.89

0.54

0.36

 

definite

0.79

0.37

0.41

16

debt

1.00

0.63

0.37

 

annual

1.00

0.58

0.42

17

invite

0.80

0.40

0.40

 

savage

0.91

0.45

0.46

18

spoil*

0.71

0.25

0.46

 

herd

0.91

0.31

0.60

Notes: The cut-point for passing the test was 16 items out of 18 (cf. Read, 1988).

Discussion

1. Loanwords as a testing item have a significant role to play on the 2000 and 3000 word levels.

2. EFL learners do not necessarily learn vocabulary in the order of English word frequency.

3. Some words on the 2000 and 3000 word levels test do not reflect a modern word frequency list very well.

 

Conclusion

  Learners' L1 needs to be taken into consideration when the VLT is revised for an envisaged learner group. In the case of a Japanese group, due consideration should be given to loanwords. Otherwise, the claimed high degree of validity and reliability cannot be accepted at face value.

Selected references

Beglar, D., & Hunt, A. (1999). Revising and validating the 2000 word level and university word level. Language Testing, 16, 131-162.

Brown, J. D. (1996). Testing in language programsUpper Saddle River, NJ: Prentice Hall Regents.

Leech, G., Rayson, P., & Wilson, A. (2001). Word frequencies in written and spoken English. Harlow: Longman.

Nation, I. S. P. (1990). Teaching and learning vocabularyBostonHeinle & Heinle.

Read, J. (1988). Measuring the vocabulary knowledge of second language learners. RELC Journal19(2), 12-25.

Schmitt, N., Schmitt, D., & Clapham, C. (2001). Developing and exploring the behaviour of two new versions of the Vocabulary Levels Test. Language Testing1855-88.