Thoughtful Consideration of Automated Essay Scoring

Automated Essay Scoring (AES) is an emerging area of assessment technology that is gaining the attention of Canadian educators and policy leaders. It involves the training of computer engines to rate essays by considering both the mechanics and content of the writing. Even though it is not currently being practiced or even tested in a wide-scale manner in Canadian classrooms, the scoring of essays by computers is fueling debate leading to the need for further independent research in order to help inform decisions on how this technology should be handled.

However, independent research on automated essay scoring is hard to come by due to the fact that much of the research being conducted is by and for the companies producing the systems. For that reason SAEE, through the Technology Assisted Student Assessment Institute (TASA) commissioned Dr. Susan M. Phillips to scan and analyze the current research on this topic from a variety of disciplines including writing instruction, computational linguistics, and computer science. The purpose of the report, Automated Essay Scoring: A Literature Review, is to communicate a balanced picture of the state of AES research and its implications for K-12 schools in Canada. The review is broad in scope including a wide range of perspectives designed to be of interest to teachers, assessment specialists, developers of assessment technology and educational policy makers.

Most AES systems were initially developed for summative writing assessments in large-scale, high-stakes situations such as graduate admissions tests (GMAT). However, the most recent developments have expanded the potential application of AES to formative assessment at the classroom level, where students can receive immediate, specific feedback on their writing and can still be monitored and assisted by their teacher.

Numerous software companies have developed different techniques to predict essay scores by using correlations of the intrinsic qualities. First, the system needs to be trained on what to look for. This is done by entering the results from a number of essays written on the same prompt or question that are marked by human raters. The system is then trained to examine a new essay on the same prompt and predict the score that a human rater would give. Some programs claim to mark for both style and content, while others focus on one or the other.

In terms of their reliability, Phillips (2007) cautions, to date, there seems to be a dearth of independent comparative research on the effectiveness of the different AES engines for specific purposes, and for use with specific populations…While it would appear that one basis of comparison might be the degree of agreement of specific AES engines with human raters, this also needs to be scrutinized as different prompts, expertise of raters, and other factors can cause different levels of rater agreement.

AES has great potential. It can be more objective than human scoring because the computer will not suffer from fatigue or favoritism. Assessment criteria are applied exactly the same way whether it is the first or the thousandth essay marked on the same prompt. The potential for immediate feedback is also considered positively when AES is used as a formative assessment tool because it allows students to work at their own level and at their own pace receiving feedback on specific problem areas.

This rapid feedback also allows for more frequent testing leading to greater learning opportunities for students. By using computers to grade essays, the marking load of teachers is reduced creating more time for professional collaboration, and student-specific instruction. Since computers are being used more often as a learning tool in the classroom, computer-based testing places assessment in the same milieu as learning and provides more accessible statistical data to inform instruction.

However, adopting AES in Canadian schools requires a careful investigation of the potential threats. Some say that it removes human interaction from the writing process. Writing is a form of communication between an author and a specific audience according to The National Council of Teachers of English, and using AES violates the social nature of writing (Phillips, 2007, p. 25). Other concerns raised are related to whether the systems can adequately detect copied, or nonsense essays. Currently, systems need to be trained by specific prompts. This limits the ability of educators to modify or create their own essay questions, potentially creating greater separation between learning and assessment. Additionally, implementing AES in schools involves not only the provision of access to computers and software, likely purchased from private companies, but also technical support and professional development to sustain its use.

Phillips (2007) highlights important issues to ponder when considering whether AES is beneficial to implement at the K-12 level in Canada and concludes her review with eight recommendations in the areas of pedagogy, technology research and educational policy.

To order a copy of the full report or to request a presentation by Dr. Phillips please contact The Society for the Advancement of Excellence in Education

Comments Off

How to Teach Ethics Education

The quality of life depends on the quality of who you are. What you actually become. It all boils down to how morally good and ethical a person is. So the question is how do we make every one become a good moral person? Since ancient times we recognize the importance of teaching moral values. We teach it the same way we teach math and science by providing the knowledge of good and bad. One very important factor that the world of education has still not completely woken up to, in spite of all these thousands of years of research, is that as there are two kinds of intelligences – regular and emotional – both require a different set of rules for education. Emotional intelligence education alters the actual physical infrastructure of the brain. Emotional intelligence education starts from the womb. And continues through hugs and kisses and a nurturing childhood environment. By the time the child is 6 years old the moral character traits are well set for the rest of the person’s life.

So to really insure moral education we must learn how to educate the individual at the fetus and the child stage. Thus we must put together templates and manuals for future parents, current pregnant ones and those with little kids.

Please consider the following:

There was a king who was very troubled because his people were very poor. He did not know what to do. He heard of this kingdom where people were very prosperous and lived in mansions and even marble palaces. So he went to the king of this place and asked him how they were able to live like this. The king told him that it was very simple he just passed a building code which everyone in his kingdom had to follow. So our king came back and passed a law that everyone must build a marble palace!

Now in his kingdom most could afford a straw hut, others could afford a log cabin. Still others could afford a cement mansion and a few could afford a marble palace. So the kings law went into effect and nothing changed. Just a few marble palaces came up. Except for the select few the rest were incapable of building marble palaces.

It is the same when it comes to morality. We have moral laws and we expect everyone to follow them. We spend billions of dollars on crime prevention and containment yet nothing changes.

The only way to change and reduce crime is by changing the physical quality of the brain that generates the moral compass of the individual. Thus ethics education means not just telling people what is good and what is bad. It is about creating the right moral infrastructure generating brain.

The brain has four basic levels as follows:

1) Premature brain – (I have quantified it as -2) Those stuck on this level have the moral values of a snake. In their mind they are everything and everyone else is nothing. They are above the law and everyone else is below the law. No amount of moral education is going to change them. Punishment is the only deterrence and even this they often ignore. Their physically brain is too far morally gone.

2) Immature brain – (I have quantified it as -1). Those stuck on this level are corrupt. In their mind they deserve the best, by hook or by crook. Current ethics education will not change them much as their moral values are generated by an entrenched selfishness producing brain infrastructure. We have to wake up to the fact that we will have to change this brain infrastructure.

3) Mature brain – (I have quantified it as +1). Those stuck at this level are driven by a trophy self image as in ‘I am the best’. This is the group that is the easiest to change. But not by the current way of moral education which practically amounts to pleading with people to do good. The trophy self image will have to be gotten rid of and replaced with a selfless self.

4) Super mature brain – (I have quantified it as +2). The brain at this developed level already generates a selfless self so moral education is not required for this group.

Just like a man with the resources of building a log cabin cannot become a marble palace owner; a man with a -2, -1. or even +1 brain can ever become morally +2. No amount of current ethics education can make a -2 brain generate a +2 brain power. The only way real moral values can take root is by making the brain +2. And this requires brain therapy more than anything else. It requires brain changing education. The sooner we wake up to this the faster we will create a morally healthy society. Current moral education amounts to like our local pastor telling us to live virtuous lives. We hear it and applaud it and then go back to our old ways.

Notice how without exception all books on self help talk of ’7/9 steps. There is just one goal/step as far as self help is concerned – become wise. Thus ethics education is more like self transformation education. It is more precisely about making ones self +2! It is more in line with self help education. Then again not like the current self help education. My main field of work is wisdom. And as philosophy is considered the love of wisdom which means one of the main goals of philosophy is to try to find out what is wisdom. Most experts on wisdom are still stuck where they define wisdom by its attributes and are still struggling to define wisdom.

As the wisdom potential is there in every brain just like blood and as absence of wisdom means presence of ignorance (where actually there is a mixture of the two) so every life is affected by lack of wisdom, from individual to group to country, so the stakes are very high. In fact much of the present mess in the world today from the economic crisis to the problems of war can all be traced to lack of wisdom. So if I can wake up main stream education/science to the correct path to ethics education, I think my work will be done.

Philosophy, religion and science in my view are much more connected then we realize! I have found that at the highest stage of the mind/brain development the quality of the character traits (as defined by religion, philosophy and science) of every person are all one and the same!!! Religion wants you to be altruistic, philosophy wants you to be wise/altruistic and science/education wants you to be emotionally super mature which is the same! So wisdom is the common thread in all of them. Thus we need philosophers, religious leaders and scientists/educationists all working together to create ethics education that transforms the brain to +2.

Comments Off