Back to All News

EDUCATION, EMPLOYMENT AND WORKPLACE RELATIONS REFERENCES COMMITTEE - 29/10/2010 - National Assessment Program-Literacy and Numeracy

Senator Hanson Young -I will go straight to the criticisms of the use of the NAPLAN test as a significant part of the information displayed and presented on the My School website which, overall, is giving a representation of a school's performance, whereas the NAPLAN test is a test that is assessing an individual student's performance. What is your response to the criticism that you cannot somehow have a list of schools or web pages of schools-they do not have to be in a list like a league table-and then compare those schools based on an individual student's performance and say that that is the overall school's performance?

Mr Adams -It is quite acceptable to get an overall measurement of the performance of students in the school by averaging the performance of the students within it. That is done in every international survey of educational achievement. We compare the performance of countries-for example, in PISA-by doing the same thing. We aggregate up from the student level to a national level and indeed to an international level. So aggregating the responses of individuals within a unit is quite acceptable in measurement terms and that is what is done in, for example, educational research. All educational research is based on aggregating performance of individuals within the unit.

Senator Hanson Young -You do not accept the criticism that we cannot be just comparing a school's performance based on this one test?

Dr Hill -I do agree that in making a judgment about the overall performance of a school you need to take those results along with anything else.

Senator Hanson Young -What is everything else?

Dr Hill -Everything else would include, first of all, an understanding of that school's context and the student characteristics within it. One needs to understand the socio-economic characteristics of the school, the proportion of students who are Indigenous and whether the school is remote. There are so many factors that do in fact impinge upon achievement and the more of those we can capture the better. However, the most powerful factors that we know about in educational research are those that relate to the socio-economic and socio-educational characteristics of the students attending the school. That correlation is very high. The correlation between the ICSEA measure from last year and the average school performance on NAPLAN was over 0.8, which is an extraordinarily high correlation. That was further confirmation that there is that very strong link, possibly too high for comfort. But it does exist.

Senator Hanson Young -What is ACARA's response to the criticisms of the statistically similar make-up of schools and how they are presented? If we are looking at ICSEA as the benchmark for how we compare, is that really a true reflection of similar schools?

Dr Hill -The concept that we judge the educational needs of students using a proxy, namely, the socio-economic character of the students within it, has been around in Australian education for probably 30, 40 years. It has been the basis for how we have funded non-government schools for all that period of time.

Senator Hanson Young -Plenty of people think that that is wrong.

Dr Hill -It is a proxy for the real thing, which is probably the thing that we really want to know: what are the actual educational needs of students? But, in the absence of that information, that is what we have used.

Senator Hanson Young -So there could be a better way of doing it.

Dr Hill -Absolutely. What everybody would say is that the best information one can have is the starting points of those students in terms of education progress, because that gives you your best measure. In the absence of that ICSEA was developed using the methods we have used for many years and using census collection district data. Overall, it is highly valid but there are those exceptions when the students from a school come from an area where those students are not representative of the census collection district from which they come. That certainly was the case last year. We asked the school in each jurisdiction to tell us if they felt that this misrepresented them-and changes were made, as you know

Senator Hanson Young -How many changes?

Dr Hill -I gave you that figure last time, Senator.

Senator Hanson Young -I think you were part way through it; that is why I am asking again.

Dr Hill -Yes. The figure is escaping me but it is the same figure. I will provide it to you again. It was under 15 per cent; it was of that order.

Senator Hanson Young -Could you take that on notice? Because I asked it in estimates does not mean the other senators know.

Dr Hill -Yes. Anyhow, ministers have said all along that they do not really want to have this indirect data. They really want direct data. They have asked ACARA to look at the possibility of using direct student data so that we would not have that problem. Indeed, that is what we will be making using of for the next version of the website.

Senator Hanson Young -How do we collect that?

Dr Hill -It is collected at the point at which the students who take NAPLAN take the test.

Senator Hanson Young -Does that survey go home to parents or something?

Dr Hill -The data derives from data held by every school at the point at which the student is enrolled at the school. This year we will have students taking the test in years 3, 5, 7 and 9. The students who took the test last year, who were also in years 3, 5, 7 and 9, this year are in 4, 6, 8 and 10. In other words, we have data over the last two years. We did not have it for the first year of testing but for the last two years we have had this direct data that can give a good characterisation of the school because we have the data for kids in years 3, 4, 5, 6, 7, 8, 9 and 10. So as we have that direct data we will be using it where it is available and where it is reliable.

Senator Hanson Young -You think that this process is about trying to pull together as much various different methods of data collection as we can so that we can try to paint an overall picture, and that that one of the key things is progress, so that you can build upon that. I accept that there are some arguments against the validity of that as well-who is responsible for the year 7 results; is the high school or is it the previous primary school? In some states they do not even start high school until year 8. Putting that aside, do you think that perhaps, if we needed to collect several years-worth of data, the launch and palaver that went with the opening of the online My School website was a little premature?

Dr Hill -I do not, although I can understand the reaction. I believe that until we published we were not going to get the data. Since we published there has been a scramble to get extra data. So things have moved along very rapidly. My view is that if we had waited for that to be in place we would still be waiting in five years time.

Senator Hanson Young -Why?

Dr Hill -I probably should not express an opinion on this but it has been my experience-and also the experience of PISA when they first established their program-that it was the publication of data that improved the data. So I believe that publication of data improves the quality of that data. But to come back to your point on change, change is important. I think we can have good measures of change, but this year for the first time-because we could not have done it in the first year; you need three years so you have the years 3 and 5, the years 5 and 7, and years 7 and 9-the changes between three and five, five and seven and seven and nine will be published, but only for students who were in the same school in those periods. So it is completely valid. In other words, only the Queensland, WA and South Australia schools will have that change for years 5 and 7, but all the others will have it for years 7 and 9. So that is being attended to, and it is an important thing that we do.

Senator Hanson Young -What is ACARA doing in relation to the copyright issues of the data on the My School website?

Dr Hill -We have been asked to look at two things. One is to ensure that with the copyright issue, which is actually set out in the current version, although very few people looked at it, people do actually accept the terms and conditions, particularly those copyright conditions, when they log on. I think that is important. Mr Gavrielatos rightly pointed out that there have been people who have taken our data and then used it in a commercial way. So we do want to stop that and calling on them to accept those conditions puts us in a stronger position in dealing with those cases.

Senator Hanson Young -Are you saying that does not currently exist?

Dr Hill -What does exist are terms and conditions, but the requirement that they accept them when they log on to our site was not there. That will now occur.

Senator Hanson Young -Once that happens, if somebody does misuse the data-publish a league table in the Daily Telegraph for example-does that mean ACARA or the federal education department has an opportunity to prosecute?

Dr Hill -Not newspapers, who are generally exempted from a lot of that. We are referring particularly to those commercial websites, as was Mr Gavrielatos, who went on a website that organised that material in ways that everybody would disagree with and have objections to, as we certainly do.

Senator Hanson Young -So newspapers are not covered by the copyright clause?

Dr Hill -Newspapers are covered by their own legislation, as you know. They are aware of all the legislation and of course are very careful to stay within that.

Senator MARSHALL -We can rely on them to act responsibly.

Senator Hanson Young -Are you saying that, in ACARA's opinion, you would not be able to prosecute if a newspaper printed a league table based on the information from the website?

Dr Hill -My advice is that, for that to occur, you would need legislation of the kind enacted in New South Wales but never actually enforced.

Senator Hanson Young -Thank you.

Back to All News