QMD Recommendations for Volume Test content
New Content: Individual Form
Sex
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
General Population
• Testing with general population found that some respondents did not notice the inclusion of
the third category “intersex”.
• Among those that did notice there was a general lack of understanding of the term “intersex”.
• Reactions to “intersex” category ranged from curiosity to inability to conceive that a third
option for biological sex was possible.
LGBTQI
• Non intersex respondents understood the term “intersex” was not applicable to them.
• Some trans respondents reported for their biological sex, others for their transitioned
identity.
• The one intersex respondent preferred the term ‘intersex’ over ‘indeterminate’. They
suggested that “sex diverse” may be a more inclusive term than intersex.
Recommendations
• Recommend inclusion of Sprint 5 question version for Volume Test with help information
available.
• While testing to date shows general acceptance of the third category, we understand that the
testing done has likely gained the perspectives of the most compliant respondents and true
feelings may be masked due to the presence of the interviewer.
• Inclusion in the Volume Test will allow us to understand reaction to the third category more
fully in a larger scale and non-observed test environment. Examples may include volume of
calls made to call centre, access to help information and for paper forms, multiple response
and form annotations.
• Comparison of observed intersex responses in the Volume Test can be compared against
administrative sources of data such as birth registration data to gain an indication of facetious
responses.
• Recommend some advice from experts in Māori language/Māori world-view in the
implications for asking in Māori language specifically and of Māori respondents in general.
1
Gender
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
General Population
• Many respondents had a sense of déjà vu’ when they got to the gender question. In many
cases the sex-based routing given in the ‘babies born alive’ question immediately preceding
gender in Sprint 5 alerted respondents to there being two similar looking questions in the
form. In Sprint 4, many more respondents didn’t have a sense of Deja-vu due to the questions
being spaced far apart on the form and the absence of sex based routing prior to the gender
question.
• Some respondents realised there was a difference between the sex question and the gender
question, but others remained confused.
• Some respondents understood the difference between sex (biological) and gender (identity),
but still had difficulty understanding that the categories were not dichotomous.
• Many respondents felt that the distance between the two questions (sex and gender) was odd
and wondered why the gender question was not placed together with the sex question on the
form.
LGBTQI
• Respondents in the LGBTQI community liked the inclusion of this question and several felt the
question was clearer than the sex question.
• Some trans respondents selected ‘gender diverse’, while others selected either male or
female.
• Some trans respondents selected either male or female for sex and the same for gender, so it
was not possible to identify them as trans from looking at the form.
• Some respondents queried if they could select multiple responses.
Recommendations
• Recommend inclusion of Sprint 5 question version for Volume Test with help information
available.
• While testing to date shows general acceptance of the gender question, we understand that
the testing done has likely gained the perspectives of the most compliant respondents and
true feelings may be masked due to the presence of the interviewer.
• Inclusion in the Volume Test will allow us to understand reaction to the gender question more
fully in a larger scale and non-observed test environment. Examples may include volume of
calls made to call centre, access to help information and for paper forms, multiple response
and form annotations.
• Recommend development of some kind of note to explain and affirm that for many people
‘sex’ and ‘gender identity’ will be the same thing as a means to reassure respondents we are
not in fact asking the same question twice.
• Question as to whether the information we can collect will meet information need expressed
in topic specifications is still an issue and one we need to take advice on from Customer Needs
and Data.
• Recommend removal of the ‘babies born alive’ question in the context of the sex and
gender questions.
o Both ‘babies born alive’ and ‘gender identity’ are questions that must come after the
routing to identify the NZ Resident Adult population as neither questions are
appropriate to ask of children. Further, if asked, we recommend ‘babies born alive’
2
question precede the ‘gender identity’ question so ‘male’ respondents are routing on
the basis of their sex, rather than have the complication of which basis to route.
However, this increases the sense of ‘déjà vu’ for the majority of respondents for
whom biological sex and gender identity are the same.
o Regardless of the effects of asking babies born together with gender identity, it is
QM&Ds view that it would be preferable to remove ‘babies born’ in the interests of
reducing overall respondent burden, especially given the continued and consistent
feedback we get from respondents regarding the insensitivity of the question.
• Recommend some advice from experts in Māori language/Māori world-view in the
implications for asking in Māori language specifically and of Māori respondents in general.
3
Sexual Orientation
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
General Population
• The majority of respondents answered ‘heterosexual or straight’ in the testing, with a few
respondents answering ‘gay/lesbian’, ‘bisexual’ or ‘other’.
• Most respondents said they were happy to answer, but wondered whether this question
might pose difficulties for other people. A small number of respondents selected ‘prefer not
to say’ or didn’t answer this question.
• Some respondents wondered why this was being asked and what the possible uses of the data
might be.
Targeted LGBTQI community:
• Respondents felt this question was asking about sexual identity, attraction, orientation etc.
• Several respondents made use of the other, please specify option.
• A few respondents found this question very difficult to answer: one is figuring this out;
intersex respondent struggled with this.
• Some queried whether they could select multiple options.
• Were mostly comfortable answering this in a census context. On respondent had concerns
about ‘flow on effects’ of this being asked more widely, data sharing.
Recommendations
• Recommend inclusion of Sprint 5 question version for Volume Test with help information
available.
• While testing to date shows general acceptance of the sexual orientation question, we
understand that the testing done has likely gained the perspectives of the most compliant
respondents and true feelings may be masked due to the presence of the interviewer.
• Inclusion in the Volume Test will allow us to understand reactions to the sexual orientation
question more fully in a larger scale and non-observed test environment. Examples of
potential issues to monitor may include volume of calls made to call centre, access to help
information and for paper forms, multiple response and the number and nature of form
annotations.
• Recommend some advice from experts in Māori language/Māori world-view in the
implications for asking in Māori language specifically and of Māori respondents in general.
4
Second Residence
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
General Population
• Sprint 4 found that respondents had difficulties with the reference period for this question.
The reference to ’30 days a year’ was ambiguous to respondents and was interpreted
differently across individuals.
• The inclusion of a more specific reference period in Sprint 5 introduced some unintended
effects. The second residence question was placed after ‘usual address’ but before previous
address questions in the form. Test findings found that some respondents (those who had
lived at their usual address for less than a year) were interpreting the ‘second residence’
question as asking about their previous address.
• Some respondents had difficulties interpreting the response categories for second residence
(Q8).
• The most commonly selected response for second residence was the ‘other’ category, with
the second most common response selected being ‘home address for a tertiary student’.
• Some respondents with shared parental arrangements found it difficult to decide which
address to put as their ‘usual address’ and which to put as their ‘second residence’
• Some respondents wondered why the government was collecting this information and what it
would be used for.
Recommendations
• Not recommended for inclusion in the Volume Test.
• Recommend further testing for possible later inclusion (Census Test 2017).
• Recommend further mass completion tests to solve unintended effects due to question order
and flow of address questions in the form.
• Recommend including guide notes in further testing to explore whether and how this is useful
to respondents answering this question.
• Recommend including an “other – write in’ response in future testing rounds to see what
other types of second residence people are including and how they are interpreting the
question.
5
• Recommend some advice from experts in Māori language/Māori world-view in the
implications for asking in Māori language specifically and of Māori respondents in general.
o For example, can the concept of 2nd residence be conveyed equivalently in the Māori
language? Do Māori have different relationships with place that suggests new
categories or a different conceptual basis for the New Zealand context?
6
Address 1 year ago / address 5 years ago
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• Testing found that respondents found the routing and cross-references in the address
question suite in the paper form difficult to follow (usual address, second residence, address 5
years ago, address 1 year ago and census night address).
• Difficulty with routing and cross-referencing of address questions may be supported with
evidence from mass completions where there was relatively high non-response to address
questions, and some routing errors made. Non-response may have either been due to
7
difficulty following routing or an introduced error due to the street number being greyed out
in the mass completion test forms.
• Some respondents had difficulties remembering ‘address 5 years ago’: “Oh boy! Can I guess?”
Recommendations
• Recommend reduction in the number of address question by removing address 5 years ago.
• Recommend inclusion of “address 1 year ago’ only in the Volume Test.
• Removal of ‘address 5 years ago’ is recommended due to the complexities in routing and
cross-references on the paper form caused by the number of address question, and, because
it is expected that quality of responses for ‘address 5 years ago’ is likely to be lower than
‘address 1 year ago’.
• Address 1 year ago allows for respondents who report any number of years at their usual
address away from this question, saving the need to answer one of the address questions.
• If conflict among stakeholders regarding utility of the two different reference periods cannot
be resolved, rather than recommend both we would be interested to understand if
developing a question to collect a previous address from all respondents would be a good
compromise. The immediately previous address of a respondent together with their length of
time at current usual address (which is already collected) may allow both groups to be
satisfied.
8
Activity limitation
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• The Washington Group short set of questions were tested in Sprint 4 & 5. Sprint 4 used a
question-by-question approach, while Sprint 5 used a matrix format. There were no
differences in question wording between sprints other than the question instruction/stem in
Sprint 5 which was amended to suit the format used. The questions were included in the all
usually resident population section of the census form (before the 15 years and over routing).
• Cognitive test respondents were shown both formats for this question, and format
preferences were fairly evenly split. None of the respondents expressed a strong preference
for one format over the other.
• Cognitive testing found the following conceptual/interpretation issues with the question
wording:
o Respondents had difficulties interpreting the response categories, in particular the
difference between ‘some difficulty’ and ‘a lot of difficulty’. Respondents said they
would find some examples useful to help them interpret different levels of difficulty.
o Some respondents took into account the use of assistive devices in assessing their
difficulty, while others did not. In some cases, respondents assessed their difficulty
using an assistive device where they did not use one. E.g. one respondent with
hearing difficulties hypothesised that they would have no difficulty hearing if they did
use a hearing aid.
o Respondents did not always assess their answers within the context of a health
problem. This was most common for ‘remembering or concentrating’ and
‘communicating using your usual language’. E.g. many respondents said they
sometimes had difficulties remembering or concentrating but that it was not due to a
health problem. Also respondents with English as a second language, living as NZ
residents said they had difficulty communicating.
Recommendations
• Not recommended for inclusion in the Volume Test. Conduct further, targeted, cognitive
testing for possible future inclusion (Census Test 2017).
• The Washington Group question set was designed for use in interviewer administered surveys
and it is still not clear whether administration of these questions in a self-complete survey is
appropriate or whether it will meet the aims of comparability with international measures.
QM&D need this to be resolved before any recommendation to include could be made.
9
• Recommend testing with respondents under 15 years, and testing of proxy respondents
with parents and other caregivers prior to inclusion in the Volume Test.
• The Washington Group question set was designed for respondents aged 5 years+ in
interviewer administered surveys. Testing done to date has tested these questions with
respondents aged 15+ years, and it is unknown how the question will work for respondents
under 15 years of age or whether parents will be able to provide a reliable proxy response for
their children.
• If programme decision is to include in the Volume Test, recommend use of the matrix
format in the paper form, but question by question approach for online.
• There were no response errors identified in mass completions directly related to the question
formatting. Therefore use of the matrix format in the paper form is recommended to save
page space, while a question by question format would be better suited to online collections,
particularly on small screen devices.
10
Step-family
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• There was some confusion among respondents over the term ‘step-family’ and what scope of
this term included or excluded. Some respondents thought about extended step-family e.g.
step-cousins.
• Some respondents didn’t personally use the term ’step’ to describe their relationships, but
understood it in the context of the form.
• Some respondents wondered about the inclusion of de-facto relationships in step-family.
• Some respondents reported having some issue with this question as it is presented
immediately after the living arrangements question.
Recommendations
• Not recommended for inclusion in the Volume Test.
• Recommend further testing for possible later inclusion (Census Test 2017).
• This question and its interplay with the living arrangements needs to be resolved before we
could recommend inclusion. This will require iterations of testing and experimentation with
order and flow.
• Consideration to be given to developing a question only to identify the presence of step
relationships in the household (yes/no) rather than attempting to establish the nature of the
step relationships. This may be easier to operationalise.
• Consider Māori language/Māori world-view implications for the term ‘step-family’.
Conceptually difficult and misunderstood in English, how will the concept be conveyed in
Māori?
11
Travel to Education (Usual)
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• Most respondents didn’t have any difficulty reporting for their usual travel. However, some
respondents needed guidance as they used different modes of travel on different days of the
week.
• Respondents tended not to read the note text in the question, therefore there were more
instances of respondents marking multiple modes of travel in earlier testing sprints. Due to
this, the question wording was amended back to “what is the one main way you usually
travel…’ in Sprint 5.
• Other modes of travel reported in testing included skateboards, longboards and scooters.
12
• Respondents had difficulty providing a full street address for their educational institute. In
most cases respondents entered the institution name, suburb and city/town only and were
not able to provide a street number and name.
Recommendations
• Recommend inclusion in Volume Test, with following changes:
o Revert to travel on census day (which will require response option of “did not travel
to study on census day’). In consultation, users were split on whether usual travel or
census day travel were most useful. Given census day travel is easier conceptually for
respondents, we recommend census day travel.
o Ask only for ‘name of educational institution’ and ‘city, town or campus’ rather than
exact street address of educational institution.
13
License to occupy
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
• Without guidance or definition the term ‘license to occupy’ was not well understood by
respondents.
• Renters were very likely to select ‘license to occupy’ as amongst the available options, to
them this seemed the terminology most similar to ‘rental agreement’ or ‘tenancy agreement’.
Recommendations
• Not recommended for inclusion in the Volume Test.
• Recommend question revert to 2013 approach. This question only established if the person
owned or partly owned the dwelling (with or without a mortgage). This could include the
addition of wording to instruct respondents not to count license to occupy arrangements.
• Detailed tenure information is collected in the dwelling form, is it necessary to also collect this
in the individual form?
14
Highest qualification
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
• Respondents have generally coped well with this new question format where level is collected
as a tick-box response.
• Where respondents have had trouble is in determining the specific level of their certificate or
diploma. Typically respondents easily recall the level as Certificate or Diploma but are left to
‘guess’ at the specific level.
• Inclusion of examples after certificate and diploma qualifications have not appeared to be
used by respondents. We are unclear if these are useful to include or not.
• Printing of main subject and identification as a New Zealand or overseas qualification are
working well with no issues observed.
• Recommend inclusion in Volume Test, with one potential tweak:
o It would save burden for respondents to aggregate Certificate and Diploma level
qualifications into single categories, especially given that we are unclear if the
disaggregated levels are required for output.
15
Travel to Work (Usual)
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• Most respondents didn’t have any difficulty reporting for their usual travel. However, some
respondents needed guidance as they used different modes of travel on different days of the
week.
• Other respondents found the ‘usual’ concept difficult due to having multiple jobs.
• One respondent on maternity leave wanted to answer this question as they “usually’ work.
• Respondents tended not to read the note text in the question, therefore there were more
instances of respondents marking multiple modes of travel in earlier testing sprints. Due to
this, the question wording was amended back to “what is the one main way you usually
travel…’ in Sprint 5.
Recommendations
• Recommend inclusion in Volume Test, with following changes:
o Revert to travel on census day (which will require response option of “did not travel
to work on census day’. In consultation, users were split on whether usual travel or
census day travel were most useful. Given census day travel is easier conceptually for
respondents, we recommend census day travel.
16
Voluntary work
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• Most respondents reported that they wouldn’t include helping friends and family as voluntary
work, even where those friends and family lived outside of their household.
• Calculation of the number of hours volunteered was difficult for respondents, particularly
where they volunteered for multiple entities over the course of four weeks.
• The category “for an organisation, group or marae” was not broad enough for some
respondents.
Recommendations
• Recommended for inclusion in Volume Test, with
• Respondents can generally answer the questions with the presence of examples seeming to
help with question understanding.
• Possibly consider disaggregation of categories for “Who did you do voluntary work for?”
• While recommending this question, we note that “voluntary work” is different to the 2013
Census concept of ‘unpaid activities’ meaning less respondents are likely to answer in the
positive. This will mean many respondents who in 2013 had options to reflect their unpaid
activities in their own home will now have no such option. This, and the principle of ‘do no
harm’ should be considered in determining content for 2018 Census.
17
New or Significantly Changed Content: Dwelling Form
License to occupy
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• Inclusion of the ‘note’ text in Sprint 5 form was helpful, though all respondents needed to
read this text in order to understand the term.
• Even with the ‘note’ text in place there was still significant misunderstanding of the term, with
the most common misunderstanding being renters feeling the term was referring to their
tenancy agreement in an unfamiliar way.
• Interplay with modified tenure question in Individual Form was problematic.
Recommendations
• Not recommended for inclusion in the Volume Test.
• Variable specification makes mention of counts of license to occupy arrangements, can we
investigate uses of administrative sources to meet information needs?
• Further research needed regarding company share and Māori land scenarios for license to
occupy.
• Consider Māori language/Māori world-view implications for the term ‘license to occupy’.
Conceptually difficult and misunderstood in English, how will the concept be conveyed in
Māori?
• Content to review and advise re the need to collect tenure information in both the Dwelling
Form and Individual Form.
18
Number of rooms
Sprint 4
Sprint 5
Summary of findings (cognitive and Mass completion)
• Generally question is performing well with no major issues.
• Sprint 4 forms had a high incidence of fields being left blank rather than having zeroes
entered. While movement of instruction from instructional ‘bubble’ to note immediately after
the question has appeared to increase the entry of ‘0’ there is still a lot of respondents leaving
fields blank when they have none of the rooms listed.
• Some categories are causing issues. The term conservatory is not well understood by younger
respondents and the presence of ‘caravan …’ in the list is surprising to respondents.
Recommendations
• Recommend this question format for inclusion in Volume Test, with the removal of
‘caravans …’ as a standalone category.
• The format appears to be working well and will derive both number of bedrooms and total
number of rooms in a single question.
• Indications from testing suggest caravans that meet the criteria (use as a bedroom) are likely
to be very rare and the burden of collecting is hard to justify. Recommend reverting to the
approach to 2013 where respondents were instructed to add any caravan used as a bedroom
to their bedroom count.
19
Heating appliance most used
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
• Some comment from respondents regarding categories they felt were missing, particularly
‘central heating’ and ‘open fire’. Expectation seemed to be that these warranted a category of
their own rather than requiring entry as ‘other’.
• Some respondents had issues resolving ‘most often’. For example, using both heat pump and
wood burner in quite different ways.
• Some evidence of multiple response, though not a widespread problem.
Recommendations
• Recommend this question format for inclusion in Volume Test with some tweaks:
o Possibly bold ‘most often’ in question text.
o Possibly include note text to provide guidance to respondent regarding how to resolve
choosing the one method used most often.
o Take advice from Customer Needs and Data regarding whether we should be
collecting the one main method used or all methods used.
20
Amenities available in dwelling
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
• Some respondents considering how ‘broken’ something needed to be before it was not to be
counted (for example a cooker with broken oven but working stovetop).
• Some respondents expressed surprise at the question, general expectation among
respondents was “doesn’t everyone have all of these things?” Some thought it might be
better to ask which of those things, if any, the dwelling did not have.
Recommendations
• Not recommended for inclusion in Volume Test in current format.
• Recommend further testing in communities where we expect some deprivation of these
basic amenities to be prevalent.
o Suggestions include Eastern Christchurch and private dwellings in holiday parks.
• On the one hand, respondents can answer this question and generally find it easy to answer.
From our testing there is no sign of unintended item nonresponse for the category listings.
Despite almost all respondents having to tick all options, this is not particularly onerous.
• On the other hand, we are unclear if census is the appropriate vehicle to be trying measure
these housing deprivation issues. Certainly census will find rare events among the general
population but is a more targeted approach appropriate – is the burden on the many justified
by the information need being filled?
• We are also unclear why some items in the variable spec (inside toilet, fridge, sink) have not
made it to the proposed question.
• Variable specification makes reference to this topic as a measure of “severe housing
deprivation (homelessness)”. The reference to homelessness should be removed.
• We encourage Customer Needs and Data to consider if an amenities type question focussed
on observable housing quality features may meet user need regarding household deprivation
and also provide context to (or replace) the subjective ‘cold’ and ‘damp’ questions. For
example, presence or absence of insulation, home ventilation systems, double glazing etc.
21
Cold
Sprint 4 & 5
Sprint 5
Summary of findings (cognitive and Mass completion)
• Sprint 4 version was problematic for respondents as they felt the question was asking two
quite distinct things and found it difficult to answer.
• For sprint 5, we adapted an existing question from the HES household deprivation module.
Respondents still felt the question asked two distinct things but found this easier to resolve as
the intent of the two things were clearly about heating.
• For both question variants, respondents were generally thinking about the rooms they ‘lived
in’ rather than the dwelling as a whole.
• Respondents volunteered comments on the subjectivity of the question – that the answers
would vary depending on who answered the dwelling form.
Recommendations
• Recommended for inclusion in Volume Test, with some reservations:
• While inclined to not recommend this for inclusion, due to the subjectivity of the question and
the absence of any means to interpret the results, testing has shown respondents can answer
the questions.
• Customer Needs and Data are comfortable with the known subjectivity of responses and
believe the subjective data will meet the needs of users.
• Inclusion of an amenities question focused on insulation, ventilation etc would potentially
allow for validation of the subjective measures against more observable, objective data.
22
Damp
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
• Some respondents answered ‘yes – sometimes’ and made comments about temporary states
of dampness, for example while cooking.
• Respondents volunteered comments on the subjectivity of the question – that the answers
would vary depending on who answered the dwelling form.
• Respondents found it difficult to answer if they hadn’t lived in the dwelling over a winter.
These respondents pointed out the inconsistent use of the ‘don’t know’ response label across
the cold, damp and mould questions.
Recommendations
• Recommended for inclusion in Volume Test, with some reservations:
• While inclined to not recommend for inclusion, due to the subjectivity of the question and the
absence of any means to interpret the results, testing has shown respondents can answer the
questions albeit with acknowledgement of the subjectivity of their answers.
• Customer Needs and Data are comfortable with the known subjectivity of responses and
believe the subjective data will meet the needs of users.
• Inclusion of an amenities question focused on insulation, ventilation etc would potentially
allow for validation of the subjective measures against more observable, objective data.
23
Mould
Sprint 4 & 5
Summary of findings (cognitive and Mass completion)
• One respondent commented that they thought the question should not apply if they have told
us their dwelling is not damp, which while perhaps not valid, does hint that the questions are
all somewhat interrelated and suggests burden for respondents with no housing issues.
• A few respondents answered ‘no’, but did volunteer that they might have a problem with
mould if they didn’t keep cleaning it up. Unclear if ‘no’ is the appropriate response for this
scenario.
• For some residents the wording “… inside this dwelling” was too vague in that they didn’t
think about moulds presence on soft furnishings.
• One respondent felt “… inside this dwelling” was suggesting the whole dwelling was mouldy
which may not be the case, suggested alternate wording “Can you see mould in any part of
this dwelling”
Recommendations
• Recommended for inclusion in Volume Test, with some reservations:
• Respondents have generally been able to answer the question despite some difficulties
calculating the total size of mould against the A4 sheet as the reference.
• QM&D would like some clarity if the A4 size is the appropriate reference to meet the variable
specification requirements regarding presence of mould creating health impacts.
24
