Post-Occupancy Surveys: Don’t ask too much from them

Courtesy of Flickr CC License / albertogp123

Post-occupancy surveys and/or interviews are a common tool used in architecture to evaluate the success of buildings. They can be very useful and should be implemented as long as architects do not expect or claim too much from them.  Much has been said of their benefits, but it is concerning to see some architects present them as some kind of scientific proof of a design’s success or failure. Although I am a strong advocate for post-occupancy surveys, I think a little pushback is necessary. A brief review of their methodological weaknesses should make any architect pause before claiming a survey has vindicated their ideas.

First, obtaining a reasonably random sample of people is difficult in any survey.[1] In the case of a single building, like a school, a survey is that much more vulnerable to selection bias. Unless the staff and students were randomly assigned to the school the survey will inevitably be biased. This means the post-occupancy surveys inadvertently bias the results toward certain socioeconomic levels. It will not be representative of the population at large. This makes it difficult to tell where the socioeconomic effects end and the architectural effects begin.

Even if a building is a representative sample of the general population, the users who feel more strongly about the building can influence the survey. They will be more eager to complete the survey and give feedback. These squeaky wheels will unintentionally or intentionally create a selection bias that favors their interests. Those less interested in the subject are less likely to fill out the survey. Subsequently, the survey might only reflect an extreme view that few other occupants agree with.

Therefore, not only is it unreasonable to expect a representative sample from a single building, but it is also difficult for a survey to obtain a truly random sample among the users in a single building. If say 70% percent don’t fill out the survey then a positive or negative response rating of 80% is meaningless. Remarkably and unethically, some do not even list the response rate when they report on the results of a survey. It is important to know how a study’s authors factored in non-responses to their assessment.[2]

These problems with selection biases help fuel the critics who believe architecture is too unique and subjective to be studied. However, if the data are reliable, architects can control for demographic issues by pooling together and cross-referencing multiple studies (meta-analysis). Of course, this is only possible if the original data are reliable. This leads us to the second methodological hazard of post-occupancy surveys. The wording of the questions and the reliability of the responses can play havoc with the data. For example, depending on the sequence of the questions, certain aspects of the design can look more salient to the users. The following hypothetical example demonstrates how this can happen.

How satisfied are you with the design?
Do you have natural lighting and a view outside?

The correlation between these two questions could be extremely low.[3] Natural lighting and a view outside might not come to the interviewees’ mind when they are asked to assess their satisfaction with the overall design. This isn’t necessarily true when asked in the reverse order.

Do you have natural lighting and a view outside?
How satisfied are you with the design?

Psychologists have shown that reordering questions in this way can dramatically increase the correlation between the two questions. “Satisfaction with the design” is not an easy assessment. It requires factoring in a good number of factors including the cultural, economic, and social impact. The question about lighting and views is a much easier question. By switching the ordering people tend to unconsciously use a heuristic shortcut to answer the more difficult question. Even though the questions are exactly the same, people that are presented the second sequence would most likely be answering how satisfied they are with the views and lighting and not the overall design.[4]

Even more disheartening, well presented questions don’t guarantee the responses will be accurate reflections of the building’s true effects on satisfaction. As a subjective measure, self-reporting isn’t always reliable, and in certain circumstances it is highly unreliable.  If a survey is conducted in person the interviewees might be inclined to give less than honest answers to appear kinder and more open-minded to the interviewer.[5] For example, controversial issues such Privately Owned Public Spaces might elicit politically correct answers rather than candid ones.

Social desirability’s effect on surveys, however, is not necessarily my greatest concern when it comes to self-reporting. Social desirability revolves around the idea that people are to some extent “faking it”. The cognitive biases we all struggle against are enough to make self-reporting unreliable without anyone faking it. For example, in the book How Doctors Think, Dr. Jerome Groopman interviewed a surgeon who fervently believes his “outcomes [for spinal fusion surgery for low back pain] are better than anything in the published literature.” Yet, “when pressed, he admitted that the long-term follow-ups are rare and that he has not participated in any randomized prospective controlled trials comparing fusion surgery with conservative measure such as physical therapy.” Despite the research at the time suggesting fusion surgery only works for a small specific group of patients (those with fractured spines or spinal cancer) Groopman spoke with several other spine surgeons who adamantly refused to enter into a trial comparing simple discectomy with fusion because they are absolutely convinced of fusion’s value.[6] These surgeons have a financial incentive to do more invasive procedures like fusion surgery, but I do not believe they are faking their belief in the procedure, I just doubt the accuracy of their self-reported effectiveness. Imagine how effective fusion surgery would look if we based our conclusion on a survey of these surgeons. Likewise, building users and architects are prone to the same kind of self-reporting cognitive biases that could skew a survey. Architects and users could be absolutely convinced a certain design intervention increased satisfaction when it did nothing of the sort, and perhaps it was even counterproductive.

With only a survey to go on, separating fact from fiction is extremely difficult if not impossible.   Post-occupancy interviews and surveys offer a quick and rough way to identify potential factors that increased or decreased the users’ satisfaction. These factors can then be bolstered by other metrics such as absenteeism and productivity and examined more rigorously in observational studies. Surveys on their own are inadequate to draw conclusions about a building. They can play an important role as long as we don’t ask or claim more from them than what they already provide.

If you enjoyed this article check out more by Christopher N. Henry here.

Photographs:
albertogp123


[1] Seethaler, Sherry. Lies, Damned Lies, and Science: How to Sort through the Noise around Global Warming, the Latest Health Claims, and Other Scientific Controversies. FT Press. January 23, 2009, Location 2273-2276.

[2] Medical researchers can make a drug or procedure look more effective and harmless by not reporting on those that dropped out of the study. The subjects could have dropped out of study for various reasons unrelated to the procedure or drug, or because of ill effects experienced from the intervention. See: Evans, Imogen, Hazel Thornton, Iain Chalmers, Paul Glasziou and Ben Goldacre. Testing Treatments: Better Research for Better Healthcare. Pinter & Martin; 2nd edition 2011 Kindle Location 1550-80.

[3] This is purely an hypothetical example as there does seem to be a fair degree of correlation of views and natural lighting with users satisfaction. See: Ulrich, Roger. “Biophilic Theory and Research for Healthcare Design.” In Biophilic Design, edited by Stephen R. Kellert, Judith H. Heerwagen, and Martin L. Mador,. Hoboken, New Jersey, John Wiley & Sons, 2008

But this “truism” might not be true in all cases See: Henry, Christopher. “Architecture for Autism: Exterior Views,” ArchDaily.com April 04, 2012.

[4] The questions I used here are hypothetical examples as this phenomenon of substitution has not been studied in architectural surveys. It has been studied in a variety of other surveys, and it seems likely that it would be true of other surveys. For a discussion on substitution see: Kahneman, Daniel. Thinking, Fast and Slow. Farrar, Straus and Giroux, 2011 Kindle Location 1854-1881.

[5] Seethaler, Sherry. Lies, Damned Lies, and Science: How to Sort through the Noise around Global Warming, the Latest Health Claims, and Other Scientific Controversies. FT Press. January 23, 2009, Location 2276-2284.

[6] Groopman, Jerome. How Doctors Think. Houghton Mifflin Harcourt, 2008 Location 3194-3298.

Cite: Henry, Christopher N.. "Post-Occupancy Surveys: Don’t ask too much from them" 11 Apr 2012. ArchDaily. Accessed 29 Jul 2014. <http://www.archdaily.com/?p=225083>

2 comments

  1. Thumb up Thumb down 0

    The fact that the survey is biased toward its users makes it an insufficient survey? That is 100% backwards. If the users agree that it is a success, it is a success. Simply because the survey does not take into account how a banker feels about the school you designed does not matter. A school is for students and teachers. If the students and teachers are happy and feel it works, its pretty simple to say that it works. Outside opinions, of people who are not using the building everyday, do not matter.

    • Thumb up Thumb down 0

      Rufus,

      Thank you for your comment. I will try to answer your points in reverse order.

      The idea of a sample bias, say socioeconomic bias for example, is that it is hard to tell between two different school designs if they are located in different socioeconomic locations. If one schools gets a high satisfaction rating and the students do amazing on tests, is that because of design or because of socioeconomic factors? Socioeconomic factors for sure, but how much can be contributed to the design? A poorly designed school might still out perform and receive a higher satisfaction rating than a well designed school if they are located in completely different socioeconomic settings. The sampling bias would have nothing to do with bankers or any other people who would never go there. You are correct those people don’t matter when concerning the satisfaction of the building’s users.

      As for your other point, “if users agree that it is a success, it is a success.” I would refer back to the example of spinal fusion surgery. We often think something is the factor that is making us succeed when really it has nothing to do with it. In the case of the spinal fusion surgery, the doctors felt the procedure helped when the literature at the time pointed to the opposite. So in this case just because they thought it was a success didn’t mean it was a success. The success of a building is much more difficult to measure with the seemingly countless confounds and misleading correlations. Thats why, based on a survey alone, it is difficult to say whether a building is aiding or hindering the success and satisfaction of its users.

      Hopefully this cleared up any confusion my article may have caused.

Share your thoughts