Nov 30 2006

Analyzing audience feedback, part 2

“Average response” vs. response of the “average person”. The point of using the middle response as a representation of the whole group is that you capture what the person in the middle of the distribution said, without being influenced by the tails (whether positive or negative):

Middle:

  • Invigorating. Very interesting dialogue on 3 tensions
  • Interesting, topical, relevant and well presented
  • Interesting to get another perspective

People were engaged in Nancy’s presentation. They connected with her and with the topic. It’s interesting to to me that none of the responses commented on the fact that audience feedback was being collected in a wiki. Good to see the “3 tensions” from our tech study project mentioned. What people in “the middle” said is the main message.


The data analysis strategy here is to chop the whole sorted list in half and then continue chopping the parts in half again. I picked out comments at the cut-points; similar to the median, here are the quartile and the “eighth” and the extreme:Positive Quartile:

  • Fantastic � her delivery style and encouraging participation by audience was great
  • Excellent entertaining presenter. It was good split up into 3 parts with food breaks which gave you a breather and time to refresh yourself

Positive Eighth:

  • Comprehensive review of e-learning resources in community environment
  • An inspiring speaker � I was a bit disappointed as I had registered my mobile beforehand (via an email) but I didn�t get messages � a very engaging and entertaining speaker

Positive Extreme:

  • Fantastic � the best. Excellent speaker
  • Absolutely excellent!! Inspiring, informative and innovative

There was a lot of enthusiasm for the session. Seems like quite a bit of variety, ranging from enthusiasm for the topic, for the process (e.g., SMS) and for Nancy herself as as a presenter.


The negative comments were what first caught my eye because they were painful and so I picked out three comments at each of the cut-points in the distribution to provide a bit more to think about.Negative Quartile:

  • Ok � food for thought
  • Some thoughts and concepts that had not considered before. Great how split into 3 parts
  • Food for thought. Good presenter

Negative Eighth:

  • Too long � I can�t concentrate that long
  • Started well but went on for too long, became boring
  • Rather long � not particularly stimulating

Negative Extreme:

  • Why do we need an American who is full of herself to tell us what we already dabbled in during our LearnScope projects?
  • Long, elitist
  • Hard to follow � esoteric. I didn�t warm to her and felt she presented poorly which immediately creates a barrier to an audience (read �me�). She was disjointed in my opinion. Not clear on outcomes.

The quartile comments are actually pretty positive, so you can say that three-fourths of the audience actually had a good experience. The “Negative Eighth” comments all seem to be about length, making me wonder why people stayed, particularly since it sounds like there were breaks where people could have left or stayed in the lobby. Were people being rewarded with “continuing education units” of some sort? The “Negative Extreme” comments stand out so that you might either pay a lot of attention to them or completely ignore them.


One question that I’m left with is about how “personal” these kinds of evaluations are � and whether they should be.What do you see in these comments?

One response so far

Nov 29 2006

Analyzing audience feedback

Although Nancy White’s posting on “Feedback from Sydney LearnScope Event” was written at the end of her marathon in Australia, I’ve thought about it quite a few times since then, perhaps because of the strong feelings I had after our workshop at the Prato Conference during the same month. Capturing feedback in a wiki is an interesting example of how technology inherently changes our experience of face-to-face meetings just as it changes what’s possible. Reading what the more than 64 people said about the session with Nancy was a bit overwhelming so I made a note to myself to come back and take another look. One easy way to do it was to put the comments into a tool like http://tagcrowd.com/ (a tool that happened to hear of from Nancy). Here’s the output:

<!– #htmltagcloud{ font-family:’lucida grande’,trebuchet,’trebuchet ms’,verdana,arial,helvetica,sans-serif; line-height:2.4em; word-spacing:normal; letter-spacing:normal; text-decoration:none; text-transform:none; text-align:justify; text-indent:0ex; background-color:#fff; margin:0 0 2em 0; border:2px dotted #ddd; padding:2em}#htmltagcloud a:link{text-decoration:none}#htmltagcloud a:visited{text-decoration:none}#htmltagcloud a:hover{text-decoration:none;color:white;background-color:#05f}#htmltagcloud a:active{text-decoration:none;color:white;background-color:#03d}span.tagcloud0{font-size:1.0em;padding:0em;color:#ACC1F3;z-index:10;position:relative}span.tagcloud0 a{text-decoration:none; color:#ACC1F3}span.tagcloud1{font-size:1.4em;padding:0em;color:#ACC1F3;z-index:9;position:relative}span.tagcloud1 a{text-decoration:none;color:#ACC1F3}span.tagcloud2{font-size:1.8em;padding:0em;color:#86A0DC;z-index:8;position:relative}span.tagcloud2 a{text-decoration:none;color:#86A0DC}span.tagcloud3{font-size:2.2em;padding:0em;color:#86A0DC;z-index:7;position:relative}span.tagcloud3 a{text-decoration:none;color:#86A0DC}span.tagcloud4{font-size:2.6em;padding:0em;color:#607EC5;z-index:6;position:relative}span.tagcloud4 a{text-decoration:none;color:#607EC5}span.tagcloud5{font-size:3.0em;padding:0em;color:#607EC5;z-index:5;position:relative}span.tagcloud5 a{text-decoration:none;color:#607EC5}span.tagcloud6{font-size:3.3em;padding:0em;color:#4C6DB9;z-index:4;position:relative}span.tagcloud6 a{text-decoration:none;color:#4C6DB9}span.tagcloud7{font-size:3.6em;padding:0em;color:#395CAE;z-index:3;position:relative}span.tagcloud7 a{text-decoration:none;color:#395CAE}span.tagcloud8{font-size:3.9em;padding:0em;color:#264CA2;z-index:2;position:relative}span.tagcloud8 a{text-decoration:none;color:#264CA2}span.tagcloud9{font-size:4.2em;padding:0em;color:#133B97;z-index:1;position:relative}span.tagcloud9 a{text-decoration:none;color:#133B97}span.tagcloud10{font-size:4.5em;padding:0em;color:#002A8B;z-index:0;position:relative}span.tagcloud10 a{text-decoration:none;color:#002A8B}span.freq{font-size:10pt !important;color:#bbb}// –>

audience bit boring clearly concepts delivery delved didn e-learners eb email engaging enjoyed entertaining esp excellent experiences feel felt flow food globally informative inspiring interaction interesting kept key learnscope loved messages mobile nancy ok parts practical presented presenter provoking really relevant resources sms speaker split stimulating techno tensions think via

The comments have an inherent linear structure that tagcrowd misses and which seems important. There’s an implicit “good to bad” Likert scale behind much of what people wrote. I thought that I could make up a kind of 5-number summary that systematically throws away some of the noise. This approach comes from John Tukey’s stem-and-leaf plots (I got interested in computing because of a long-standing interest in exploratory data analysis and statistics). So I did a rough sort of all the comments from the most positive to the most negative, and picked out the middle, the extremes, the quartiles, and so on.

(The text editor in WordPress seems to have trunkated some of the text, so I’m going to post the second part separately.)

3 responses so far

Nov 23 2006

Ambiguity ok, according to some

Lilia Efimova and Nancy White were talking about some of the ideas in our technology for communities project and Lilia comments on how “community” seems like a problematic term to some people because it’s ambiguous – it gets used in many different ways and at different scales.

I like the argument that King and Frost make that ambiguity has many uses in practice and that dis-ambiguation is a bit over-rated (John Leslie King and Robert L. Frost, “Managing Distance over Time: The Evolution of Technologies of Dis/Ambiguation” in Hinds, Pamela J. and Sara Kiesler (eds), Distributed Work (Cambridge, MA: MIT Press, 2002), pp. 3-26. One important thing that communities do is manage ambiguity. I think that managing the nuances and ambiguities involved in a term like “community” is an important thing to do. When a community uses a term to point to something ambiguous it’s often a flag for one of those “it depends” conversations. Among other things, looking for familiar but ambiguous characteristics of something like “community” in the “blogosphere,” for example, is a useful thing to do.

No responses yet

Nov 20 2006

Online sclera: bringing the whites of their eyes ever closer

In his article about “How to use del.icio.us to foster collaboration” Shawn Callahan talks about social bookmarking as a way to distribute work across a team, community or network. I do notice myself using the bookmarks that people in my community store in del.icio.us as he describes (in the sense of copying their bookmarks, referring to them later, using the content in some way). But I think that a more common use for me is just to notice what people are thinking about and noticing as they surf and bookmark.

A recent article titled “Eyeing up the collaboration” from the November 2nd, 2006 issue of The Economist remarks that “being able to identify what someone else is looking at is thought to have been so important to humans that people evolved to have eyes surrounded by brilliant whites to assist with the process.” Humans are different from other species, which tend to have sclera (the “whites” around our irises) that are not white.

So, fast forward a few million years, and we are evolving ways of being aware of what others are looking at — noticing in which direction someone else is gazing. Presence indicators on Skype or Instant Messengers give us information that’s useful in related ways. When I was at the CIRN conference in Italy a month ago, my wife would “see me online” and wave immediately, since we’re not used to a 9-hour time difference. It seems to me that del.icio.us provides an equivalent resource: it allows us to follow someone else’s gaze.

No responses yet

Oct 31 2006

Practice makes perfect?

Published by under Technology

Very nice to see a photograph that I took a month ago doctored up in such a way as to make its meaning even clearer and more amusing.

I wonder whether we get better at multi-tasking? It certainly does have an adictive power. At the same time I find that focusing on one thing for a long time is very productive.

No responses yet

Oct 28 2006

Synchronizing daylight saving change

When you work across many time zones, you do time arithmetic in your head, counting on your fingers and very roughly if need be. Portugal is 8 hours ahead of Portland, etc. However, when it comes to publishing the time for a meeting that can involve people from all over the world, I always use http://www.timeanddate.com/ to check the time. Also, I always write invitations with UTC (or GMT/Zulu)-time, plus New York or San Francisco just to support the intuitive side. In fact, usually I go the extra mile and provide a link to a specific time for the meeting, so people can just click to see what time it will be for them.

How very nice that tonight is the night for changing daylight saving in such a synchronous fashion! Europe and the US fall back and Australia springs forward on the same date. Would that it were always so easy. When next spring comes around, it all happens on different dates, so your short-hand calculations don’t work for the last week in March / first week in April. And a year from now the US changes at a different time, so there will be a few “gotcha weeks” when planning meetings will be more difficult and your rules of thumb won’t work. How about a little global integration for such things?

Anyway, I find that adding and subtracting time zones is hard for me, but a good rule of thumb, supported by a widget like Foxclocks, to keep my intuitive mental clock as accurate as possible, seems to work pretty well. It’s still a lot of work. This quote from Jean Lave certainly applies very much to doing time-zone arithmetic: “Activity such as arithmetic problem solving does not take place in a vacuum, but rather, in a dialectical relationship with its setting.” Change the setting and problem-solving is likely to change, too.

We’re just wrapping up the September 2006 Foundations of Communities of Practice workshop and one of the things that seems to bend the minds of so many participants is actually working intimately with people in distant time zones. Part of the magic of the workshop is that all of a sudden the hassle of working across time zones is worth the effort. The October 12th, 2006 issue of The Economist has a survey of telecoms convergence that suggests that, “even if the traditional telephone is not quite dead yet, its business model certainly is: metered telephone calls whose cost depends on the length of the call and the distance covered are becoming an anachronism.” And yet time zones persist and they are not necessarily managed in a coordinated fashion (well, at least Indiana now conforms with the rest of the US, but what about coordination with the time in Mumbai or Moscow?). More and more of us have to deal with more and more time zone calculations, thanks in part to the falling cost of a phone call (and of synchronous interaction in general) but it’s not as easy as it should be.

2 responses so far

Oct 27 2006

Seeing if the talk will walk

Although the whole technology for communities project comes out of our practice with specific communities, during gestation and birthing it’s easy to start worrying about how our thinking may have gotten far removed from actual practice. It’s easy to wonder, “Maybe we’re just talking to each other… (Maybe we’re having too much fun.) Maybe it’s a bunch of words that don’t have traction or actually refer to anything that’s real.” Fortunately, to keep ourselves going we’ve gotten some feedback from a friend or two along the lines of “this is great stuff” and “makes sense, but you’ve got work to do bridging between deep theory and practice…” On another level, Nancy White is using some of the ideas about tensions that we’ve been talking about in her Australian speaking tour (see Melbourne Presentations where she reports on some interesting feedback).

I guess that’s the great thing about practice: it involves feedback. And feedback keeps a project like this going. (Now to find the time to work on Chapter 7!)

No responses yet

Oct 23 2006

Intuitive clocks

I used to use The Economist as a measure of how busy I was. Ever since I began reading it in 1989, I’ve always read it from front to back, one issue after another. I’ve chuckled at myself, comparing myself to the Joseph Conrad character who lived in the Congo and opened up an issue of The London Times every day from a bale of newspapers brought up the river by boat — very methodical, never peeking further down into the bale to find out what happened in the end. And I’ve always felt a bit guilty whenever I skip articles (or whole sections, like finance and economics, which I have found less interesting than the rest of it). So last night I finished an issue and today I am two issues behind.

But now I have a new intuitive clock to measure how “behind” I am: blog reading — and writing! What fun! (Although it might be productive to turn a clock around to give myself positive strokes about how much I’ve read and how much I’ve learned from reading, in daily life I find it a useful spur to focus on what I want to read or accomplish that I haven’t yet got to.)

I’ve been thinking a lot about intuitive clocks — how we set them up for ourselves and how we use them, either individually or in communities. I think that communities of practice come to have intuitive clocks that keep them in synch. Here are some questions on the subject that I’ve been thinking about:

  • During a conversation we all have a sense of time — that the conversation has more time to go or needs more time. How do we do that? As leaders, how do we read that clock precisely?
  • If the sense of time is shared — there’s some kind of shared clock, somehow — the conversation seems more satisfying because we’re “in synch.” How can we make the clock more visible to more participants?
  • Experienced community leaders develop a sense of pace — knowing when it’s time to push and when it’s not worth it. What clock are they using?
  • In very organized communities of practice, such as Toastmasters, an elaborate set of progress measures tells members and leaders “what time it is.”

One thing that seems clear is that articulating an important question is a way of offering an intuitive clock against which a community’s conversations and activities can be measured. Of course, the person who offers the question needs to have a certain amount of status and respect for the question to be considered and it has to be articulated in a way that makes sense to the community. And a good question is a good clock only for people who speak the same language — so it may not make much sense to people outside a given community.

No responses yet

Oct 09 2006

Being there

The view outside my window this week reminds me of the Klauder style of architecture at the University of Colorado in Boulder, where I worked as an intern after finishing a masters degree in architecture. The Prato Dialog is over and I’m now at the CIRN Conference, which is held within the electronic boundaries of Monash University. Among other things that means I can’t upload email. But, just like CU, it’s very beautiful here.

No responses yet

Oct 01 2006

On my way to the Prato Dialogue

I’m sitting in the Portland Airport enjoying the calm between storms: all the packing and harried traversal of to-do lists is behind me. The Portland Airport has free wi-fi, which seems very civilized and changes waiting for a late departure into a chance to slow down and do some more electronic catching up. During the nextt two weeks I have some work in Utrecht, some time with friends in The Netherlands and then two events in Italy: our Prato Dialog which is actually in Florence and the CIRN Conference which is actually in Prato. These two events follow the Ethnographic Praxis in Industry Conference (EPIC 2006) directly.

I’ve been thinking about Beth Kantor’s comments about blogging presentations at conferences and related (like, Capturing Collaborative Knowledge from Nonprofit Events: Your Event’s Tag ). There are several variants here:

  • For the Prato Dialog, we’re trying to make part of the event happen in public on the event’s blog and inviting participants from the Foundations Workshop to observe and participate.
  • For the EPIC conference, there was a lot of fascinating material, but I was too busy soaking it up to be able to take notes, much less publish them. I hope to write about the experience soon, including an interview with Brigitte Jordan.
  • For the CIRN conference, I will try to report what I hear here. I’m thinking they’ll have wi-fi, just like home and so all the challenges of reporting will be social and intellectual, not logistical!

powered by performancing firefox

No responses yet

« Newer Entries - Older Entries »