If I’d only known…
I am working on the Introduction to an edited volume on the nitty-gritty behind-the-scenes work involved in empirical social science research (to be published by The University of Michigan Press in 2008). While each chapter in the book gets into considerable detail about how to approach various types of projects (from sampling online populations to interviewing hard-to-access groups, from collecting biomarkers to compiling cross-national quantitative data sets), I want to address more general issues in the introductory chapter.
One of the topics I would like to discuss concerns larger-level lessons learned after conducting such projects. The motivation behind the entire volume is that unprecedented things happen no matter the quality and detail of preparation, but even issues that can be anticipated are rarely passed along to researchers new to a type of method. The volume tries to rectify this.
I am curious, what are your biggest lessons learned? If you had to pick one or two (or three or four) things you really wish you had known before you had embarked on a project, what are they? I am happy to hear about any type of issue from learning more about a collaborator’s qualifications or interests, to leaving more time for cleaning data, from type of back-up method to unprecedented issues with respondents. If you don’t feel comfortable posting here, please email me off-blog. Thanks!
June 7th, 2007 at 7:40 pm
I’m not sure if this is related to your chapter but when I was working on my thesis (which involved sampling bloggers), dealing with the IRB consumed A LOT of my time (which we didn’t anticipate). It doesn’t seem like IRBs understand the nature of public/private information on the Internet or Internet research. My small scale harmless study of public data turned into an inquisition (and even required full board review).
The network analysis part of my thesis required a lot of work to be done with free/shareware software, most of which wasn’t very user-friendly or well-documented. I also ended up writing a lot of ASP code – for data mining (from Technorati) and cleaning up the mined data. And things didn’t always go smoothly – the scripts often took hours to execute, failed, couldn’t connect to Technorati, databases got too big and slow, etc.
Had I known the amount hacking around I’d have to do to get it all done, I probably would have picked a different project/research questions (I tend to be more qualitative than quantitative). In general, there is a certain amount of tacit knowledge that isn’t very well documented with some of these methods (necessary steps are assumed or simply omitted). I was lucky that I was being mentored by someone who was an expert in the field and could steer me in the right direction when I got stuck.
Hope this helps — otherwise thanks for reading my venting! 🙂
June 7th, 2007 at 8:41 pm
Very good comments, Noor, thanks! You make several helpful points. Much of it relates to the issue of _everything_ always taking longer than you anticipate. But here we also have issues of depending on others (whether IRB staff or Web sites being accessible). I’ll definitely incorporate these pointers, thanks!
June 8th, 2007 at 7:50 am
When doing an online survey, don’t just pilot the instrument. Pretend you’re in a QA lab for a software company, and try to break it. This would have saved me from using the wrong question type on a survey.
Also, a general recommendation from dealing with hand-mined social network data for my thesis is to consider your intended analysis (skills, techniques, methods, etc) when determining how data is recorded. I saved myself a lot of misery by collecting data in the rawest possible flat-file format, because I already knew that I would end up using Perl scripts on it to turn it into a network input file (or two) but that I would also analyze the raw data directly.
I was also very lucky to have a mentor who was willing to invest the time to get me going with a number of the tools I needed to do the research–not just Perl scripting, but also LaTeX, R, GUESS, GraphViz, and quite a bit more. The learning curve just to get yourself started on data collection and analysis can be significant, but the experience has really improved my computing skills much more than I could have foreseen, and I wasn’t any slacker to begin with!
June 8th, 2007 at 8:02 am
Thanks, good points.
And yes, when I pretest students’ surveys and tools (or my own), I try to do unexpected things precisely to see how unanticipated actions are handled.
June 9th, 2007 at 7:26 am
OK this is really obvious but when doing interviews make sure to do pilots early – preferably before you start formulating research questions. Even if you think you know your interviewees’ subject well don’t assume you will start by asking the right questions.