A Personal Paper-Reading Method

I was thinking about how I could read and understand academic publications better. I came up with the following skeleton for myself.

First, grab the following points (bolded are the must-learns):

  1. Topic: What are the relevant fields of this paper? (Later, how are the authors contributing to these fields?)
    • Audience: To whom are the authors speaking?
  2. Argument(s): What statements/arguments are the authors making?
  3. Research questions (RQs): What research questions do the authors ask?
  4. Methodology: How do the authors answer the research questions?
    • Theoretical lens: Have they used any theoretical lenses in answering RQs?
    • Model: Have they applied any models?
    • Experiment: Have they conducted any experiments?
  5. (Findings: What do they find to answer RQs? How do the findings relate back to RQs?)
  6. Contribution(s): What new/critical knowledge have the authors produced?
    • Novel theory: Have the authors produced novel or built upon existing theories? Theories can also be models, frameworks, etc.
    • (Theoretical) New perspective: Have the authors offered any new perspectives to understand existing phenomena?
    • (Theoretical) New methodology: Have the authors offered any new methodologies for answering a certain type of RQs/phenomena?
    • New intervention: Have the authors built any new, important, meaningful interventions?
    • (Empirical) New, important ways of using technology: Have the authors discovered any novel and critical ways of using technology? Usually, such empirical findings can feedback to new learnings to existing theories or even inspire new theories.

Second, while reading, I can ask these questions to myself (bolded & italicized means it’s only important for myself):

  • Significance: How do the authors argue the significance of the study?
  • RQs-Contribution: What is the relationship between RQs and contribution(s)?
  • Related work: How do the authors write Related Work (to support their arguments, maybe)? (How does this study build on past learnings?)
  • Actors: What are the actors (human or non-humans) involved in this paper?
  • Terminologies: What are the terms the authors relied on? Did they create any new terms?
  • Strength and weakness: In my opinion, what is the one single strength (and weakness) of the paper?
  • Future work: How can future work build on this study?
  • Personal belief/value: Is this work speak to my belief/value? Why or why not? (Do I want to become a researcher who does this kind of work?)

Research in Academia vs. in Industry: From My Experience as a UX Research Intern at Facebook

I spent the past three months at Facebook as a user experience researcher intern. I worked in the Facebook App Monetization org and did research with two different product teams. I interviewed Facebook users inside and outside of the US, and I also conducted usability tests of prototypes. This is my very first formal industry experience as a user researcher, so everything (literally, everything) was new to me. The journey has been more challenging than I expected, but it was definitely a great learning experience. I applied for this position last Fall, wanting to know what research is like in industry (or at Facebook, in my case). Luckily I got in, and more luckily, I got most of the answers I was looking for — if not all.

1. Objective. The focus of research in industry is to point out directions and offer recommendations for product design, development, and more. So it’s all about the product. Research in academia cares more about studying and exploring important questions in the field. If not necessary, it is often unlikely for research in academia to be related to any product.

Hence, the significance of research in industry is to help product teams make appropriate decisions that fit the company’s overall mission/goal/plan/interest. What about defining research questions? What about finding the right methodology and implement it? These are all important. However, in industry, oftentimes decisions are not made solely by the researcher but based on the entire product team’s discussion or the product’s current need/requirement. For instance, if a product team wants to know what a certain population will think about the product, then the researcher can only research this topic to find answers from users that will guide the team’s decision-making process.

In academia, research questions are usually determined by one or several principal investigators (PIs, who are commonly professors). Things can be researched no matter it’s a new topic that worths studying or a critical realm that worths being dug deeper (even though one might not get an answer). How do you know if a research topic is significant? It all depends on how the field looks at it. A research topic about how a mosquito’s flying will be affected if its wings are hit by raindrops sounds meaningless to our real lives. But if it’s meaningful for its field, then it could be studied. What is not important is whether it can lead a product. This is why many studies in academia look imaginative and less valuable to the reality. This is also why research in academia is way ahead into the future.

2. Significance. Research in the industry is unnecessary. Without researchers, product teams can still make decisions and keep developing products. The point of doing research (i.e. user research) is to establish the communication channel between users and product teams, so that product teams can make informed, wise, and better decisions. There are product teams who develop products freely without taking into consideration of researchers’ suggestions — but this is another story.

Because research is unnecessary, there are cases that researchers are neglected by the product team. Researchers thus have to work on helping the team to understand the value of research.

In comparison, academia is all about research. A research university (my institution is an example) pays all its attention to research. Professors and their teams have to do research and meet a certain criterion of research, especially before professors are tenured. So the importance of research in academia goes without saying.

3. Resource. Facebook as more than 40,000 people in total with hundreds of UX researchers, which is a lot in the industry. Naturally, more people get more resources. Examples include research labs built especially for conducting user research (better than the labs I have seen in academia) and a separate recruitment team for helping researcher recruit participants. Researchers also support each other by running training and creating discussion groups.

On the other hand, I think the resource in academia mainly comes from connections. Industry depends heavily on connections as well, but best researchers mostly stay in academia. Connections in academia can lead to knowing more researchers in the same or related fields and then creating more opportunities. In my opinion, the distinction between resource in industry and resource in academia lies in their types.

4. Who does the researcher represent? At work, UX researchers represent users in product teams; they speak for users and communicate users’ needs and difficulties to product team members. UX researchers represent their company in front of users.

Researchers in academia represent experts in the field. They represent themselves too.

5. Forms of outcomes. Researchers in industry present users’ needs and recommendations to product teams through presentation slides, charts, tables, reports, etc.

In academia, the major form of presentation is publications, such as conference papers or journal articles. Publications present research outcomes (e.g., new experiments, new methods, new algorithms, etc.)

6. Pressure. This is really all about my personal experiences… In industry, user research is only a small part of the product team and it is usually in the earlier stage of product development. Designers and engineers wait for research outcomes to move forward. This means researchers’ pressure often comes from their product team members and the progress timeline of the team. The quality of research is crucial too, but not as crucial as some other things, relatively speaking. That why people say “done is better than perfect.” It’s not okay to ask your colleagues to wait for you.

About pressure in academia… Can’t finish talking about this in one paragraph lol. I feel stress in academia comes not only from producing high-quality research that will truly contribute to the field (which includes a lot more smaller “sub-pressures”) but also pushing oneself to his or her intellectual and mental limits of learning and pursuing knowledge. Because of this, I will not recommend anybody doing a PhD degree without true passion and love of research…

7. Income. Of course, industry offers higher income… Much higher than academia, in fact.

There should be more about research in academia and industry, but I can only think about things listed above for now.

If you ask me which one I will choose… Actually, I still have no answer. But at least I can see more clearly the differences now since I experienced.

Dynamics

I went to CRC (Georgia Tech’s gym), drank iced milk tea, ate crispy chicken nuggets, and did grocery shopping this morning. This was all because I submitted a conference paper last night. I guess I need to celebrate, so I offer myself some treats.

I don’t have many experiences with writing conference papers. Previously, I had turned in two papers with myself being the first author and co-authored another paper. Yet writing a conference paper always teaches me new lessons. The lessons that I’m going to share might be cliché — smart people may learn this quickly, but I need to wait until I’m writing another conference paper. A bit dumb.

The thing I learned is that different papers have different dynamics. I know, this seems obvious — this seems obvious only when we are reading papers. It may not be the case when we are the ones who are writing. Why? I didn’t really feel this when I was writing my first two papers. I thought it was me who was writing, and since I was the author and I did not change, so I supposed the papers should look pretty similar in terms of style? Well, this is not true. The paper that I was writing recent days had an entirely distinct dynamics in a sense that it kept leading me, instead of being led by me. It refused being structured. The findings of this study were complicated. There were so many details and I didn’t know which ones I should pick. In fact, I couldn’t even find a proper way to tell a story for a couple of days. Findings were already messy, hence the Discussion section was even more difficult to write. I remember I had a hard time on the day I planned to write the Discussion section. Sitting in front of computer screen and hearing my heart bouncing anxiously, I told myself that no matter what I had to at least outline the section before I went to sleep. I jotted down whatever I thought of that might be useful in the section, and to my surprise, these random lines seemed to represent some commonalities that I could talk about. The paper had something it wanted to say. I was only its medium.

I want to record my feelings and thoughts because they are important to me as forms of reflection. This is also to remind myself that I learned a lot by going through the process regardless of the outcome.

Do Justice to Your Data

I met with my advisor two weeks before a major conference deadline. It was a Saturday. She had a coffee in her hand, and she was telling me how some other students had so many things to write in their paper:

“I guess this may due to culture. You know, both of them have so much to say. The finding section is already 9 pages and it’s only one-third of all the findings!”

I listened closely and replied that I never had so much to say. It in fact sounded pretty weird to me that people wrote so long. My advisor continued to say, data had to go through the researcher to be visible and I might affect those data. I was stunned. Yes, I know as a human being I’m not the kind of person who can talk on and on. I feel more comfortable when I can keep silence. However, for writing an academic paper, I know I have to write and I will certainly make sure that I offer as much data as it requires. But my advisor said:

“Yes I know. But still, your data will go through you. They are expressed in the way you express.”

I suddenly realized her point and yelled: “OMG I feel so bad about my data! They are suffering just because I don’t have much to say!” Since then I know I have to do justice for my data – these are great data which have value in themselves. They should not suffer because of me.

Interviews

I guess it’s fair for me to say that my June and July have all been spent on interviewing people: I did sixty-eight interviews in total. Some of them were short, some of them were long. I have never talked to so many people during any summer in the past twenty-five years. Interestingly, neither have I ever answered so many questions in two months.

I interviewed people who were old, who were young, who were men, who were women, who lived in rural villages, who grew up in urban cities. I asked them how they used their mobile phones, what they did with WeChat, how they played around with emojis and stickers, how they learned farming knowledge, and how they communicated with their children abroad. People answered my questions. Some of them answered in hasty, so I often had to repeat my question. Some of them were super talkative and energetic. They led the conversation, while I had to find my own way of asking them questions, if possible. Sometimes people were quite nervous when hearing the word “interview”, so I ensured them that there would be no provocative questions and they were all protected by the law in the U.S. (people laughed when they heard this). A common reaction that I got was that people felt they had nothing to offer: “I don’t know anything. I don’t really use smartphone/WeChat/emojis. I don’t know how to use them. You probably should interview that guy, he knows more than me.” When I said I would love to know why they didn’t use those things, they were usually more confused – anyway, why does one want to know why other people don’t know a certain thing?

I interviewed people in days and at nights. I visited them by walking, by motorcycle, by car, by bus, by subway. In rural villages and the small town, I simply went and visited without bringing anything other than my notebook and pen. Instead, people offered me watermelons to eat generously (because it was summer). Occasionally, they even invited me to lunch or dinner, and they were sincere. For urban citizens, I paid for their coffees, milk teas, juices, snacks, etc. Basically, I paid for their time so that they could answer my questions. In the city, it was not unusual for me to spend more than half an hour to get to the location for interviewing. However, in rural regions, all I needed to do was to walk or hop on a motorcycle – it wouldn’t take me more than ten minutes to greet someone in person. Because of the interviews in rural areas, I was able to do some exercise by walking. Because of the interviews in urban areas, I was able to enjoy various beverages.

Was I nervous? Was I afraid of anything as a first-time interviewer (an interviewer, how dubious!)? Certainly! While most of the time I was confident and well-prepared (in my opinion) to both ask and answer questions, I became infinitely nervous when people asked me this question: “So, what are you doing these interviews for?” There was a more intimidating version of this question: “What’s the meaning of these interviews?” I worked hard to try to calm myself down by blattering whatever I could think about at that moment: “… You know… I’m interested in how culture affects people’s use of technology and how technology affects people’s lives… I study how you use your mobile phone so that I can improve these phones…” People were confused. They twisted their brows and lost in thought, trying to translate my nonsense into human language. Later I became tired of explaining my “research objective” to people: “I need to write a paper. It’s my summer homework. That’s why I have to interview you.” People were more convinced by this answer: “Wow, you still have to write a paper? You study so hard! Why not just send out some questionnaires or get some data online? Take a break young girl!” My latter answer worked better, because people were more open when they knew this was for my homework, not some mysterious research projects.

I have to confess that I was tired of traveling around and talking to people from now and then. But, I always felt happy talking to people and listening to them sharing their thoughts and feelings with me. Here was a new person, somebody I didn’t really know about, yet he/she was willing to communicate with me. Sounded like magic. Each of them had an individual world with them. I was fortunate enough to visit sixty-eight different worlds in two months, and I was pretty satisfied that I now had a chance to understand the larger world through so many diverse lenses. For some of the worlds, if it was not because of the interviews, I would never be able to imagine them. Communication is indeed beautiful.

Reflection, on I2S Competition and groupwork

My team, Eating Right, won both Ideas Track Runner-Up and Best Poster awards in the 2016 Ideas to Serve (I2S) Competition held at the Scheller College of Business at Georgia Institute of Technology on April 8th. This competition is intended to encourage current GaTech students and recent alumni to think about how to create a better world through feasible and concrete ideas. During the night of the 8th, all the finalist teams presented their posters and gave a 1-min pitch to the audience. I was so happy to see that there were so many projects that cared about issues of global relevance, not only in the United States. Initially, I didn’t really expect us to win anything, since I didn’t know if people valued this kind of work. However, it left me feeling deeply encouraged, because people did care, and they were even willing to spend money on this.

Just to give you context: our team, Eating Right, aims at creating a mobile app that will provide valuable and trustworthy dietary information for diabetes affected households in India by presenting entertaining cooking show videos. Our project started in October 2015( as a collaboration between Georgia Tech and the Indian Institute of Management Ahmedabad (India) ) , and Jasmine started working on it then. After one month, I, a Human-Centered Computing PhD student, joined the team. Recently in March, two more Human-Computer Interaction students (Sam and Tanisha) came on board.While Jasmine and I focused on the research aspects, Sam and Tanisha brought design expertise to the team. Prof. Neha Kumar advised us from the onset and helped us communicate with collaborators in India. All of us, including our professor, are women.

What makes teamwork impactful in our case?

  1. People. All four of us have sufficient background in Human-Computer Interaction design and research, which makes it easier for us to communicate. Although we have similar backgrounds, we do hold diverse skill sets. Sam and Tanisha were originally from India so they can speak different Indian languages and draw on their understanding of Indian culture. Jasmine had done fieldwork in India, and could anticipate the challenges we might face later. I am a researcher who has read a lot about ICTD and have a deep understanding of how to approach this research using human-centered design. Together we are advised by Prof. Kumar, who is an expert in ICTD research. We are also supported by our collaborators in India, who offer first-hand data about users and valuable inputs for our research and design outcomes.
  2. Work ethic. I believe this is important in every field, not only in teamwork. Nevertheless, to keep a good work ethic can be really challenging when there are too many things going on at the same time, which is often the case for a graduate student. We face this challenge bravely and deal with it professionally. We rely on emails to communicate most of the time and all of us are very responsive – to emails, text messages, or phone calls. Because we are invested, the project is able to make steady progress. We stick to deadlines as best we can. If we cannot keep our promise, we let all team members know.
  3. Coherent understanding, all the time. We communicate in timely fashion to make sure that everybody is on the same page. This not only covers work assignments, but also high level objectives. For example, we do believe that user research can make a difference and design should be based on specific contexts and users’ needs. Communication sounds tedious and time-consuming sometimes, but it can foster understanding, which allows the entire team to work more effectively.

These bullet points above address some things that good teamwork should include, but they are not enough. What I find amazing in the Eating Right team is that we are all open to making mistakes and we learn through failure – the working environment we create is a safe learning environment, not a competitive one. When one of us says something that sounds not so right to others, others will ask questions and start a discussion. No one is offended. For instance, I’m not familiar with Indian culture so I have tons of questions (I know some of them must sound naive to Indians). My teammates are always willing to explain whatever detail I’m looking for. I try to do the same. I believe, because of this positive learning environment, people’s skills are leveraged, good work ethic is acknowledged and followed, and a coherent understanding can be nurtured.

Certainly, we are all motivated and passionate about this project from the bottom of our hearts.

I used to be a believer of individual work, but my experience in the Eating Right team changed my mind. When a team works well, productivity will be high, and everybody on the team can learn. It is amazing when teamwork presents its power, and I hope to experience more awesome teamwork in the future. Of course, to make this world a better place, you need a team, or maybe a larger team. 🙂