Learning with Moocs 2015 - Case studies ii - Papers sessions

>> I wanted to welcome everyone to the second case study session. The case studies we're looking insights and lessons learned from actual MOOCs that have run and what we learned from them. So the first one that we're going to have comes from Davidson College. We have Kristen Eshleman. >> Eshleman. >> Eshleman, Dr. Erland Stevens and Dr. Ann Fox. Here you go. >> Hello. Okay, so we're going to you a little bit about some hybrid moves that we did at Davidson College. We are calling this a live and learn because we're really focused primarily on the residential component of our MOOC experiment. So why Davidson acts it's pretty much the same reason everyone else went into the MOOC game, but the main point -- can't hear me? The main point I want to make in all that is that we are really interested in how this might impact residential students at Davidson College. So we started with research in the on-campus hybrid mode first and why we decided to do a case study is because we're talking about 50 students. So we didn't have a huge number of students we were researching here, a small sample size. We have students who are really practiced in the art of reflection, so we knew we'd get really good qualitative data, so we decided to go with that approach. And this is the main research question we were asking and it's very broad on purpose, the reason we did this is we didn't know what we might learn from this. So this is the main question, how if at all, is the MOOC experiment impacting residential teaching and learning. And we partnered with George Latinos and Amy Collier to help us design this qualitative study, we were new to this. We had five questions that we wanted to cover as the secondary research questions and these are the three that we want to cover today. So the first two that you see up here are really pedagogical questions and our faculty who are with me are going to talk about their experiences there. The third one is focused more on the student experience and I'm going to talk about that really in the context of the larger implications for residential education. So with that I'll talk a little bit about the method. We did not have any selection bias in this, the students did not know they were going to be participating in a hybrid MOOC. So as far as they knew this was a straight up residential course. The study consisted of classroom observations, focus groups, a pre and post survey and interviews and these were primarily focused again on the student experience. And so with that I'm going to turn it over to Ann who will tell you about why she decided to do this. >> Thanks, so my name is Ann Fox and I'm an English professor at Davidson College, so I feel like a bit of an imposter here today which is great. I team teach a class called representations of ***-AIDS with a microbiology professor named Dave Wessner and that class came about in 2012, it came out of a visit that Dave did with his genetics class to an art exhibition that I had co-curated on disability, women and culture. And it got us to talking about the social context of illness and disease and that led to a further discussion about well how can we transform this into a sort of semester long experience for students. And so that gave rise to a course called representations of ***-AIDS in which we talk about how scientists and literary critics create knowledge about ***-AIDS in similar ways. And the class was really successfully, we've done three iterations of it and by the third iteration we were asking ourselves how can we take this beyond the 30 students that we meet with in the Davidson College classroom. And so that's why we decided to do a MOOC called presentation of ***-AIDS and that MOOC launched a year ago. And the residential class and the MOOC both discussed how artists and scientists sort of think about illness, disease, and specifically ***-AIDS. But the MOOC itself was seven weeks long and focused on a narrow range of topics, including how do artists and scientists create knowledge and ask the questions they do, what is the language around ***-AIDS, how do we think about imagery both in science and art around ***-AIDS, what about the personal experience of both the scientist and the person living with ***-AIDS and finally, why put the arts and sciences in conversation. And we involved our residential students at Davidson College very heavily in the MOOC. They were part of the production of the MOOC content. The Colgate students who presented so wonderfully in the previous session talked about fireside chats and we call them Charlie Rose discussions where the students were talking with us about topics in ***-AIDS. The students in the residential class learned the material a few weeks ahead of the MOOC students, the MOOC launched in October and so the residential students ended up leading discussion forums for the MOOC students on the edX platform. And the residential and MOOC students co-curated an online exhibition that you can look at, representingAIDS.org, to which they contributed what we called cultural artifacts or examples of representations of ***-AIDS that they had found in culture. And so what were the takeaways for me as an instructor of this MOOC. Well I think primarily bringing global voices and perspectives into our Davidson classroom. The experience for our students based in suburban North Carolina of hearing from people who lived with ***-AIDS since the early 80s, hearing from a doctor in India who is treating patients with ***-AIDS, hearing from a mother in upstate New York whose daughter is very ill with AIDS, it was something we could not replace. So these global perspectives and voices were really great. And this is a Google map of all the places where students were from in the MOOC who contributed to the representing AIDS.org exhibition. Some of the other takeaways, well things that I feel like worked really well for us it extended the impact of liberal arts right, this idea that complex problems involve multi-perspectival and multidisciplinary solutions. There were exciting opportunities to bring in outside experts. For example, Stephen R. Potty who is an AIDS researcher here at Columbia was interviewed for our MOOC and was part of that process. The residential students were able to facilitate discussions with their international colleagues and I think that's something that allowed them to be less US centric in their approach to ***-AIDS and their assumptions about it. And the students who took the MOOC and who helped facilitate discussions in the MOOC said that it enhanced their understanding of ***-AIDS. I think one of the things I've seen today that I really loved is this idea of compassion. How can MOOCs make us kinder and think about social justice and I feel like that was something that this MOOC allowed us to think about. And the MOOC curation is something that I've taken into other courses at Davidson. I am a disability studies scholar and so right now I'm building another curation based on the experience of representing ***-AIDS. Things that surprised David and I, the personal and very opinionated nature of the community and I mean opinionated in a really good way. I think the MOOC created an international community in which people sort of struggled, talked to each other very passionately about ideas. We heard in the post-lunch keynote that these communities are very respectful and I think we were afraid we'd have trolls, we'd have reactionary participants. No, it was a very cohesive supportive community. We took away readings in the MOOC, but the students who were involved in the residential class said that they still learned a lot. So it was interesting and Kristen is going to talk more about these paradoxes. And another paradox was even though our residential students were very nervous about facilitating discussions, they really valued that experience. So there was some interesting kind of contradictions and we found that the residential students frequently brought comments and questions that had arisen in the MOOC back into the classroom without any prompting from us. So it was a really rich and wonderful experience. My colleague Erland Stevens is going to talk to you about his experience in a very different kind of course, medicinal chemistry. >> Thank you Ann. So I had a little bit different experience from Ann in my course. When I found out I was going to have the opportunity to teach a course and it was -- oops, it was a course in medicinal chemistry. In that semester I was not teaching my residential course so, therefore, the MOOC was created really as a standalone course. We cut the material down to seven weeks to get just the bare essentials behind medicinal chemistry and I kicked and screamed when that decision was made, but in the end it really made for a much more solid course. Now I think the course went great, it was well received, but then I had a problem because then the next academic year I wanted to bring my MOOC into my course, but now my MOOC and my residential class did not match. So something had to give and in the end I decided to kind of bail on my residential course the way I had originally designed and completely change it so I can incorporate the MOOC. Oops, I keep going backwards, come back okay, here we go. So here's my redesigned residential class is chemistry 374. So the first seven weeks we really did a blended and flip model, you know, I think it's a pretty familiar method in which people bring in MOOC content. The students didn't use the MOOC by itself, but they had a small private online course on the edX platform. Now for the next seven weeks my course reverted to a traditional class. The content I had ejected and had to strip out of the MOOC became the lecture content for the second half of the course. Now also one thing we did was concurrent with the second half of the course, we restarted the MOOC online. So now my students have completed the MOOC content and presumably on some level they're experts and then the idea was that they would be discussion facilitators to the newly introduced online students. So that was the idea and, you know, sometimes things work the way you expect, sometimes they don't. So for the good, I feel at the residential students had much a higher sense of student agencies, you know, they could control lectures, they could watch them again if they wanted, they could watch them at double speed if they wanted. And I was talking too slowly, you know, so they had a lot more control over the lectures. They also had more control over the classroom, so since we had flipped the classroom the students -- we were trying to have discussions in the class about literature, we tried to bring in some social media. And so the students really, you know, sometimes those reverted to lectures because I mostly lecture in my classrooms, that's a bad word. But that's you know they at least had control and they knew I was welcome to having them derail my lectures if that's what they wanted to do. They certainly saw more real-world applications because we were absolutely in the literature more than we were in the old classes. Not so good I'd say the forum moderation did not work as well as I had hoped. Now some people have talked about in discussion boards students simply want answers in the discussion board and in my course one thing I found that there weren't enough of those simple questions. I need help on question five, someone rescue me and then my students would swoop in and save the day and answer question number five. That simply wasn't the case, I had cases of question number five that's an interesting answer, but what if we did this instead. And then all of a sudden my students were kind of tentative to try to intervene on that. And so then they kind of wigged out and it was like am I being graded on these interactions and there's nothing, you know, so all of a sudden different things kind of got in the way of the interaction that I was hoping to occur. So I could probably do a better job framing that for my students, but that didn't work out as well as I hoped. And one thing that did work was a final project in the course. So I had 18 students in my residential class and I broke them into groups of three, so we had six groups and each group was assigned a specific drug class and their task was to create a 20 minute video presentation on this drug class. So these were when I say presentation, these became PowerPoint presentations with voiceover. And they uploaded these presentations on YouTube and I told them the best presentations we're going to put into the MOOC and so that's going to be week eight of this seven week MOOC and then the students will create content for the MOOC. And the students did a fantastic job, I think the extra motivation of possibly having their work, not simply presented in class and kind of be a one-time deal, but that they have it presented in the MOOC, they would read the comments of students who had watched it in the MOOC. That made it a more valuable experience and so that was a huge success and certainly something I'll move ahead with in the future. So these are some of the big outcomes for the course that I taught on medicinal chemistry, we're heard from Ann about the ***-AIDS course, and now we'll come back to Kristen who will give us some institutional takeaways. >> Thanks [inaudible]. I think this one's a little better. So what are the larger takeaways for residential education? We saw a number of fanatic findings from this, but we wanted to point to two that we think are very interesting in how we're going to pursue the research further. And those two were really brought about by these paradoxes that they both referred to between the student expectations and the experiences in the course. And these are the two topics that interested us the most. So the concept of rigor and implicit bias. Rigor in the sense that both faculty and students in both courses all indicated across the board this was a less rigorous experience for them. So going through the MOOC I mean not necessarily the hybrid piece, but just the MOOC piece of the course was less rigorous and the implicit bias in all that is we think this is probably because our students are paying a lot of money to attend a private liberal arts college that's really privileging the face-to-face interactions in the classroom. And what does this mean about elitism and power and how our students view education. So we think the implicit bias there is in the mode of delivery and here's why. So in the med chem MOOC students would point to that self-directed part of their learning, these are problems sets they do outside of class as the most transformational part of their experience and they credited the MOOC with improving that part of their learning experience because it made it much more efficient. While at the same time and almost the same breath referring to the MOOC as being a lot less rigorous, so that was interesting to us. In the ***-AIDS course it's really centered around the discussion forum and in the first part of the class students ahead of it in the pre-survey were indicating they expected to spend about 6 to 10 hours per week working on this class. They ended up spending about 3 to 5 in the first half of the class and when the MOOC released that went way up to 6 to 10 hours per week. And this was all centered around being TAs in the discussion forum. So you can see this first quote here really nails that whole, you know, this is really a less rigorous experience, the online content very brief, it's very watered-down and then you'll see these additional quotes here and we have a lot more if these that say pretty much the same thing. That was the most transformational part of the course was being in that discussion forum, driving that conversation and talking with people beyond the classroom. So again compared to other Davidson classes this rigor was lacking. When we talked with both students and faculty afterwards and asked them about the outcomes from the course, they indicated the outcomes were pretty much the same as what they would expect in a regular face-to-face course. So this is interesting another paradox for us to dig deeper into. And ultimately, what I wanted to point out with this is these really aren't questions about the MOOC necessarily, but questions about what we're doing a residentially and this is why we're so interested in this MOOC experiment. It has given us the same as the Colgate students said, it's given us a critical lens on what we assume to be the value in our residential courses. So these are the questions we would like to think about exploring next and really we're just focusing on Davidson at this point, we're not looking at the MOOC. When is rigor really rigor and when is it just time management and when are you going to hit these points of diminishing returns. So at what point are you just really hammering them with a lot of content and a lot of reading, but it's really not impacting their learning and how can we begin to understand when that happens. How can we take advantage of online learning MOOCs, how can we maybe use these to reimagine the curriculum in new spaces. If we've got freed up time looking at rigor in a different way. What does this mean? So we really want to take the outcomes of this, dive a little bit deeper residentially and see how we could reimagine the liberal arts. And I'm at the end, so I'm going to briefly just say the biggest takeaway for us has been this whole move toward R and D and we are now kicking off a formal R and D process on our campus and we'll be experimenting in these ways going forward. And I only have a couple of minutes for questions, so I'm just going to end there. [ Applause ] >> Are there any questions? >> So when you give the MOOCs a second, time how do you plan to make them better? >> I've had a chance to repeat not the residential combination of the MOOC, but my med chem has run twice. And so one thing that came out of the first run was we had the opportunity to collaborate with Novartis, so Novartis had expressed an interest to collaborate with us. And so they are providing some forum discussion support and they've also given us access to some of their scientists and so we have interviews with their scientists interspersed to give real-life information. I think I've talked to you before Dr. Prichard about we did some pre-and post-testing in the last run and I think our pre and post-test is probably a more rigorous concept driven tests than it was before. So I think we're trying to improve some of our assessments as well. >> I would only add we have not rerun our MOOC yet, hopefully we'll do it next year. I think one of the key things that Dave and I sorry, want to do is sort of better prepare our students to be forum moderators. I think when we sort of entered in we said okay, this is a grand adventure we're kind of throwing you into the deep end of the pool. And now thinking more carefully about preparing them pedagogically as well, so I think that for us is a big difference. And there'll be others, but we haven't contemplated them yet. >> All right, thank you so much. [ Applause ] Greg. >> Okay, hi my name is Greg Bybee and I'm the director of learning solutions and products at NovoEd. And what that means is I'm responsible for the end learner experience, so both the platform piece of it, as well as the actual instructional design and delivery of our courses. What I want to do today is walk through about six different sort of mini studies that we've done across our over 200 MOOCs that we've run. But in conversations with folks before I realized it might be helpful to start with a quick background on NovoEd and specifically, what's different that leads to the following studies that are all about teamwork and collaboration. So in short, much like Mike today started by thinking about like what can online do better. We sort of thought about online learning as what works best offline and how can we bring that online. And you think back to the Columbia MOOC as well the virtual studio. We think that the best offline experiences are those that involve peer-to-peer collaboration, constructivist project-based learning, working together around a shared workspace in constructing their own learning. And so we're all focused on how do we bring that online. And so that's through our platform, so it has a lot of the similarities to future learn as you'll see up top you've got the making learning visible to itself and to the instructor and the learner. But we also make the community visible to itself and so you see who's active, what trends, what conversations are bubbling up. And then perhaps most importantly is we make it clear what user generated content is being created, so trending submissions, trending work that then people can vote on and provide feedback. Lots of discussions and also identity and reputation built in the platform. So the online learning platform is our key product and I don't call it a MOOC platform for a reason because it's used for small private courses, corporate courses, on-campus education, as well as MOOCs. But we're also different in the fact that we have a very instructional design team, it's actually larger than our engineering team that works with our partners if they like it to help them convert from offline to online, how to design peer-to-peer constructive pedagogy in an online format. So we power a lot of the world's best leadership development programs and that's where our sort of secret sauce really shines is around the softer skills communication, storytelling, creativity, design thinking. And these are a couple of our courses. So everything from music to negotiation to storytelling, design thinking, innovation. We really try to bring online what never could be done before. And kind of the key social elements that you'll see throughout the studies. So course discussions, that's where most social starts, but you've got to be discussing around something and so that is around projects, around a shared initiative that the entire community is working on. And then they share their work with the rest of the community, so that the feedback comes from the peers, not necessarily from the instructors, although it can also come from instructors, from mentors, from coaches, from community TAs, catalysts. But what we find is people care most about the peer-to-peer feedback. And then finally, engendering this idea fell to accountability, which we'll talk more about, the social obligation that comes from visibility among your peers. So first let's start with Y teams. So this is a good one to start with because Chuck Easley is really one of the inspirations behind the platform. We were founded back in 2012 out of Stanford when Chuck Easley wanted to teach his entrepreneurship class online. And he looked at the platforms and said, you know I don't know what quiz question can test are you an entrepreneur. And so it means [inaudible] worked with him and said, well how do you teach offline. He said, well we pull people into groups, we work together on projects, and then we share those projects and iterate on those. And so that's what the platform was built up on. And so he's run his class about a dozen times now on NovoEd and decided to start doing some experiments with it. So in this particular study he sort of ABC tested a class and divided the cohort of about 26,000 students into three groups. Those that worked individually, but also had the social features so still had discussions but worked independently on the project. A group that worked with a team on the project. And then a group that worked in a team on the project, but also added in peer-to-peer feedback and mentorship. They're not quite even because a number of the independent students had dropped out, so it's about 1,300, 1,400 in the team cohort and in the team with feedback cohort and then the remainder were in the independent. And what you see is of the 23,000 some that were in independent working alone, about 2% completed. And this is very similar to sort of a standard MOOC completion rate and this is based -- the denominators all enrolled students. So this is actually pretty standard. But among those who were in teams, 21% passed the course and then among those who had teams plus the peer-to-peer feedback had the entire suite of peer-to-peer tools, 44% finished. And that's of enrolled learners and finished actually means I think for this class it was like five of six assignments, but most importantly they had to do the final project, which is actually quite substantial. So here's the same data sort of framed a little more easily. So you see about 16 times higher completion by adding on the teamwork component. And that's seen across the variables, so he looked at number of sign-ins, number of messages, activity in the platform and very consistently across. You see here in this case it's five times more sign-ins for the people who are in teams. Yeah. [ Inaudible Comment ] This one he randomly assigned. I should say I'm not the [inaudible] on any of these, so I don't know all the details. So then that got us to ask well so why, why are the teams doing so much better than those who are working independently. And we worked with Bob Sutton and Huggy Rao from Stanford, they were doing this research independently and we collaborated to try to figure out what's driving this. And they have this concept of felt accountability, which they published a book on now. And their idea is that there's a social obligation among people within the team to perform their best. And so [inaudible] happens is just like in the in-person experience when you get in a small group to work some people are probably thinking I can use this time to check my e-mail or how good we're kind of grouped off now I can do my own thing. But all you need is one person in that small group to sort of say hey guys, so what do you think and then everyone around the table is like oh crap, you know, we're going to have to do this, we're going to have to hit the whiteboard and work together. And then 10 minutes later someone else jumps in and then more and more pile on and it sort of rolls up to having a really good experience and it's all because of this on peer-to-peer social obligation that you didn't want to be the one in the group who held back. And there's also a lot of sort of autonomy and relatedness connectedness really focused on the intrinsic motivation because teams are self-formed, we can do them algorithmically of course, but we prefer to have people form their own teams. And so what we see is that because someone has to create the team and has invite other people to join and vice versa someone who wants to join has to ask join the team, there's this obligation that is set up from the very beginning and that really drives high completion. And so they actually talked about NovoEd in their book as an example of how you can drive this felt accountability in an online environment. So let me start asking questions like okay well what do these teams look like and that's most of the rest of the studies are what makes for a good team. So the first one's not a study so much sort of a retroactive looking back at the data among eight different MOOCs we picked out that we thought had the cleanest data to work with, including actually Huggy's MOOC on scaling up, a couple of Stanford ones, an IDO acumen MOOC, two from Princeton and one from Maastricht. And Andrew on my team did this analysis. And if we look at what teams people prefer to form, we see the most work independently, so we still have that happen. The vast majority do still work independently, but of those who decide to work in teams the average team size is about four people. And then we see this across -- this is the average of those eight classes, I'm not saying this has been pretty true across all of our classes, including the private ones, four to maybe six is sort of the sweet spot. But then the obvious question is well what leads to better results. And so looking at completion and in this case defined by each course, so every instructor can set their own criteria. We looked at what percent of assignments did the members of the team complete. And so you sort of see this curve that the optimal team size is actually right around seven people. And I think this makes sense you have more people, you're more likely to get that catalyst, the one who's going to get started with it, the people who are going to pull you along. You have more accountability, more people watching you. But at some point when you get too many you have sort of a diffusion of responsibility or you become too anonymous once it becomes too large and so we do see it start to taper off at the end as well. So we wanted our [inaudible] even better and so we brought in Mlad [assumed spelling] who is a PhD student at University of Toronto. And we said okay, now we want to do some real rigorous data. So some of the stuff before that we did, we're not PhDs in this, we brought a PhD in computer science, engineering and mathematics to really study this and figure out what's going on and what works. And so he looked at 11 different courses ranging from 200 to 25,000 students and looked at okay, what are people's preferences for forming teams and then more importantly, what leads to better outcomes. So the first he looked at was age and what he found was that people prefer to work with people of the same age. So you see, you kind of tell the diagonal in the middle there people who are of one age choose to join teams with an average age that's roughly in the same range as they do. And the good thing is that it turns out that that results in a better performance. So all of these charts are roughly sort of the same styles. On the X axis you have the average distance, in some cases it's obvious, in this case it's years or we have kilometers and in some cases he's defined essentially entropy measures or ways to measure distance between variables. So that's the measure of dissimilarity. And the Y axis is sort of the cumulative density of how many people within this group had less than or equal to that distance. And so what you see is that actually the less the distance the more the success. So if you see the orange -- it is a little tricky to follow, the orange will show the most successful teams and then red less and blue less than that. And you can see that the orange are always up and to the left of the less successful teams and so age homophily is not just a preference, but it also results in higher completion. The same is true for distance. He looked at both latitude and longitude and the hypothesis here because latitude, if I get these right, latitude distance didn't matter, longitude it mattered a lot and we think this really came down to the time zones that people found it preferable to work with others in the same time zone. And just like in the other -- those with a less latitude distance or longitude difference tended to work better and had a higher completion rate. Next he looked at educational level, so in our profiles we ask what's the highest level of education you've completed and the same thing here, teams prefer to work with other teams with members of similar education levels. And thankfully, that also results in higher completion rates. So by now you're hopefully asking well what about diversity, you know, I thought diversity is going to lead to better team outcomes and so that's where we do start to see that. So we also ask about experience, we ask for company, we ask for industry and some courses actually ask for, you know, what are you interested in working on. And here thankfully we see the result we would like to see, which is teams actually prefer a diverse set of skills and a diverse set of skills results in a higher completion rate. So this was the one that made us all very happy to see that it is true that a diverse set of skills results in a better experience, not necessarily a diverse set of demographic characteristics. Gender, you know, there's not a good chart on this one out, essentially there was no statistical difference in terms of team formation, so there's no homophily there. But our data is a little different than future learn, we're about two thirds male. It varies by course, but something we're definitely trying to work on is to increase the gender balance. So in summary because I think it's a little tricky to follow. The characteristics age, location, education, gender and skillset or what we found to be most effective were similar age, similar location, similar education level, but diverse gender and diverse skillset. And fortunately that was people's self-selected team preferences as well. So that's sort of the first piece of forming a good team then is what's the right team size, seven. What's the right composition, a diverse set of skills, diverse gender, but the rest of the variables similarity might be better for collaboration. So then we thought of well how can we scaffold teamwork, what can we do to make the teams collaborate together. So this one was a study done by Allison and Drew, Drew's on my team and Allison's at the Presidio Institute and they're running a private beta of this class, Introduction to Cross Sector Leadership Building Teams. And while the class is on teamwork, let's actually do some experiments around teams. And so for this one they also A B tested randomly assigned between two completely different courses and the courses were identical except for the instructions that were given. Specifically, one group, I think it was group one, received extra guidelines around how to do team meetings. So this one said well, first we recommend assigning roles so there's a lot of science around in teams if you assign people roles they'll naturally gravitate and actually do the roles, so timekeeper, you know contrarian, and so forth. So recommends roles, recommends an agenda for the meeting and how to go through, and also to take minutes and so forth. And this was a small study so it's not statistically significant, but some of you may want to expand on. But the results, you know, we like them because they sort of matched our hypothesis. Was that quality of teamwork as self-reported by the teams was higher than the ones that were selected into the scaffolded group with the instructions and if you look at the number of assignments they complete an average. Out of 10 assignments the groups with the scaffolding completed eight and those without completed seven. So this is statistically significant, but an area for additional work and at least directionally promising. So the last one I want to show is one that's sort of counterintuitive and this is also one that was sort of recently done and is something we've been experimenting with. And this was done in one of our Princeton classes and so this is the, which course was this, making government work in hard places. And the course had a set of, you know, lecture pages, videos, team assignments, quizzes, just like all the courses, but what they decided to do was in every other one they decided to offer topic suggestions. So if maybe you wanted to write about international loans you could pick, you know, one in Greece so you can look at different topics so something that people would find more interest in. And the hypothesis was that the paths where you had to pick one path there was only one lecture page and only one quiz would have lower completion than those where we gave you three options where you could actually pick a topic area that interested you, work on a project, you could pick your project a little more specifically, and then the quiz and the other assessments were aligned to your selection. And we did not see this. The blue there are submissions, which what we actually saw is that we don't know if people were confused or they got excited, but the ones that had multiple options people actually submitted multiple times. So they would go through it several times, once for each of the options. But the actual number of learners participating followed our sort of traditional pattern of a decline slightly after each assignment and we didn't see any pickup for the ones that had the option. So this one was a little counterintuitive and something that we're going to be testing, we want to try out more and there was no random assignment for this one, this one we just sort of tested on-the-fly, but definitely an area we want to explore a little more. So with that I'll pause and answer any questions and I'll say I have for about half of the these I have some case studies with me with more details, so feel free afterwards we'd love to continue the conversation, I can give you handouts with details as well. >> Great, thank you. We have time for one quick question [applause]. >> Maybe this isn't fair because I know NovoEd, but I know that you can also fire team members, if you're a team lead you can have your team vote. Have you done any studies on what does that do if you are someone that's been from a team, do you stay in the course, does it make you more productive because you don't want to be fired from a future team? I'm very curious because we want to do that in the Sherlock move, but we're a little hesitant to pull the trigger on it. >> Yeah, good question. Not that I'm aware of, that is definitely one of the trickiest areas to manager is when there's discord among a team. And so typically we like someone to just choose to create their own teams. So most of our teams are dynamic which does make some analysis tough, you can come and go from teams. But we haven't actually done a study on that. That's a good idea, you're volunteering. Nice. >> Yeah, use us, we want to try it. >> Thank you. Cool. I'm out of time? >> Yes. >> Cool. >> Thank you, thank you so much. >> Thank you very much. >> Thank you. >> I appreciate it. >> This is David Pritchard from MIT, thank you. >> Yes, so good afternoon. So really this is the work of my group, but [inaudible] is in Turkey so I'm going to take credit for it. No seriously, okay so what are study habits and why should we study them. Let's start with why you should study them. Okay, so what is that causes or is responsible for, is correlated with performance besides the ability of a student. Okay it turns out -- this is an interesting question and it turns out that there are a number of studies that show that probably after GPA study habits are the best predictor and it depends on which study habits we're talking about of student performance. Sorry, I want to get my stopwatch going so I'll stop myself. Okay and another thing is it's very hard to change somebody's IQ. So if you find that better IQ students learn better well so what, I mean. But if you find that students with poor study habits do badly, then maybe you could help them change the study habits and maybe that will help them do better. Okay so what are study habits? Well actually they're defined operationally by a series of standard surveys which I've listed here and these are the surveys that also have some research associating the response of the survey with academic success. And on these surveys there are 246 questions in total, which we didn't want ask that many, surprisingly enough. And so we boiled our things down to four categories with a total of 17 questions. And then some of those questions are not written appropriately for online environment, so we changed the wording a little bit. Okay, so what are these four categories? The first one is concentration and you notice one of these questions is black and the other one is red. If you answer the black one positively that is I study in a place with few distractions that adds to your concentration score. If you answer the red one positively yes, I have the radio on, then that subtracts from it. So it's a balanced survey you don't is a balanced survey, you don't know when you look at a question which way you're supposed to answer it if you're trying to gain the system. And so the categories include concentration and time management. I organize my work or I procrastinate basically, but that's me. And then study methods, I break large projects into smaller ones or well it's important to memorize, so I memorize stuff even if I don't understand it, that's a red one. Okay and then I think the big one which is maybe hardest of all to measure and that is motivation. I try to do well in things or I enjoy the challenge and so on. Okay and I have to give a nod to Ryan Baker whom I don't see in the room here, but he's done some interesting work in trying to infer some of these things in an online environment and sometimes by watching students working in the online environment. And then correlating it later with what they actually did. Okay, so the sample that we used was the MOOC that we've taught this -- oh wait a minute is that right? No, it's the one we taught in the spring, okay well that's all right. And it's a derivative of an earlier MOOC we've taught going back not only through edX, but also through LON-CAPA if you know about that. LON-CAPA is an open source platform which is one of the few success stories of sharing resources broadly, it has 140,000 questions that have been run. Okay and so you have to measure the ability of the students and we did that first by restricting -- oh yeah, so the size of the course was 9,000, 191 certificates and a good response from them and a significant response from people who were non-earners of certificates. But all those that we included in the dataset that we analyzed that had attempted more than 30% of the 1,600 items in our courses got a lot of work in our course. And we've analyzed the ability using both item response theory and classical test theory. Okay, now so I'm an experimental physicist and I never believe the apparatus, I never believe the meter unless it's got some checks that I can do. So how can we check this, well first thing is that we just noticed that good students generally possess more than one of these characteristics. So not surprisingly, there are quite a few positive or significant or very significant correlations among these variables. And second thing is that in general, older people get better habits and in fact, we see a good correlation with age. So this is a little bit reassuring that by cutting down to only four questions in a category we nevertheless are still getting some consistent results. Okay, now what's kind of interesting is that we noticed a lot of correlations of our SHS, that's study habits survey, with the motivations that edX collects for why people are taking the MOOC. And well one of the more dramatic ones is if you look down APC that means the reason I'm taking this MOOC is to prepare for the APC course and it turns out those have some very, very strongly negative correlations with skill at the study habits. So why should that be? Well first of all these people are pretty much guaranteed to be young because it's only high school people who are going to take the AP course. And maybe even it's the weaker ones those, but I'm not sure about that but I certainly about the age. And it's interesting and for those students who are going to take the test obviously this is very valuable to their future because they want to pass the test. So that's probably the future correlates with the APC for that reason. Okay and as I said before, there's a there's a big correlation between students having good study habits and students being born earlier. That is to say that they're older now. Okay, so the big payoff is how do study habits correlate with academic performance. And there it is, as far as we can see in this case the red indicates statistical significance at the 5% level. So out of 16 different correlations the learning habits, oh I hope that we have a pointer -- so the learning habits here on the, oops is that working, no? Yeah, so. I guess you can see it, okay so learning habits down here and across the bottom, different measures of academic success, the average points the percentage corrects the IRT ability and the grade that they got from the course, we see only 1 out of 16 at the .05 level. Well, how many would you expect, one? Okay, so basically there is effectively no significant correlation between how the students filled out the study habits and how they did academically. But if I average everything together, if I just say oh, I don't care about the particular study habits if I just care about the average that they got, you know, for all four study habits and then it's barely significant at about .05, maybe .02 level. Okay, so that was a disappointment, I mean all these other studies are really showing a lot of correlation between the surveys and performance and we're not seeing that. It may be one more sign of having a very diverse group in many respects in the MOOC. Okay, so now I want to switch gears and talk about operationalized habits or what I mean by that is so you can ask somebody I procrastinate a lot. Well if you have a whole bunch of log data from the MOOC, what does procrastinate mean? So you have to decide what procrastinate means. It might mean that you do it in the last 4 hours before school or 8 hours or 12 hours or 2 days okay, you know, pick your number right you have to -- or maybe you're going to use some [inaudible]. So you have to operationalize these concepts and then you can move from surveys to observations of study habits. So the one we took is in the promptness of submitting homework, which we measured in two different ways. One of these is the obvious one, well actually number two is the obvious one, how long between when you submitted this thing or when you had gotten to 60% submissions, 60% is passing in our MOOC, you don't have to do all those 1,300 items you can just do 700 of them. And then it turns out that we sometimes release two or three weeks at a time and sometimes we just release one week. So another thing you can do is you can just say, well *** compulsive are the students, how long can they not -- what drives them to, right after you put out two or three weeks of assignments they'll go do them all. Okay, so that's the time [inaudible]. So these things are slightly different, but they're basically on the idea of do I do things right away or do I wait. And then the next question is are we measuring anything that is consistent across the course okay. I mean maybe week to week people have different obligations and sometimes they do it early, sometimes they do it late. Well it turns out that both of these variables the time after release and the time before deadline have a correlation of about .85 from week to week. In other words, however they behaved in week 3 they're going to behave the same way in week 4, however they behaved in week 11 they're going to behave the same way in week 12. And then if we took 2 weeks that were 10 weeks apart, you know, like the first assignment or the second assignment and the 10th or the 11th, then that correlation dropped to .65, but it's still a very significant correlation. So students show stable habits across the course. Now then the next question is, does this have any influence on how well they do. And what we found was that the earlier submitters did much better and it didn't matter what time interval we picked, we could pick four days, sorry 48 hours, 24 hours, 12 hours or 4 hours, all those time intervals. And then we would just say okay you're procrastinator if you procrastinated until four hours before the deadline or you're a procrastinator if you waited after you had two weeks to do it or three weeks to do it, actually we have a minimum of three weeks available, if you waited until the last day to do it you're a procrastinator or if you waited until the last two days. Okay, so it turns out it doesn't matter very much which of these criteria you use to operationalize the concept of waiting until the last minute, but we typically see that those who do it before the last minute get it 35% correct and those who wait until the last minute get 25% correct. And we're measuring they're correct the first time. There's also in the MOOC a record to show that they make multiple submissions and there are more sophisticated ways of judging ability, IRT ability based on multiple attempts, but we didn't use that more sophisticated approach. And this is a very substantial result, it has a P value of about .003, but it's also quite different sets depending on which hour limit we take. So that's the first thing. And now you say, well now think about this, you have operationalized the concept of promptness or procrastination and you have survey results that show the same thing, but the survey results don't seem to correlate with the academic performance and the observations, the operationalizations of these concepts do correlate. Okay, now I have to confess that this is a work in progress and I can't answer that question partly one of our collaborators who was supposed to pick up the ball when [inaudible] left to go back to Turkey has had some serious health issues and hasn't been able to work on it, so we don't know. I should say one other thing, our students are conscientious. In this thing, of course, looking at the different time intervals we would find a different percentage of the students who had done the homework early and 61% submitted 60% of homework assignment number one before the 48 hours, before two days, and it 81% had submitted it before 24 hours and those numbers went down a little bit with later assignments as you might expect, but not all that much. Okay, so I just want to have a summary here of what we found so far. We've surveyed these motivation, concentration, time and study habits and the surveys are consistent and the primary result is that older students do much better and I put older experienced students because it also was very significant if students had taken several previous online courses not past just taken. That also correlated with their having better study habits on the survey. And the study habits survey results do not correlate with the scores, but when we observed promptness, operationalized it and observed it, we saw a strong correlation with performance. Although we didn't get any with the -- so this is our current big question and maybe I guess I have about well, I think I'm right on time right? [ Inaudible Comment ] Yeah, I just have one more slide. So this is as I said a work in progress. We have a number of questions in the survey, complete tasks on time, organize the work, work regularly, break work into blocks under two hours or I procrastinate basically. And then some on the tutorial sessions, you know, I prefer just to lay back or I actually participate and we'll see if there's a correlation between how students answer these questions and what they actually do. Finally, next we should look at how these habits go with learning and not just ability or score. That is to say with learning we have enough resolution using item response theory and because we have so many items that every week we get a fairly less than a standard deviation measurement of the student's skill we can fit week to week learning and see that week to week change in skill, which is learning. And finally, I just read a paper by Dunlosky about a thing called learning techniques. A learning technique is something like I use yellow to highlight the reading, which turns not to be very effective at all or I try to make up and take test questions which is effective. >> Okay, thank you, we've got to call that end. Thank you so much that was fantastic. I can definitely relate to the topic. All right, we have our last presenter Michael Cennamo from Columbia. I'm sorry. >> Do we have time for questions? >> I'm sorry, we don't because we have another presentation. Okay, so I can take this out and you can figure out how to [inaudible]. >> Hi everyone, I am so happy to be here talking with you today. I'm Michael Cennamo, I'm an educational technologist here at the Columbia Center for Teaching and Learning. I'm also a doctoral student and instructor here at Teacher's College, so this is a real honor for me. At the center I've had this awesome experience -- opportunity of working on blended courses and online courses. Also, these massive courses and really small courses, so I've been able to sort of go between those two worlds and think about what influences what and take lessons from both and that's really why I'm here today to talk about building MOOCs and building small blended classes and what I learned from both of them. Oops, here we go. So I've worked on several MOOCs here at Columbia in the Coursera system, I've designed them and maintained them, big data in education, virology one and two sustainable development. I've also worked on the four edX courses, four so far, we're new to edX, but now we have both edX and Coursera. The three-part Civil War and Reconstruction with Eric Foner, as well as Ryan Baker's big data in education, we moved it over from Coursera and now it's in both platforms. Nelson in both platforms. But I've also worked with lots and lots of faculty here at Columbia on much smaller classes thinking about how to blend their class and use technology effectively. And that's how I met Professor Brent Stockwell, he's an instructor here, undergraduate biochemistry course and together we redesigned his course and that's what we're here to talk about today. I love this book, The Power of Two, we sort of came together two different people, two different ideas and energies and we complemented each other. We formed a bit of a relationship, it's in its fourth semester, we made a lot of mistakes, a lot of iterations and we have a lot of things that I think would be helpful to other people who want to use technology in a classroom or in a MOOC. So before I talk about MOOCs let me just talk through what Brent and I did together for his online class. We started with this idea that the idea of blended learning that if we can integrate the best of online and the best things that he can do face-to-face it would create something more powerful than either of those that was our starting point. And we then looked at blended learning and we whittled it down to these five considerations that we followed pretty carefully. Backward design, we started with his goals, what's your goal, Brent's goal was so that his students can take what he was lecturing on and be able to problem solve real-world problems and tangential problems, not just the problems based on the material which they couldn't do very well when he was just straight lecturing. We looked at the strengths and limitations of face-to-face learning, as well as online and we figured out what's better for both. What could Brent do better than technology could ever do, what can technology do better or faster than Brent can do and how do we put those together. We tried to make it student centered even though Brent was still the organizer, orchestrator of it. We made it so the students were more responsible for their own learning. We reimagined how we could use assessment in tools and only at the end did we come up with any or think about technology. So we came up with these sequence of events and by the way these considerations really helped me thinking about MOOCs because it really sifted out what technology can really do better or really do well. So we came up with these sequence of events and it's probably difficult to look at or visualize this way, so let me show it to you in a couple different lenses. There were things we did online, there were things we did face-to-face every week, but also there were things that we did prior to class like showing one of these videos that Brent made himself and a quiz that Brent constructed himself. The day of we had a deeper presentation and he also used polling and he had problem-based learning and group learning happening in the class, time that he never had before because of the videos that he created. And the day after uploaded solutions, gave feedback, allowed students a space to do more problem-based learning. So we did this iteratively twice a week for 15 weeks. It was written about, we wrote it up in a couple of journals and now Brent and his wife have conducted a survey testing whether the videos and the group learning was helpful for learning. So that's where we are. Now for the MOOCs. So what did I learn from MOOC video creation that I could bring to campus and help faculty like Brent. First I'd like to just give a thank you to the video team here that I worked with, the CTL because they really taught me a lot about what a video should look like and feel like. So Jose and Michael and Stephanie, thank you. But they really taught me through that Foner MOOC that, you know, there's a lot of power in angles and zooms, there's a lot of power in the ability to have multi-cameras, but also using captioning. I didn't realize how many people need those captions and download them and use them, so that was very important. I learned the importance of integrating multimedia, a lot of this multimedia was not in the course, I'm sorry I forgot this was Professor Foner's last year of teaching this course here after 30 years here. So we videotaped it and turned it into this MOOC, but his students didn't have access to a lot of this multimedia. Professor Foner and Jose curated all these materials and added them into the video. So here we have video lectures that actually are a bit better than what was in the course. I also saw the power of teaching in different ways, so teaching through interview or dialogue, teaching through discussions. So all this and I also started to feel that the synchronous chats got a little less daunting after we did a few, it was hard at first having these synchronous Google hangouts, but we got used to it. And I also learned about the power of direction or sort of redirection with some people who built those MOOCs and how to focus people's attention. So I took all that that I've learned in the MOOCs and I sort of showed Brent how to use that and do those and he did them very well. He got really used to it and he built these great videos for his class. And what I brought from the MOOC space was all the things that we've been talking about for the last day right, they should be sequenced okay, they should be chunked down to a certain time, you should annotate them, you should zoom in, you should use multimedia when you can. We sort of made all those and built them into the videos and he got really good at it. And the he embedded into his LMS easily access and then he gave a short quiz for just a little credit to make sure that his students were watching them and he could tell that they were from his free YouTube account. So this all worked great, those videos changed his class and they looked great. But now the next step that we're thinking about is are those videos really good learning tools, do they really work, are students learning biochemistry or anything else, physics, math through these sort of lecture videos. If not, why not. So that's what really got me started on the next set of R and D. How do we make videos in a MOOC, what's the next step. So by thinking about how learning works you see that researchers really put a large amount of emphasis on prior knowledge right. So that really got me thinking about this whole other field of misconceptions and students having misconceptions and maybe this is the problem that people have when they're watching these videos right. So I don't have a lot of time to go through this study, but it's a great study. Let me run through a really quickly. This has to do with videos that can expose the student to a misconception and how much they learn. Four videos all on a physics topic, video one called exposition. Just a great video like the ones that Brent made clear, concise, multimedia principles, not too long, a good video. Extended exposition is that video with a little bit of more interesting information added in, in case interest is a key factor. Refutation is a video where in the video a student shows that they have a misconception about some rule or law and it's refuted in the video. And then dialogue, so a student and teacher discuss that misconception in the video, resolve it and that's a conversation that happens in the video. So it turns out that this is where most of the learning happens and students usually when they watch a video and they think they know something that they don't, they watch the video at a lower level of attention. They say, I know this, I know this, I know this is and if they don't there's a problem there. They don't know what they don't know and what happens is with these videos is you could get a cycle then of faulty foundational information and then the new material doesn't really make sense and what you're not getting -- what's happening then is people are moving toward that I can't do this mode when the purpose of the video was to sort of broaden out that middle area, that zone of proximal development just a little bit to get that student to learn a little more. So it got me really thinking that maybe the next step for these videos is to expose students to misconceptions, especially in the science courses. Now one way to do that would be and we're thinking about this, this is R and D, should we have at the beginning of a video a small dialogue with -- this isn't Brent this is an example, with Brent and a student having that misconception exposed. And that's what we're probably going to do in this semester to come add this little bit of a skip at the beginning a video. But Brent also has this other way of getting at misconceptions in his class and he does that through polling. So now he's freed up time to do live polling and he has this happening in his class every day and he can see the misconceptions happening in his class and this has led to great discussions, so I was thinking well how do I then take the power of polling and bring it into the MOOC. I brought the lessons from the MOOC into his class, how do I take what he's doing in polling and move it out into a MOOC. So in the Eric Foner MOOC what I did was every week I started the week with an introductory opinion poll and I let the students answer it and see the results and everyone else's results. But we didn't stop there, at the end of the week we gave the same opinion poll and then we let students see and compare the answers at the beginning of the week and the end of the week and that led to these great discussions right inside of edX. Why did your opinion change, if it changed why, if not why, where in the video lectures did Professor Foner get you to change your mind. So this was a great way for us to think about polling in a MOOC not just to get one answer, but to get two answers and hopefully compare and see a change, maybe a change in learning, a change in opinion and let students see that. So this is some of the discussions that happened right in that polling area, which I thought was great. Another thing that I think I took a lesson from Brent's course was in supplemental materials. He freed up so much time in his class that he was able to bring in these extra materials, these peer-reviewed journal articles and had discussions about them and students really loved this. So I thought wow, what a cool thing to be able to bring in these supplemental things, how do I bring that to the MOOC. So that led to a relationship with the history librarians here at Columbia, Ty Jones, May Freeman, and they lovingly curated dozens and dozens of primary sources and we put them into the edX MOOCs and then had discussions around them, let me show you that real quick. This is just a page of all the primary sources in one page, it was awesome. So these are things that and they also created questions if you're a history teacher how would you use these primary sources and you could click on any of these and zoom in on them and analyze them. So it was just a great way to have a discussion and then discussion boards went around these primary sources every week and students were able to sort of go through this history class, through the analysis of a lot of documents and deeds and images and contracts and things like that. So this was something that Professor Foner didn't have time to do and it shows that there's some things you can do in a MOOC where you have the flexibility, the strength, the storage, all those things were available in the MOOC that we didn't have in the course. We also curated, which we didn't have time to do in the actual live class, we also curated maps and an image gallery every week. So it made me realize these are kind of things that you can do in a MOOC that you couldn't do -- really didn't have time to do in face-to-face time. So we brought this lesson into the MOOC. Another lesson that we got, sorry had to do with assessment and I know assessment is a very murky hard thing to deal with and grapple within MOOCs. Basically we still in Brent's class and in Professor Foner's class, we still use the basic multiple-choice question, but we were thinking and this is more R and D for us, is there a space between the multiple-choice assessment and say the peer assessment. Nothing wrong with either, but is there something between them because to have automation, reliability and the ability to really assess learning all in one is kind of hard. So what we're going to do next fall when we launch a teacher development MOOC here at Teachers College, we're going to try the situational judgment tests. Has anyone heard of these situational judgment, okay. They're, you know, around in the corporate world, in the workplace, sometimes in medical and it's these sort of scenarios that can be done written, text, audio, video and then there's a nuance complicated set of responses. So I was thinking that this might be the middle ground, this might be the answer to an assessment that can get closer to actually assessing learning. You could have them in videos, you could make video scenarios, audio scenarios, text of a situation, a case, a dilemma, a problem and then have those responses. Is this a lot more work, yes? Does it take more resources, yes because you have to build the case, you have to film it, you have to design the nuance, the responses. But is it worth it. What do we need that can maybe be that middle assessment that's lacking in the MOOCs you know? These are just ideas that I came up with where you could imagine watching a video of a certain scenario and then answering some questions about it. So we're going to try this in the next year in our Teachers College, teacher development MOOC and see how it goes. And then lastly, problem-based learning. Brent had freed up enough time to have this problem-based learning going on in his class and he was able to free up 20 to 25 minutes. I know that m might not sound like a lot, but to Brent that was a transforming amount of minutes. So he had these groups working, but he also used this tool called Piazza, has anyone not heard of Piazza? Oh everyone has heard of it, okay great. So he had his students and this goes back to the student center part, he had his students helping each other and I thought well how about using this in a MOOC. So I wasn't sure it was strong enough to handle working in a MOOC, but I contacted Piazza and they told me that yes it is, it's in some Coursera MOOCs and it's in edX MOOCs. So this is another thing that we're going to try to get a little closer to that problem-based learning and students helping each other. And we're going to do that in an upcoming MOOC too. So I think that Brent and I by working together just scratched the surface on what we can do in a blended class, what we can bring over to the MOOC space, and what we can take from a MOOC and bring into his campus class. So if you had any other ideas about or you see things from a different angle that we didn't look at like why don't you try to do this in a MOOC or why don't you take this and bring it into the class, we'd love to hear. And if you have any ideas or want to contact us, just let us know. Thank you. >> Thank you Michael [applause]. So we have in about 5 or 10 minutes a [inaudible] lounge because of the weather outside, which is over in the Zenko building. We have a reception and all that jazz, which will be followed by.