Chrome Mobile Summit 2013 - Tools and testing

MALE SPEAKER: OK. Our third panel is going to be about testing and tooling. And our moderator is Todd Anglin, who's executive VP at Telerik. They produce tools for developers and developer teams. You might know him best for Kendo UI, and I've seen him. So take it away, Todd. TODD ANGLIN: Great. So I am strictly the time keeper up here. Telerik obviously creates a lot of tools, but I think our panel, which is quite diverse today has the full perspective. We've got about six questions total in the moderating queue. So we'll probably start there to kick it off, but I definitely want to make this as interactive as possible. I know we're starting to stand between you guys and alcohol, which is always a great place to be. So to kick it off, I will let the panel introduce themselves. I was introduced Todd Anglin, work with Telerik, you guys known as Kendo UI, but I'll let the panel introduce themselves. PAVEL FELDMAN: I'm Peter Beverl-- oh no. Sorry. I'm Pavel Feldman, software engineer working on Chrome Developer Tools. YEHUDA KATZ: Hey, I'm Yehuda. I work on Ember mostly these days I've historically worked on jQuery and Rails. ANTON KOVALYOV: Hey, I'm Anton Kovalyov. I work at Mozilla. We make a web browser and a mobile operating system. And I work on Firefox developer tools there. JONATHAN LIPPS: I'm Jonathan Lipps, and I work at Sauce Labs. And this year, I've been working on open source project for mobile automation called Appium. BRIAN LEROUX: Hi. I'm Brian. I work on PhoneGap and Cordova and a thing called Topcoat at Adobe. TODD ANGLIN: Very good. So very diverse panel. Guys, kick it off. First question here, it's from Rob. And the question is basically asking how does Google test its mobile products. So what are the things you use to test your mobile products? But to make a bit more generic, I think we can just open it up. How do you test your mobile products today? What are the techniques you use to ensure that you're delivering good mobile experience? So we'll kick it off linear and start here with Google. PAVEL FELDMAN: OK. so I'm working on Chrome, not really any mobile app or web for Chrome. But we use the same tooling for testing Chrome aggressions as our apps are using, and that's telemetry, which applies pretty much to everything WebGet Chrome based. It has been mentioned earlier. Check out it on the Chromium web site. There's documentation for it. It's a lightweight script framework with a friendly Python framework front end. It's based on remove debugging prodigal. You can automate things there, simple automations, navigations, scrolling. And you can capture all of the data that Chrome dev tools can capture, all of those pages and all of those scenarios. So you get full-fledged timeline there, and everything is Jason based. You can analyze this data. And we are internally building regression charts for that, a lot of birth data for each of the commits, each of the builds we see what has happened there. And we can go back and [INAUDIBLE]. So that's pretty simple. And that's what we expect everybody to do. YEHUDA KATZ: Hey. So I pretty much use your product for testing. I don't have a huge array of physical devices and little start up. So I definitely like the test as many pieces of hardware as I can, and you guys have a really good group. And if I was much bigger, I would want to have more devices to test with but sadly enough. ANTON KOVALYOV: So I work on Firefox for desktops, so I don't really work on actual mobile web apps. But the closest we have, I guess, to the testing mobile web apps would be Firefox apps, and as far as I know we have another Mozilla here. So if I'm saying something stupid, correct me. I think we use the same framework as our own written framework as we use for Firefox or desktop, which basically simulates all the events you can have. And as all testing frameworks out there, no offense to any framework writers, it could be way better. And its partial our job as well as Firefox Developer Tools engineers to make it better. So far it's getting there. But really, unfortunately, I don't have a better answer what we use for actual testing our mobile web apps, if Mozilla has any mobile web apps. JONATHAN LIPPS: That's it? ANTON KOVALYOV: No. TODD ANGLIN: This is a highly-qualified panel. JONATHAN LIPPS: Yeah. So at Sauce, we use this project called Appium, which if any of you guys have done testing of your web apps, you might have used Selenium. Appium uses the same protocol, and it can test the native mode of your applications, or native Android, or native iOS applications. It can also test if you have a mobile web app, you can use the native web application, Safari, and Chrome, and also test kind of the hybrid views of each of those. So you kind of get all of the levels of your application especially if you have a hybrid application, that's pretty hard to do. So it works pretty well. BRIAN LEROUX: OK. So on the PhoneGap project, we built a big *** wall, and we stuck a *** load of phones to it, and we wired all those phones up to a Mac mini. And every time a commit lands inside of Apache Cordova, which is the open source or the project, we compile PhoneGap, we load it with the web app that has a bunch of Jasmine tests. Some of the tests are in queue unit, but that's just historical. And then we post those results of those tests back to a couch. And you could see this stuff, I think it's like at test.cordova.io. And so you have to test on real devices. You can't get around that. Devices have sensors, and you can test for things like, say is there an accelerometer, but you can't actually test the accelerometer data coming through unless you build a robot, which we're talking about with Sauce. So we're doing that. It's an open source project. It's called medic. If you're interested in automating, pushing native apps to phones, and get test results, we can talk about later. There's another project at Adobe call Topcoat, which is CSS project, and we're using telemetry from Chrome to get the data out of that. So we have a custom build, at Chrome, CF, and when we land a commit into our CSS, we add that to a page, and then we load it, and then we add like 200 elements to a page. And we load that, and we scrolled it. And we measure load time, layout, and frames per second. And so we know that in each commit that the CSS is actually getting faster and faster. And all that's open source too. It's at Topcoat organization inside of GitHub. ANTON KOVALYOV: I have a [INAUDIBLE] a question or more like rambling. So I feel like there's a notion in the mobile net world where you have to test on the actual device. And this is true. You have to. You can get away from that, but I don't know if we're actually working so it's reducing the amount of testing we need to do on actual hardware, because attaching a *** load of phones to the wall doesn't scale when you have 200 engineers. Because then you have hundreds of commits per day, every commit launches a phone and then you wait for the test to be done in like what, 2 hours, 10 hours. So shouldn't we like I said tell to the developers, I guess, work on reducing the hardware dependency or adding-- and I'm just throwing out ideas because I haven't actually thought about that-- or adding better simulation to devices, or even better assimilation so the log grade hardware to the hardware limitations. So that I could say, run this test and then run this test when you have only 2 megabytes of memory left. And so the question actually mostly to you guys. JONATHAN LIPPS: So I do think that simulators and emulators provide a good kind of base-level testing, they're actually advantages over using real devices, things like making sure that you have a clean, stable environment. But you're really only testing the changes in your code making it easier to run as part of your continuous integration on every commit. Maybe you want to run your full battery of mobile functional test on emulators because that's cheap and easy, but you don't want to run on all of the huge matrix of real devices. You want to do that when you're ready to cut a release or whatever and then have a little bit more confidence around it. So I think you can view it as kind of a two-phase process, or you do more of the cheaper resource, which is emulators/simulators which you can virtualize the *** out of and then less of the more expensive resource, which is sticking devices on walls. BRIAN LEROUX: Yeah, the device wall does not scale. That's true. But you've got to do it. So the simulator/emulator thing, the only problem with it is, you're in a virtualized environment and then you're probably on good hardware, with good network, with lots of memory, which is basically what mobile isn't. YEHUDA KATZ: I think Anton's point was, hey, people who make simulators, maybe you could not have it emulate a perfect environment. BRIAN LEROUX: Yeah. YEHUDA KATZ: Right? BRIAN LEROUX: Yeah. I agree. It would be nice. I know the Book to Gecko simulator, what was it called, R2-- it was a terrible name. What was it called R2D2, V2G or something crazy? Anyways the Firefox OS simulator is kind of cool because it is that and it is the environment. Whereas like iOS and everything is not quite the real thing. And the other problem is like all these sensors like, I guess we can just chuck fake data at it, hope that it's doing the right thing. But it's a complicated problem for sure. There's another way to test and that's just release often. That's a bad, bad solution. PAVEL FELDMAN: There's another aspect to it. Everything we're talking about today has this performance aspect, so everything is performance when we're talking testing. Testing is testing, and testing as a group performance. And the latter is important to us. And the later is challenging to emulate or simulate. We've been doing something profiling for Intel and ARM. And the pictures that we were having were just radically different. And hardware assimilated stuff isn't entirely different. So if you're interested in the end result and the interest in not regressing the end result on the device, as of today, there's no other way other than getting it to the hardware and testing it there. BRIAN LEROUX: Like one other thing to throw out there for people, like measuring is like super important because then that's how you get to a better result ultimately. But the problem with all this is that it requires a lot of boilerplate set up that developers just do not want to do. When you're writing code, you don't want to go put on your profiler hat and start profiling. You want to write code, right? And ideally, you're notified that hay, something just got slower, fix that. Instead of finding out after you deploy out of context in a context switching. So there's a definite need for better tools that happen during authoring time. PAVEL FELDMAN: I think we'll get there. Telemetry is a perfect first step there, because USS performance as your app runs, you want to add the script running. WebDriver is just not evolving fast enough to capture all that data, so telemetry is kind of a fast and hackier way for WebDriver plus the instrumentation. And running profilers, and raising flags, doing all that, that will all come. We're just only starting there. YEHUDA KATZ: So I have an honest question about profilers in general, which is do people who write the profilers actually, is there like any user testing at all? And I don't mean like, get your grandmother, but I mean like, get me to sit in front of it and look at me trying to use a heap profile dump. Like, if you just watched me for 10 minutes, I think you probably have 10 good ideas on how to make it better. I guess it's a question for Pavel. PAVEL FELDMAN: Yeah. OK. So memory profiling is hard. And no matter what platform you're in, be it Native or Java or JavaScript, historically, memory profiling has been a separate task that you were forced to do. You didn't want to do that. Just because you are always forced to do it because you're out of memory. And you don't want to do it, because it's is hard and challenging. Because memory heap is a graph and it's really complex. But I guess what you're talking about is Chrome Dev Tools UX for heap profiler. We are making some good improvements there. Some of them are recently demoed. We are aware of certain complexities there and certain degree of education that you need to gain the hard way before you get a sense of it. We threw out the garbage of the UI and of the heap, we're removing certain data, but you shouldn't fool yourself. It's a hard problem to tackle, and it will remain hard. Hopefully, we are trying to make it simpler for you. BRIAN LEROUX: You know, one thing to throw on top of that, this is something sort of like we're ashamed of in the Topcoat part of the reality, we're using telemetry. We love it. It works really well. It's a good way to get a measurement and our assumption, which is probably not true, is if we're using telemetry data, that means we're using Chrome and that means we're measuring Chrome. And we figure if it's getting faster in Chrome, it's probably faster in other places. But that's an assumption. And so we need these APIs actually back in the browser proper. We need them in IEs. YEHUDA KATZ: So a couple of examples-- BRIAN LEROUX: Not calling out IE specifically any more. I probably should shut up. YEHUDA KATZ: I could get a little more concrete. So a couple of examples on the heap profiling thing, one of them is, so heap profiling basically tells you if you're leaking memory. It says like, hey, after running a GC pass, we can see that there's X amount of memory. But actually the thing that sucks about garbage is not the garbage that's left, it's often just like you're generating a ton of garbage and it's causing jank. And as far as I know none of the heap profilers allow you to say like, hey, I want you to tell me what's on the heap now, but don't run a GC first. And this is not a very complicated thing. It literally means don't run the GC first and then tell me the same thing that you're telling me already and use the same differencing tools to do that. And I think there's things like this that, I mean, I wonder how the Chrome team is-- surely the Chrome team and the Firefox team occasionally run its issues where they're writing some self-hosted code and they generate a lot of garbage. They want to know what the garbage is. I'm curious what they are doing for these types of problems. PAVEL FELDMAN: So this particular one, the underlying implementation is that you can visit all your heap within the GCM. That's the only way you can visit your heap in [INAUDIBLE]. You could work around it. And I think this particular thing just did not come up. To answer your previous question, our clients, the biggest client that we have is Gmail. And all of the headache that we were getting was Gmail. And it was more than a gig of RAM before they started using it, and then it was like, back and forth iterative process. Gmail was improving, tools were improving. So in-house, we were working like with large web apps where memory matters most. YEHUDA KATZ: so you were worried more about applications that had a lot of persistent memory and less about just like a bunch of memory that got created like during scrolling, let's say. PAVEL FELDMAN: We're just picking Gmail as the largest like crazy app out there. ANTON KOVALYOV: To answer this question, so I run into these issues when writing Firefox front end code since it's all JavaScript. And what I do, I usually cry myself to sleep. Then I wake up and use our very basic tools that we have to kind of fix the problem. So it's not the answer that we have a solution for this. My answer is that everything pretty much sucks, but we're working towards improving that. So Yehuda already gave us really good feedback, but if anyone else has feedback on how the memory profiling, what's in the garbage, and other things could improve your lives, that's really helpful. Because I used to be a web developer. Now I mostly work on the platform for desktops, so I don't do that much web development. So we need input from the outside. TODD ANGLIN: So I think that's a fair point to transition almost as a natural follow up. I mean, I think we would all agree dev tools and browsers have come a long way in a very short time. But they're clearly not done improving. From your perspective, working with them, really digging into them, what are the things we most need still in browser dev tools? And this can be Chrome or just cross browsers in particular. YEHUDA KATZ: Yeah. Hey. This is my topic. So in my opinion, the dev tools have come a long way, and the biggest gaps in the current dev tool story verses like some native tools is there's a very limited ability for framework and library developers to help the dev tools present information in a more useful way. So let me give you a simple example. If you are writing-- a lot of the code that's inside of Chrome is actually written in JavaScript. And for example, you might imagine that for each forum, native arrays is written in JavaScript. But of course, when you press the down arrow to step into your code, you don't actually see the native code, you just see your own code. Because the browser has the ability to say, I do not want to show you this internal code. But as a framework author, I don't actually have any ability tell the browser like, hey, unless the user really wants to see this, they shouldn't see this. So you may have like a number of frames that the user has to step through that's an abstraction. And maybe sometimes they want to see it, but 99% of the time, they want to see their own code. There's other things like ability to annotate a stack tray so that if there's async code you can basically stitched together a bunch of stack traces. And this is information that I have as a framework developer because I'm the one doing the deferral. So I do a lot of crazy stuff in Ember, but even simple things like jQuery and Backbone will benefit a lot. And I think as far as I can tell, there's basically been essentially no work done on this area when there's been a lot of work done on a lot of things. So of course, it's possible to say, well, this is a low priority, but I think given the amount of people that use dev tools and use frameworks and libraries, I think will be very beneficial. JONATHAN LIPPS: Yeah. I think another thing that would be pretty cool is better cross platform dev tools so each of the platforms has their own somewhat good in some cases set of dev tools. But then you have to learn something completely different. And so if there was, I don't know, maybe there are standards in the works. You seem really anxious to talk about something. But I mean, you could say the same thing with the native mobile layer, which we're not really talking about too much right now, but is also obviously important right now in history. ANTON KOVALYOV: I'm very anxious to talk about it because people ask this question a lot for Firefox mostly since we're still catching up obviously with Chrome dev source. So like, oh, why don't you just do the same debugging protocol? Well, this is not going to happen anytime soon mostly because both Chrome and us and, I don't know, whatever Internet Explorer has, we all want to iterate really fast, and we have different priorities. For us, for example, I just want to describe why the standardization of their mobile debugging protocols is not happening right now. For us, for example, the Firefox is a priority, right? So what we are working on, like among other things, is the ability to send files or giant blobs over the protocol. Giants multi megabyte and encrypted-generated files is also a priority for us. I don't know if it's a priority for Chrome. And it might not as well be, and it's cool. But if we will start converging the protocols and doing any kind of synergization, this will slow everyone's down. So the solution to this, I think, and I ask my team yesterday, so my team's solution is to write a shim to the protocol. So when the time comes and we want to use our protocol, our developer tools for Chrome, for example, we can write a shim that will just make a bridge between two protocols, and then we can do that. BRIAN LEROUX: So just quickly. It is going to be standardized TC-39. And I know folks at Adobe are trying to take the lead on that, and yes, it's not going to happen any time soon. And I totally acknowledge that. But it's kind of ironic, I don't often do this, but I'm going to praise Java. Java did this. It has debugging as a part of a language first class, and JavaScript does not, and hence, here we are. And this is desperately needed to have third-party tool ecosystem that sits outside of the browser. Right now, it's almost a browser differentiator. PAVEL FELDMAN: I would second though the standardization is going to slow us down. We don't want to evolve as fast as Java. We want to evolve a bit faster. And when we were on the beginning of our journey, there was a protocol. There was scope, [INAUDIBLE], and we didn't adopt it. We came up with our own thing, and our thinking was we need to do it as fast as possible, and as convenient ourselves as possible, as close to the underlying system as possible. So you would see that our network domain, for example, is very much a WebGet specific. It has things like loader IDs, resource IDs, which are not used in Firefox, so the concepts are different. And of course, we were thinking if it becomes a big thing, we'll be standardizing, but the further we are ahead in the process, the more chances that the protocol that we implemented natively becomes the de facto standard. And I could understand the willingness to do the same thing again. And we were also thinking that changing adapters in the protocols on the Jason or Wire level will solve it. Because debugging domain is very conservative. You just step through stuff, and you don't do anything fancy. And once you stop doing anything fancy, it starts to differ from browser to browser. So we are still waiting for a number of big players to become clients of several browsers to see that there is a need for that, there is an investment to it, and then it will be a clear signal that there is a need for such a standard. BRIAN LEROUX: Yeah. I don't think one comes at the expense of the other at all. And just to like throw some weight behind Yehuda's comment, I think it's well beyond time that we have better instrumentation points to the browser so that we can see the framework start providing better hooks. Because the abstraction layer, like nobody wants to step through jQuery. That's insane. TODD ANGLIN: So we've answered two questions. We don't have a ton of time. If we can move it forward. MALE SPEAKER: We'll take the final reply. TODD ANGLIN: We'll take the final reply on this then we'll move on to-- YEHUDA KATZ: The final reply it was, what do you want from tools? So whatever you want from tools, please file the bugs. We are bugs driven. It's yourbug.com If you are a framework author, please post to all your framework clients to start this bug, and we'll have no other option of implementing it. TODD ANGLIN: There you go. That's how you can force your way. Before we're forced into our final count down, I want to make sure we at least giving our audience a chance to ask a question directly. So if there's anybody who has a question in the space of tools and tooling, I was told there are a lot of questions. Any questions here? Here we go right here. BEN DILTS: Ben Dilts from Lucidchart. One thing that's been a real challenge for us over time is trying to collect error reports in production. So like in the Closure Library, there's something about like an error collector and it has very widely-varying usefulness of data that comes out of that based on what browser, what version, or what have you. And it's almost impossible to get a stack trace of where you currently are because you have like the caller on the function. But obviously, once you have a circular reference. Anyway so is there some master plan for making that better, because it's really terrible right now. ANTON KOVALYOV: I'll start answering. Everything is very sad. So when I was working at Disqus, we had this problem where we wanted to send reports, and everything was really sad. Now, I checked actually a couple of weeks ago because a friend of mine asked, and across a browser, it's like it's pretty good in Chrome, think it's kind of good in Firefox. But when you get like cross browser, it's really basic. And unfortunately, this is not what developer tools team can fix at Firefox. This is either DOM or language design thing. And with DOM, you have WebID and then it starts to cry. BRIAN LEROUX: What about TowTruck? ANTON KOVALYOV: Oh, our thing? BRIAN LEROUX: Yeah. You're thing. ANTON KOVALYOV: Yeah, it's pretty cool. BRIAN LEROUX: Great. ANTON KOVALYOV: So yeah, one thing I want to say is that if there is even more important for mobile development because there when you have an error on the desktop, you're like, oh, what are you using? Are you using Firefox 21. OK. You'll launch our Firefox [INAUDIBLE] for use. On mobile like, oh, we're using this Firefox 900 Android-- I don't know what their version is so I'm going to have to use 2.4 or something, something. And then you cannot really reproduce it. So your best choice is to collect data and save data to the server all time. It will only be grounded about every sub trace and stuff. And I was helping my friend in Century to do it for JavaScript, and that's why I was really anxious to answer that. You should bug everyone you know about the browser vendor to fix that. You should just not stop. You should demand this, because this is really Ember for the web that we cannot send the stack traces to the server normally across a browser. So I'm just acknowledging. YEHUDA KATZ: I also agree, and I've raised this a few times on TC-39. And there is a low appetite for standardizing stack traces, but I think this is a mistake. So the issue to get past is that when you standardize stack traces, you are forcing the VM authors into particular pattern of how they have to collect and maintain their code. And this is not a good excuse, and I agree with Anton. You should make sure that all the browser vendors and VM pointers that you know know you don't think it's a good excuse. Like this is probably, it's like a first step of debugging. It's like before print after debugging, it's like give me a stack trace. And we don't even have it on the browser. And it is probably sufficient to leave open a little bit of wiggle room in terms of what exactly you are required to do. But having some notion of like some standardization here would be very helpful. Yes. And people should ask. TODD ANGLIN: So I've got the five-minute warning, which means that it's probably time to take our 30-second answers. So I hate to be repetitive to the other panels here, but there's not many better ways to wrap up than to ask you guys, with the perspectives represented here both two authors and consumers and platforms, if we were back on the stage a year from now, and we're talking about tooling and testing, either one given the subject, what is the thing you would hope is most changed a year from now? I mean, there's a lot of things that need to be done. We've been told it's a very sad state of affairs right now. It's a very depressing panel. But what is the thing you would hope a year from now is not depressing relative to today? We'll just run down the line. PAVEL FELDMAN: There'd just be better tooling. TODD ANGLIN: That's your job, right? PAVEL FELDMAN: Right. And we'll make that. So imagine you're running your app-- 30 seconds, right? You're running your app, and you can go forward and back in time explore everything, everything that was consuming time. All of the causes fixed in live, the replay thing, it works faster, then make sure that it does regress. Just a miracle. TODD ANGLIN: So that's May 16, 2014. And we've got a commitment here. All right. YEHUDA KATZ: The answer I used to say to this was some browser, any browser, pause on Stack Overflow. Thankfully, Firefox does it. Chrome, please pause on Stack Overflows. This is ridiculous. But in all seriousness, what I want, in general, is just more ability as a library and framework author to control what it is that the tools are showing to my users, because having my code be essentially treated as equivalent to their code is really not a good experience, and it is, I think, beyond a small issue. ANTON KOVALYOV: So understanding in terms of mobile tooling, I think it would be really nice, and I know both Chrome and Firefox are working on that, is to lower the barrier entry to using the dev tools. The fact that we require, like Firefox requires web developers to install Android SDK and use ADB to connect to Firefox to Android to debug their god damn web application is insanity. There should be a Firefox add on or something, Android plug-in or Chrome plug-in that says, OK, now, you can connect your device. Now, you're all good. And we're actually working on that. Someone told me today that Chrome is working on that. So it's really good, and hopefully, in a year or less, we'll have a much lower barrier of entry for web developers to actually use tools to debug mobile web sites. JONATHAN LIPPS: Taking a more testing prospective, what I want to see is you check code for your iPhone application, your Android application, and your web application. And to GitHub your continuous integration pulls that down. All three versions of the application are built. All three versions of the application are automatically tested using the one, single test suite, not three. And then it's built, signed for deployment, deployed to your fleet of beta users, and then given some standard metric that you're tracking, automatically deployed to your app store to the web. I think continuous deployment integration for mobile is awesome, and it's not there yet. BRIAN LEROUX: ***, I'd love that. Good luck. So there's like two ways to look at the whole problem. Steve Souders has an awesome article about how the mobile web is getting faster. It's like something-- I don't know. I can remember. It's a percentage faster. But the problem is developers are still writing bigger and bigger web apps. And the web apps themselves are getting bigger, and so the performance gains on sort of the platform side are being lost due to the weight. So to me, I'd view very much as a bit of a documentation issue that we got to get in front of developers and tell them to start measuring their stuff so they start actually building improvements themselves instead of just loading a whole bunch of script tags and large images, they have to start thinking [INAUDIBLE] side about how they're going architect their apps. That's what I hope for. TODD ANGLIN: Cool. So on that note, thank you, panel.