530418 - create a feed of crash report user comments to engage more people in watching feedback and trying test cases

Status: RESOLVED WONTFIX

(Socorro :: General, task)

Description

[chris hofmann]

Attachment: example feed from actual comment data received on 11/20

There is a lot of noise in crash data commments, but there are also some very valuable nuggets of information that can be gained by looking at a time base view of user comments and having a view of the data that testers and sumo contributors could look at as another channel of user feedback.

Ken's talked about big monitors around the office that would show these feeds  of user comments to better connect everyone working on Firefox to the people that are using the software.  Crash comments are a perfect connection point for that idea to get people focused on things that affect users.

More eyeballs make all bugs shallow!

So the proposal here is to create a twitter-like/rss-kind-of/irc-kind-of feed that would stream user crash comments in near real-time.

The attachment is a rough wire frame of the kinds of data that might be presented 

It includes:

-process time of the incoming crash report, 

-firefox version number (maybe we even have filters on this so I can just view the firefox 3.6b3 right after it goes on the wire to watch for new regressin crashes...)

-maybe a short indication of the stack signature/directly link to the crash report so viewers of the report can quickly get to  more details

-The comments  --  we could also highlight/bold special words like "ever time"  or words that people use that indicate they might have some insight to a repeatable situation or interesting test case.

e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=521844#c31

-Any related bugs that are on file for the crash signature they hit.

Comment 1

[chris hofmann]

https://bugzilla.mozilla.org/show_bug.cgi?id=529485#c1  is another example of the needle in the haystack, but golden nugget comments that would become much more visable in this report than any other view of the comments that we have in reports right now.

Comment 2

[chris hofmann]

Attachment: crash comment rates

I took a quick look at what the raw comment rates might be for a Firefox only feed after comment throttling was fixed earlier this month.   it looks like daily peaks are around 20 per minute but most of the time we are around 1-10 comments per minute.   that should be slow enough for most people to digest.

I think an update rate of around every 1 or two minutes is fast enough to keep people interested in watching the feed, but slow enough not to over burden socorro,  although if there is a way to make the stream update as the comments come in that would be pretty cool.

Comment 3

[chris hofmann]

this might be a really effect tool for finding reproducible crashes in the long tail per examples in comment 1 and 0.    adding crashkill-metrics

Comment 4

[Bob Clary [:bc] (inactive)]

I think this would be a good idea. Maybe as the default feed in 10-forward and the conference rooms.

I now realize that crash-stats makes the comments publicly accessible, but I would like to be sure to tell them that the comments will be public so they don't inadvertently make their name/phonenumber/email much more accessible to the planet.

Should we put this behind some sort of authentication?

Comment 5

[chris hofmann]

not sure it needs auth since all this data (see list in comment 0) is public now.  this just slices it differently.

the signal to noise of people commenting on fx crashes in facebook is pretty low. the signal to noise in this pile of data is still going to be pretty low too, but much higher than facebook... at least it has signature and link to details in a crash report and bugs filed.

Comment 6

[chris hofmann]

ok, to show what this might look like, see the volume of data, and to get more people finding golden nuggets in firefox 3.6 beta crash data I published this 

http://people.mozilla.com/~chofmann/crash-data/comment-feed/

Comment 7

[chris hofmann]

Attachment: with out much work this one line command could make a lot prettier output if someone want to get it a try.

Comment 8

[Justin Dolske [:Dolske]]

While a feed of crash comments would be mildly interesting to watch scroll by on monitors, I can't agree this would be helpful or productive in terms of actually doing something about crashes. It could even be very counterproductive, if people start passing around the feed as "proof" of how terrible Firefox is. [The comments are from people who just crashed, so they're always going to be overwhelmingly negative -- classic sample bias.]

Crash comments are, on occasion, useful when investigating a particular crash signature. But the reverse isn't true -- just from watching comments you're not going to solve general crash problems. Skimming through the lists in comment 6, I find them to be useless.

OTOH, what *could* be helpful is to watch for trending words / phrases -- ala Twitter -- and put that list onto Socorro. We'd probably have seen "farmville" pop up that way, although we'd see that anyway for trends that correlate with specific crash signatures (assuming someone investigates that signature). However, it could help catch problems that stem from a single cause but manifest as a variety of signatures (and would thus be lost in the long-tail of crashes).

[As a bonus, you could probably reuse the trend-detection code to watch for trending crash signatures.]

Comment 9

[Bob Clary [:bc] (inactive)]

Attachment: crash-comments.py

python crash_comments.py ffversion crashlog

Comment 10

[Bob Clary [:bc] (inactive)]

Attachment: out.html

python crash-comments.py 3.6b 20091119-crashdata.csv.gz > out.html

Comment 11

[chris hofmann]

(In reply to comment #8)
> While a feed of crash comments would be mildly interesting to watch scroll by
> on monitors, I can't agree this would be helpful or productive in terms of
> actually doing something about crashes. 

I agree to some extent.  Its a bit of a two edged sword to publish this slice of the data.  every report we have ever made draws a bit of this concern.  its worthwhile thinking about this a bit.

> It could even be very
> counterproductive, if people start passing around the feed as "proof" of how
> terrible Firefox is. [The comments are from people who just crashed, so they're
> always going to be overwhelmingly negative -- classic sample bias.]

anyone who is looking for links to pass around to "prove" firefox is crashy is already will armed with data that we already publish on crash-stats.  In fact with not much digging and a wget running in a loop the data can be gathered and publish already.   

> 
> Crash comments are, on occasion, useful when investigating a particular crash
> signature. But the reverse isn't true -- just from watching comments you're not
> going to solve general crash problems. Skimming through the lists in comment 6,
> I find them to be useless.
> 

I agree with some of this.  the signal to noise ratio is very low. the key to using this is will be if we can start to look at the stream of data and develop better filters to reduce the noise and find "interesting" comments.

> OTOH, what *could* be helpful is to watch for trending words / phrases -- ala
> Twitter -- and put that list onto Socorro. We'd probably have seen "farmville"
> pop up that way, although we'd see that anyway for trends that correlate with
> specific crash signatures (assuming someone investigates that signature).
> However, it could help catch problems that stem from a single cause but
> manifest as a variety of signatures (and would thus be lost in the long-tail of
> crashes).
> 
> [As a bonus, you could probably reuse the trend-detection code to watch for
> trending crash signatures.]

yeah,  I'm thinking the first sage is to get people to find interesting trends by scaning the data.   last summer we didn't know that we should be looking for "farmville" across a variety of different signatures.  

these kinds of reports being scanned by many would bring attention to that particular "farmville" comment reappearing on many reports and across a variety of stack signatures, and a lot of the time its NPSWF32.dll@something in the signature, and there is/or is not a bug on file

once an interesting word or phrase pops up the using "quick find" in the browser allows some fast scanning for similar reports.

this definitely taps into the idea the humans are pattern matching animals, and the risk as you point out is that sometimes the pattern matching isn't very good or accurate.

I think the next step on this is to start developing filters that look for interesting slices of comments.  Filters show only things like

 "every time"
 "whenever I"

might lead to reproducible test cases.

 filters that remove any comment with the pattern

   [SsFf][Uu][Cc][Kk]

  might just be reflected as a single number "f'index" that measures general user frustration.   estabilishing a baseline for the f'index and watching for breakouts from the general trend might also have some general use in helping to draw attention for efforts like crashkill.  I agree those comments don't add value or understanding to fixing particular bugs.

Comment 12

[chris hofmann]

add

   [Ss][Hh][Ii][Tt]  

to the f'index.  We could also develop localized f'indices.  There also is some general utility in knowing things like the Polish f'index took a big climb after we shipped firefox 3.6b1.  something like that might have peaked some interest, some investigation, and maybe earlier discovery of bug 525581 before it became a top crash.

Comment 13

[Bob Clary [:bc] (inactive)]

Attachment: crash-comments.py

"hints" at end, wraps bugs list and sanitizes comments by changing & to &amp; > to &gt; < to &lt;

Comment 14

[Samuel Sidler (old account; do not CC)]

Seems like you really want comments by stack, no? That already exists in the comments tab:

https://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox:3.5.5&query_search=signature&query_type=exact&query=&date=&range_value=1&range_unit=weeks&do_query=1&signature=UserCallWinProcCheckWow&page=1

We can probably make it better, but it seems like we should just be improving that report.

Comment 15

[chris hofmann]

(In reply to comment #14)
> Seems like you really want comments by stack, no?

yes, in some cases.  no, in other cases.   we know that some crashes stretch across multiple signatures.  this helps to identify similar user situations that end up in crashing at a variety of different signatures.  we don't have a report that helps to do that.

Comment 16

[Austin King [:ozten]]

(In reply to comment #13)
@Bob and anyone else hacking on this, you can use the live RSS feeds and poll...

http://crash-stats.mozilla.com/feed/crashes_by_platform/ALL
http://crash-stats.mozilla.com/feed/crashes_by_platform/Win
http://crash-stats.mozilla.com/feed/crashes_by_platform/Mac
http://crash-stats.mozilla.com/feed/crashes_by_platform/Linux
http://crash-stats.mozilla.com/feed/crashes_by_platform/{platform optional}

http://crash-stats.mozilla.com/feed/crashes_by_product/Firefox/3.5.5
http://crash-stats.mozilla.com/feed/crashes_by_product/Camino/2.1a1pre/Mac
http://crash-stats.mozilla.com/feed/crashes_by_product/{product}/{version}/{platform optional}

(In reply to comment #0)
These comments are terribly de-motivating... I don't know about a big shiny screen full of frustration is that great.

Right now the comments tab is paginated by the current page of crashes you are looking at, but we could make it paginate on it's own for exploring the long tail, and have each comment link to it's crash details page.

Comment 17

[Bob Clary [:bc] (inactive)]

(In reply to comment #16)
> (In reply to comment #13)
> @Bob and anyone else hacking on this, you can use the live RSS feeds and
> poll...
> 
> http://crash-stats.mozilla.com/feed/crashes_by_platform/ALL

cool, but... fails to load in firefox / thunderbird due to undefined entities...


> 
> (In reply to comment #0)
> These comments are terribly de-motivating... I don't know about a big shiny
> screen full of frustration is that great.

The truth will set us free!

Comment 18

[Laura Thomson :laura]

(In reply to comment #17)
> (In reply to comment #16)
> > (In reply to comment #13)
> > @Bob and anyone else hacking on this, you can use the live RSS feeds and
> > poll...
> > 
> > http://crash-stats.mozilla.com/feed/crashes_by_platform/ALL
> 
>

Also seems to be timing out (connection reset errors).

So if we fix the bugs (entities, timeouts) is this sufficient to close out this bug?

Comment 19

[chris hofmann]

yeah, once that is in place we could start using it and file follow ups.

Comment 20

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

I'd also suggest exposing the http://crash-stats.mozilla.com/feed/crashes_by_product/{product}/{version} feed in the <head> on the http://crash-stats.mozilla.com/topcrasher/byversion/{product}/{version} pages before closing the bug.  (You might also include the {product}/{version}/{platform} feeds there, and there may be some other good 1:1 page:feed mappings, but I'd say product+version on the product+version topcrashers page is the minimum.) 

Apparently these feeds have been around since last Nov (per comment 16) but there's absolutely no indication that they exist unless you stumble upon this bug (or are the webdev who implemented them?) ;)  Putting them in the <head> will let browsers’ feed detection notice the feeds, which in turn would get my attention when working with the topcrash reports. :)

Comment 21

[(not currently active) Ted Mielczarek]

The feeds were implemented in bug 492017. I wanted them purely for machine-readable reasons, so I guess we never looked into making them human-readable or visible.

Comment 22

[Laura Thomson :laura]

I think deinspanjer has some tools for sentiment analysis which might be useful here.

Comment 23

[Daniel Einspanjer [:dre] [:deinspanjer]]

It is a project we hope to pick up again sometime this year, but it is not currently active and we don't have a general Q2 goal for it.

Comment 24

[Robert Kaiser]

Is this the same stuff that runs for input? I guess it might make sense to run something like the same harness against crash comments.

Comment 25

[Laura Thomson :laura]

A couple of things.
1. The RSS feeds are gone.
2. We can, however, use the API to get this data.
3. Seriously, most of the comments are either obscene (F you, I'm switching to Chrome) or give steps to repro (I was doing X). I'm genuinely not sure this is in any way useful in isolation (ie on a screen in the offices) and would probably be horrible for morale.

I'd like to WONTFIX. Any objections?

Comment 26

[Bob Clary [:bc] (inactive)]

I feel that we are sometimes disconnected from our users and need to be reminded that every thing is not all ponies and rainbows for them, but I'm fine with WONTFIX.

Comment 27

[Matt Brandt [:mbrandt]]

Bumping to wontfix per comment 25 and comment 26