Yamyam: 1版をインポートしました

2018-06-28T21:21:39Z

1版をインポートしました

wiki>Eibriel: /* Validation */

2016-11-02T04:58:45Z

‎Validation

新規ページ

=Proposal=

Automatically classify and tag bug reports.

==Motivation==
Bug review takes a lot of valuable time to developers, that task could be automatized at least in part.

==Scope==
The software should read bug reports tagged as '''Needs Triage''', apply a statistical model to predict if the developer will tag the report as '''Incomplete''' ([https://developer.blender.org/T49807 example]) , if the likelihood surpass some threshold it will leave a message like:

''Please follow our submission template and guidelines and make a complete, valid bug report, with required info, precise description of the issue, precise steps to reproduce it, small and simple .blend and/or other files to do so if needed, etc.''

<u>The developer still needs to set the triage of the bug report, but at that time the user will correct the description of the issue.</u>

The software could also detect when a bug report is in fact a Support Request ([https://developer.blender.org/T49889 example]) and could redirect the user to the Community Support.

As a more advanced feature the software could automatically assign bug reports to the corresponding developer.

=The software=
The software will be a Python3 script coupled with a database.
The script will communicate to developer.blender.org through the Conduit API using an user specifically created for this task.
The script will use '''SKLearn''' to infer the current status of the bug report.

The username, avatar and message should be easily recognizable as a Bot and not a real Developer.

==Machine Learning==
The inference will be done using Machine Learning techniques, I propose using [http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.BaggingClassifier.html BaggingClassifier] over a [http://scikit-learn.org/stable/modules/feature_extraction.html#the-bag-of-words-representation bag of words] (to investigate:[http://scikit-learn.org/stable/modules/feature_extraction.html#common-vectorizer-usage Common Vectorizer]).

===Dumping the Database===
I'll make a dump of the Bug Reports on the last 12 months in such a way that we know what was the report at the moment it received the firs triage.

===Features===
Then I'll analyze the reports and extract features (words) that better describe the quality of the Bug Report.
Also will be useful to know if the report have at least one blend file attached or at least a text file attached.

===Training===
Then I can train the model to predict the triage received using as feature the count of the words on the "bag".
For instance if the Bug don't contains any of the words "Linux", "Windows" or "MacOS" the probability to be an '''Incomplete''' bug is high.

===Validation===
I'll test the model on new bug reports to measure its performance, when the model have a performance higher than some value selected by blender core developers (for example 95%) the model will be ready for production.
We can also choose to prefer false positives or false negatives. Maybe is better to have a high performance on classifying correctly all '''Incomplete''' bug reports even when the system misclassify some '''Complete''' bug reports, or maybe is better on the other way to avoid confusing the users, classifying correctly all '''Complete''' bug reports at the expense of misclassify some '''Incomplete''' bug reports.

===Maintenance===
The model can be retrained regularly using new data gathered automatically (and deleting old data) to keep up to date with new ways of write '''Incomplete''' bug reports.

=Conduit API=
https://developer.blender.org/api/conduit.ping

{"result":"developer.blender.org","error_code":null,"error_info":null}

← 古い版	2018年6月28日 (木) 21:21時点における版
(相違点なし)

利用者:Eibriel/Automated Bug Classification - 版の履歴

Yamyam: 1版 をインポートしました

wiki>Eibriel: /* Validation */

Yamyam: 1版をインポートしました