一行代碼不用寫，就可以訓(xùn)練、測(cè)試、使用模型，這個(gè)star量1.5k的項(xiàng)目幫你做到
一行代碼不用寫，就可以訓(xùn)練、測(cè)試和使用模型，還有這樣的好事?
最近，軟件工程師 Nidhal Baccouri 就在 GitHub 上開源了一個(gè)這樣的機(jī)器學(xué)習(xí)工具——igel，并登上了 GitHub 熱榜。目前，該項(xiàng)目 star 量已有 1.5k。
項(xiàng)目地址：https://github.com/nidhaloff/igel
該項(xiàng)目旨在為每一個(gè)人(包括技術(shù)和非技術(shù)人員)提供使用機(jī)器學(xué)習(xí)的便捷方式。
項(xiàng)目作者這樣描述創(chuàng)建 igel 的動(dòng)機(jī)：「有時(shí)候我需要一個(gè)用來(lái)快速創(chuàng)建機(jī)器學(xué)習(xí)原型的工具，不管是進(jìn)行概念驗(yàn)證還是創(chuàng)建快速 draft 模型。我發(fā)現(xiàn)自己經(jīng)常為寫樣板代碼或思考如何開始而犯愁。于是我決定創(chuàng)建 igel?！?/p>
igel 基于 scikit-learn 構(gòu)建，支持 sklearn 的所有機(jī)器學(xué)習(xí)功能，如回歸、分類和聚類。用戶無(wú)需編寫一行代碼即可使用機(jī)器學(xué)習(xí)模型，只要有 yaml 或 json 文件，來(lái)描述你想做什么即可。
其基本思路是在人類可讀的 yaml 或 json 文件中將所有配置進(jìn)行分組，包括模型定義、數(shù)據(jù)預(yù)處理方法等，然后讓 igel 自動(dòng)化執(zhí)行一切操作。用戶在 yaml 或 json 文件中描述自己的需求，之后 igel 使用用戶的配置構(gòu)建模型，進(jìn)行訓(xùn)練，并給出結(jié)果和元數(shù)據(jù)。
igel 目前支持的所有配置如下所示：
- # dataset operations
- dataset:
- type: csv # [str] -> type of your dataset
- read_data_options: # options you want to supply for reading your data (See the detailed overview about this in the next section)
- sep: # [str] -> Delimiter to use.
- delimiter: # [str] -> Alias for sep.
- header: # [int, list of int] -> Row number(s) to use as the column names, and the start of the data.
- names: # [list] -> List of column names to use
- index_col: # [int, str, list of int, list of str, False] -> Column(s) to use as the row labels of the DataFrame,
- usecols: # [list, callable] -> Return a subset of the columns
- squeeze: # [bool] -> If the parsed data only contains one column then return a Series.
- prefix: # [str] -> Prefix to add to column numbers when no header, e.g. ‘X’ for X0, X1, …
- mangle_dupe_cols: # [bool] -> Duplicate columns will be specified as ‘X’, ‘X.1’, …’X.N’, rather than ‘X’…’X’. Passing in False will cause data to be overwritten if there are duplicate names in the columns.
- dtype: # [Type name, dict maping column name to type] -> Data type for data or columns
- engine: # [str] -> Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.
- converters: # [dict] -> Dict of functions for converting values in certain columns. Keys can either be integers or column labels.
- true_values: # [list] -> Values to consider as True.
- false_values: # [list] -> Values to consider as False.
- skipinitialspace: # [bool] -> Skip spaces after delimiter.
- skiprows: # [list-like] -> Line numbers to skip (0-indexed) or number of lines to skip (int) at the start of the file.
- skipfooter: # [int] -> Number of lines at bottom of file to skip
- nrows: # [int] -> Number of rows of file to read. Useful for reading pieces of large files.
- na_values: # [scalar, str, list, dict] -> Additional strings to recognize as NA/NaN.
- keep_default_na: # [bool] -> Whether or not to include the default NaN values when parsing the data.
- na_filter: # [bool] -> Detect missing value markers (empty strings and the value of na_values). In data without any NAs, passing na_filter=False can improve the performance of reading a large file.
- verbose: # [bool] -> Indicate number of NA values placed in non-numeric columns.
- skip_blank_lines: # [bool] -> If True, skip over blank lines rather than interpreting as NaN values.
- parse_dates: # [bool, list of int, list of str, list of lists, dict] -> try parsing the dates
- infer_datetime_format: # [bool] -> If True and parse_dates is enabled, pandas will attempt to infer the format of the datetime strings in the columns, and if it can be inferred, switch to a faster method of parsing them.
- keep_date_col: # [bool] -> If True and parse_dates specifies combining multiple columns then keep the original columns.
- dayfirst: # [bool] -> DD/MM format dates, international and European format.
- cache_dates: # [bool] -> If True, use a cache of unique, converted dates to apply the datetime conversion.
- thousands: # [str] -> the thousands operator
- decimal: # [str] -> Character to recognize as decimal point (e.g. use ‘,’ for European data).
- lineterminator: # [str] -> Character to break file into lines.
- escapechar: # [str] -> One-character string used to escape other characters.
- comment: # [str] -> Indicates remainder of line should not be parsed. If found at the beginning of a line, the line will be ignored altogether. This parameter must be a single character.
- encoding: # [str] -> Encoding to use for UTF when reading/writing (ex. ‘utf-8’).
- dialect: # [str, csv.Dialect] -> If provided, this parameter will override values (default or not) for the following parameters: delimiter, doublequote, escapechar, skipinitialspace, quotechar, and quoting
- delim_whitespace: # [bool] -> Specifies whether or not whitespace (e.g. ' ' or ' ') will be used as the sep
- low_memory: # [bool] -> Internally process the file in chunks, resulting in lower memory use while parsing, but possibly mixed type inference.
- memory_map: # [bool] -> If a filepath is provided for filepath_or_buffer, map the file object directly onto memory and access the data directly from there. Using this option can improve performance because there is no longer any I/O overhead.
- split: # split options
- test_size: 0.2 #[float] -> 0.2 means 20% for the test data, so 80% are automatically for training
- shuffle: true # [bool] -> whether to shuffle the data before/while splitting
- stratify: None # [list, None] -> If not None, data is split in a stratified fashion, using this as the class labels.
- preprocess: # preprocessing options
- missing_values: mean # [str] -> other possible values: [drop, median, most_frequent, constant] check the docs for more
- encoding:
- type: oneHotEncoding # [str] -> other possible values: [labelEncoding]
- scale: # scaling options
- method: standard # [str] -> standardization will scale values to have a 0 mean and 1 standard deviation | you can also try minmax
- target: inputs # [str] -> scale inputs. | other possible values: [outputs, all] # if you choose all then all values in the dataset will be scaled
- # model definition
- model:
- type: classification # [str] -> type of the problem you want to solve. | possible values: [regression, classification, clustering]
- algorithm: NeuralNetwork # [str (notice the pascal case)] -> which algorithm you want to use. | type igel algorithms in the Terminal to know more
- arguments: # model arguments: you can check the available arguments for each model by running igel help in your terminal
- use_cv_estimator: false # [bool] -> if this is true, the CV class of the specific model will be used if it is supported
- cross_validate:
- cv: # [int] -> number of kfold (default 5)
- n_jobs: # [signed int] -> The number of CPUs to use to do the computation (default None)
- verbose: # [int] -> The verbosity level. (default 0)
- # target you want to predict
- target: # list of strings: basically put here the column(s), you want to predict that exist in your csv dataset
- - put the target you want to predict here
- - you can assign many target if you are making a multioutput prediction
這款工具具備以下特性：
- 支持所有機(jī)器學(xué)習(xí) SOTA 模型(甚至包括預(yù)覽版模型);
- 支持不同的數(shù)據(jù)預(yù)處理方法;
- 既能寫入配置文件，又能提供靈活性和數(shù)據(jù)控制;
- 支持交叉驗(yàn)證;
- 支持 yaml 和 json 格式;
- 支持不同的 sklearn 度量，進(jìn)行回歸、分類和聚類;
- 支持多輸出 / 多目標(biāo)回歸和分類;
- 在并行模型構(gòu)建時(shí)支持多處理。
如前所示，igel 支持回歸、分類和聚類模型，包括我們熟悉的線性回歸、貝葉斯回歸、支持向量機(jī)、Adaboost、梯度提升等。
igel 支持的回歸、分類和聚類模型。
快速入門
為了讓大家快速上手 igel，項(xiàng)目作者在「README」文件中提供了詳細(xì)的入門指南。
運(yùn)行以下命令可以獲取 igel 的幫助信息：
- $ igel --help
- # or just
- $ igel -h"""Take some time and read the output of help command. You ll save time later if you understand how to use igel."""
第一步是提供一份 yaml 文件(你也可以使用 json)。你可以手動(dòng)創(chuàng)建一個(gè). yaml 文件并自行編輯。但如何你很懶，也可以選擇使用 igel init 命令來(lái)快速啟動(dòng)：
- """igel init <args>possible optional args are: (notice that these args are optional, so you can also just run igel init if you want)-type: regression, classification or clustering-model: model you want to use-target: target you want to predictExample:If I want to use neural networks to classify whether someone is sick or not using the indian-diabetes dataset,then I would use this command to initliaze a yaml file:
- $ igel init -type "classification" -model "NeuralNetwork" -target "sick""""
- $ igel init
運(yùn)行該命令之后，當(dāng)前的工作目錄中就有了一個(gè) igel.yaml 文檔。你可以檢查這個(gè)文件并進(jìn)行修改，也可以一切從頭開始。
在下面這個(gè)例子中，作者使用隨機(jī)森林來(lái)判斷一個(gè)人是否患有糖尿病。他用到的數(shù)據(jù)集是著名的「Pima Indians Diabetes Database」。
- # model definitionmodel:
- # in the type field, you can write the type of problem you want to solve. Whether regression, classification or clustering# Then, provide the algorithm you want to use on the data. Here I'm using the random forest algorithmtype: classificationalgorithm: RandomForest # make sure you write the name of the algorithm in pascal casearguments:
- n_estimators: 100 # here, I set the number of estimators (or trees) to 100max_depth: 30 # set the max_depth of the tree# target you want to predict# Here, as an example, I'm using the famous indians-diabetes dataset, where I want to predict whether someone have diabetes or not.# Depending on your data, you need to provide the target(s) you want to predict heretarget:
- - sick
注意，作者將 n_estimators 和 max_depth 傳遞給了模型，用作模型的附加參數(shù)。如果你不提供參數(shù)，模型就會(huì)使用默認(rèn)參數(shù)。你不需要記住每個(gè)模型的參數(shù)。相反，你可以在終端運(yùn)行 igel models 進(jìn)入交互模式。在交互模式下，系統(tǒng)會(huì)提示你輸入你想要使用的模型以及你想要解決的問題的類型。接下來(lái)，Igel 將展示出有關(guān)模型的信息和鏈接。通過該鏈接，你可以看到可用參數(shù)列表以及它們的使用方法。
igel 的使用方式應(yīng)該是從終端(igel CLI)：
在終端運(yùn)行以下命令來(lái)擬合 / 訓(xùn)練模型，你需要提供數(shù)據(jù)集和 yaml 文件的路徑。
- $ igel fit --data_path 'path_to_your_csv_dataset.csv' --yaml_file 'path_to_your_yaml_file.yaml'
- # or shorter
- $ igel fit -dp 'path_to_your_csv_dataset.csv' -yml 'path_to_your_yaml_file.yaml'"""That's it. Your "trained" model can be now found in the model_results folder(automatically created for you in your current working directory).Furthermore, a description can be found in the description.json file inside the model_results folder."""
接下來(lái)，你可以評(píng)估訓(xùn)練 / 預(yù)訓(xùn)練好的模型：
- $ igel evaluate -dp 'path_to_your_evaluation_dataset.csv'"""This will automatically generate an evaluation.json file in the current directory, where all evaluation results are stored"""
如果你對(duì)評(píng)估結(jié)果比較滿意，就可以使用這個(gè)訓(xùn)練 / 預(yù)訓(xùn)練好的模型執(zhí)行預(yù)測(cè)。
- $ igel predict -dp 'path_to_your_test_dataset.csv'"""This will generate a predictions.csv file in your current directory, where all predictions are stored in a csv file"""
你可以使用一個(gè)「experiment」命令將訓(xùn)練、評(píng)估和預(yù)測(cè)結(jié)合到一起：
- $ igel experiment -DP "path_to_train_data path_to_eval_data path_to_test_data" -yml "path_to_yaml_file""""This will run fit using train_data, evaluate using eval_data and further generate predictions using the test_data"""
當(dāng)然，如果你想寫代碼也是可以的。
交互模式
交互模式是 v0.2.6 及以上版本中新添加的，該模式可以讓你按照自己喜歡的方式寫參數(shù)。
也就是說(shuō)，你可以使用 fit、evaluate、predict、experiment 等命令而無(wú)需指定任何額外的參數(shù)，比如：
- igel fit
如果你只是編寫這些內(nèi)容并點(diǎn)擊「enter」，系統(tǒng)將提示你提供額外的強(qiáng)制參數(shù)。0.2.5 及以下版本會(huì)報(bào)錯(cuò)，所以你需要使用 0.2.6 及以上版本。
如 demo 所示，你不需要記住這些參數(shù)，igel 會(huì)提示你輸入這些內(nèi)容。具體而言，Igel 會(huì)提供一條信息，解釋你需要輸入哪個(gè)參數(shù)。括號(hào)之間的值表示默認(rèn)值。
端到端訓(xùn)練示例
項(xiàng)目作者給出了使用 igel 進(jìn)行端到端訓(xùn)練的完整示例，即使用決策樹算法預(yù)測(cè)某人是否患有糖尿病。你需要?jiǎng)?chuàng)建一個(gè) yaml 配置文件，數(shù)據(jù)集可以在 examples 文件夾中找到。
擬合 / 訓(xùn)練模型：
- model:
- type: classification
- algorithm: DecisionTree
- target:
- - sick
- $ igel fit -dp path_to_the_dataset -yml path_to_the_yaml_file
現(xiàn)在，igel 將擬合你的模型，并將其保存在當(dāng)前目錄下的 model_results 文件夾中。
評(píng)估模型：
現(xiàn)在開始評(píng)估預(yù)訓(xùn)練模型。Igel 從 model_results 文件夾中加載預(yù)訓(xùn)練模型并進(jìn)行評(píng)估。你只需要運(yùn)行 evaluate 命令并提供評(píng)估數(shù)據(jù)的路徑即可。
- igel evaluate -dp path_to_the_evaluation_dataset
Igel 進(jìn)行模型評(píng)估，并將 statistics/results 保存在 model_results 文件夾中的 evaluation.json 文件中。
預(yù)測(cè)：
這一步使用預(yù)訓(xùn)練模型預(yù)測(cè)新數(shù)據(jù)。這由 igel 自動(dòng)完成，你只需提供預(yù)測(cè)數(shù)據(jù)的路徑即可。
- $ igel predict -dp path_to_the_new_dataset
Igel 使用預(yù)訓(xùn)練模型執(zhí)行預(yù)測(cè)，并將其保存在 model_results 文件夾中的 predictions.csv 文件中。
高階用法
你還可以通過在 yaml 文件中提供某些預(yù)處理方法或其他操作來(lái)執(zhí)行它們。關(guān)于 yaml 配置文件請(qǐng)參考 GitHub 詳細(xì)介紹。在下面的示例中，將數(shù)據(jù)拆分為訓(xùn)練集 80%，驗(yàn)證 / 測(cè)試集 20%。同樣，數(shù)據(jù)在拆分時(shí)會(huì)被打亂。
此外，可以通過用均值替換缺失值來(lái)對(duì)數(shù)據(jù)進(jìn)行預(yù)處理：
- # dataset operations
- dataset:
- split:
- test_size: 0.2
- shuffle: True
- stratify: default
- preprocess: # preprocessing options
- missing_values: mean # other possible values: [drop, median, most_frequent, constant] check the docs for more
- encoding:
- type: oneHotEncoding # other possible values: [labelEncoding]
- scale: # scaling options
- method: standard # standardization will scale values to have a 0 mean and 1 standard deviation | you can also try minmax
- target: inputs # scale inputs. | other possible values: [outputs, all] # if you choose all then all values in the dataset will be scaled
- # model definition
- model:
- type: classification
- algorithm: RandomForest
- arguments:
- # notice that this is the available args for the random forest model. check different available args for all supported models by running igel help
- n_estimators: 100
- max_depth: 20
- # target you want to predict
- target:
- - sick
然后，可以通過運(yùn)行 igel 命令來(lái)擬合模型：
- $ igel fit -dp path_to_the_dataset -yml path_to_the_yaml_file
評(píng)估：
- $ igel evaluate -dp path_to_the_evaluation_dataset
預(yù)測(cè)：
- $ igel predict -dp path_to_the_new_dataset
參考鏈接：
https://medium.com/@nidhalbacc/machine-learning-without-writing-code-984b238dd890
【本文是51CTO專欄機(jī)構(gòu)“機(jī)器之心”的原創(chuàng)譯文，微信公眾號(hào)“機(jī)器之心( id: almosthuman2014)”】
