معرفی شرکت ها
el-compX-scraper-0.1.7
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
This package scrapes features of elements and compounds from specified websites
| ویژگی | مقدار |
|---|---|
| سیستم عامل | - |
| نام فایل | el-compX-scraper-0.1.7 |
| نام | el-compX-scraper |
| نسخه کتابخانه | 0.1.7 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Bilal Armiyawo |
| ایمیل نویسنده | - |
| آدرس صفحه اصلی | https://github.com/barmiy01/Webscraping_Project.git |
| آدرس اینترنتی | https://pypi.org/project/el-compX-scraper/ |
| مجوز | MIT |
# Webscraping Project
## Predicting the Band Gap of Compounds using Elemental Descriptors
## Abstract
It has been shown that there is a predictive relationship
between 'known discrete scalar descriptors associated with crystal and electronic structure and observed properties of materials'.
However, the property space of these materials is of high dimensionality which highlights the complex nature of predictive models at the fundamental level. Additionally, the elemental descriptors at this level have a certain degree of co-dependence which makes prediction even more complicated. It has been demonstrated using data reduction methods that the property space of observable material properties can be diminished. In this project, a dataset of elements and some of their corresponding elemental descriptors have been collected using webscraping techniques. The elemental descriptors/features were limited to five since it has been shown that it is possible to predict band gap energies using only five (5) elemental descriptors.
## Motivation
Recent advances in material science and engineering have been focused on
producing rational design rules and principles for material fabrication.
The development of these design rules have huge implications for various fields such as crystal engineering, opto-electronics and photonics . In this regard, considerable attempts have been made to utilize already accumulated datasets to create models that facilitate the prediction of various material properties using machine learning techniques. Despite recent advances in this field, there is a dearth of machine-learning-based models to predict band gap energies.
## Methodology
Implementing python libraries such as selenium and pandas, a database of elements with elemental descriptors have been extracted. The code was written to function in a multifaceted way as detailed below:
* Open the desired website containing information on the elements or compound.
* Extract specific information on the attributes of the element or compound.
* To do this, the python codes ```OQMD_new_version.py``` and ```periodic_table_new.py``` utilized. Within this code, a scraper class has been defined with the following attributes:
* `n` whose definition depends on the code being executed. For ```OQMD_new_version```, `n` is
an integer that defines the number of pages to extract data from while `n` in ```periodic_table_new.py``` defines the number of elements to extract data from. In this case data from 60 elements where extracted.
* `root` defines the target url where data is being extracted.
* `features` initialises a dictionary with keys that define the necessary data to be extrach for ecah element or compound.
It takes either of these forms: ```features = {'Element_Name':[], 'Atomic_Number':[], 'Electronegativity':[], 'Boiling_Point':[]}``` or
```features = {'Name':[], 'Spacegroup':[], 'Volume':[], 'Band_gap':[]}```
These codes contain the function ```extract_data()``` that carry out the extraction of elements and compounds data.
* Specifically, ```extract_data(to_DF)``` depends on ```to_DF``` which only accepts boolean values. to_DF determines whether or not the data will be converted into a Data Frame.
* Convert the result into a dataframe for further processing us
* Using Pandas, the extracted data was stored as a data frame using the ```convert_to_DF()``` function which depends on the following variables: ```data_name```, ```file_out```, ```to_csv```.
* Clean the data
* For the extracted elements data, the boiling point of elements was extracted in both °C and Kelvin(K) with the boiling point in °C within parentheses. For consistency, we delete all values in parentheses leaving values in K. To do this, we import the module ```regex``` and implement the following code :
```re.sub(r'\([^)]*\)', '', '[filename]')```
* Import as an SQL database
* Implementing the python modules ```psycopg2``` and ```SQLAlchemy``` , the output data was imported to SQL in tabular form.
Specifically, the main python library used for the extraction of data was selenium. Pandas was used to convert the raw data into a desired output (i.e. a csv file).
## Setting up venv
* Using anaconda3 set up a virtual environment (venv) while meeting necessary code requirements. All necessary requirements can be found in the file ```requirements.txt```.
```source activate [env name]```
```pip install requirements.txt```
# Installing and Running
* To install this package:
```pip install el-compX-scraper```
## Running the Project
1. Within python import the necessary modules:
```
import scraper
from scaper.OQMD_new_version import CompoundScraper
from scraper.periodic_table_new import PeriodicTableScraper
```
2. To instantiate a scraper object, we can implement the CompoundScraper class. Hence,
```
root = "http://oqmd.org/api/search#apisearchresult"
features = {'Name':[], 'Spacegroup':[], 'Volume':[], 'Band_gap':[]}
scraper = CompoundScraper(n=1, root=root, list=[], features=features)
scraper.extract_data()
```
Similarly for the PeriodicTableScraper,
```
root = "https://pubchem.ncbi.nlm.nih.gov/periodic-table/#view=list"
features = {'Element_Name':[], 'Atomic_Number':[], 'Electronegativity':[], 'Boiling_Point':[]}
scraper = PeriodicTableScraper(n=5, root=root, list=[], features=features)
scraper.extract_data(to_DF=True)
```
3. We can also example script which instantiates specific scraper objects depending on the url. Thus to execute this,
```import scraper.example```
نحوه نصب
نصب پکیج whl el-compX-scraper-0.1.7:
pip install el-compX-scraper-0.1.7.whl
نصب پکیج tar.gz el-compX-scraper-0.1.7:
pip install el-compX-scraper-0.1.7.tar.gz