معرفی شرکت ها
Spark-Matcher-0.2.2
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشتر
تبلیغات ما
مشتریان به طور فزاینده ای آنلاین هستند. تبلیغات می تواند به آنها کمک کند تا کسب و کار شما را پیدا کنند.
مشاهده بیشترتوضیحات
Record matching and entity resolution at scale in Spark
| ویژگی | مقدار |
|---|---|
| سیستم عامل | OS Independent |
| نام فایل | Spark-Matcher-0.2.2 |
| نام | Spark-Matcher |
| نسخه کتابخانه | 0.2.2 |
| نگهدارنده | [] |
| ایمیل نگهدارنده | [] |
| نویسنده | Ahmet Bayraktar, Stan Leisink, Frits Hermans |
| ایمیل نویسنده | - |
| آدرس صفحه اصلی | - |
| آدرس اینترنتی | https://pypi.org/project/Spark-Matcher/ |
| مجوز | - |
<!--- BADGES: START --->
[](https://pypi.org/project/spark-matcher/)
[](https://pepy.tech/project/spark-matcher)
[][#docs-package]
[#docs-package]: https://spark-matcher.readthedocs.io/en/latest/
<!--- BADGES: END --->
# Spark-Matcher
Spark-Matcher is a scalable entity matching algorithm implemented in PySpark. With Spark-Matcher the user can easily
train an algorithm to solve a custom matching problem. Spark Matcher uses active learning (modAL) to train a
classifier (Scikit-learn) to match entities. In order to deal with the N^2 complexity of matching large tables, blocking is
implemented to reduce the number of pairs. Since the implementation is done in PySpark, Spark-Matcher can deal with
extremely large tables.
Documentation with examples can be found [here](https://spark-matcher.readthedocs.io/en/latest/).
Developed by data scientists at ING Analytics, www.ing.com.
## Installation
### Normal installation
As Spark-Matcher is intended to be used with large datasets on a Spark cluster, it is assumed that Spark is already
installed. If that is not the case, first install Spark and PyArrow (`pip install pyspark pyarrow`).
Install Spark-Matcher using PyPi:
```
pip install spark-matcher
```
### Install with possibility to create documentation
First clone the Spark-Matcher repository and add `[doc]` like this:
```
pip install ".[doc]"
```
### Install to contribute
Clone this repo and install in editable mode. This also installs PySpark and Jupyterlab:
```
python -m pip install -e ".[dev]"
python setup.py develop
```
## Documentation
Documentation can be created using the following command:
```
make create_documentation
```
## Dependencies
The usage examples in the `examples` directory contain notebooks that run in local mode.
Using the SparkMatcher in cluster mode, requires sending the SparkMatcher package and several other python packages (see spark_requirements.txt) to the executors.
How to send these dependencies, depends on the cluster.
Please read the instructions and examples of Apache Spark on how to do this: https://spark.apache.org/docs/latest/api/python/user_guide/python_packaging.html.
SparkMatcher uses `graphframes` under to hood.
Therefore, depending on the spark version, the correct version of `graphframes` needs to be added to the `external_dependencies` directory and to the configuration of the spark session.
As a default, `graphframes` for spark 3.0 is used in the spark sessions in the notebooks in the `examples` directory.
For a different version, see: https://spark-packages.org/package/graphframes/graphframes.
## Usage
Example notebooks are provided in the `examples` directory.
Using the SparkMatcher to find matches between Spark
dataframes `a` and `b` goes as follows:
```python
from spark_matcher.matcher import Matching
myMatcher = Matcher(spark_session, col_names=['name', 'suburb', 'postcode'])
```
Now we are ready for fitting the Matcher object using 'active learning'; this means that the user has to enter whether a
pair is a match or not. You enter 'y' if a pair is a match or 'n' when a pair is not a match. You will be notified when
the model has converged and you can stop training by pressing 'f'.
```python
myMatcher.fit(a, b)
```
The Matcher is now trained and can be used to predict on all data. This can be the data used for training or new data
that was not seen by the model yet.
```python
result = myMatcher.predict(a, b)
```
نیازمندی
| مقدار | نام |
|---|---|
| - | pandas |
| - | numpy |
| - | scikit-learn |
| - | python-Levenshtein |
| - | thefuzz |
| - | modAL |
| - | pytest |
| - | multipledispatch |
| - | dill |
| - | graphframes |
| - | scipy |
| - | pandas |
| - | numpy |
| - | scikit-learn |
| - | python-Levenshtein |
| - | thefuzz |
| - | modAL |
| - | pytest |
| - | multipledispatch |
| - | dill |
| - | graphframes |
| - | scipy |
| - | pandas |
| - | numpy |
| - | scikit-learn |
| - | python-Levenshtein |
| - | thefuzz |
| - | modAL |
| - | pytest |
| - | multipledispatch |
| - | dill |
| - | graphframes |
| - | scipy |
| - | sphinx |
| - | nbsphinx |
| - | sphinx-rtd-theme |
| - | pyspark |
| - | pyarrow |
| - | jupyterlab |
| - | pandas |
| - | numpy |
| - | scikit-learn |
| - | python-Levenshtein |
| - | thefuzz |
| - | modAL |
| - | pytest |
| - | multipledispatch |
| - | dill |
| - | graphframes |
| - | scipy |
| - | sphinx |
| - | nbsphinx |
| - | sphinx-rtd-theme |
زبان مورد نیاز
| مقدار | نام |
|---|---|
| >=3.7 | Python |
نحوه نصب
نصب پکیج whl Spark-Matcher-0.2.2:
pip install Spark-Matcher-0.2.2.whl
نصب پکیج tar.gz Spark-Matcher-0.2.2:
pip install Spark-Matcher-0.2.2.tar.gz