Skip to main content

Beaver 🦫

Beaver is DSL for Machine Learning in live data. It's purpose is to simplify the process of data retrieval and preprocessing, model training, model prediction and output display. It uses multiple tools to achieve this:

  • Kafka
  • Quixstreams
  • River
  • Plotly
  • Dash
  • Docker
  • TextX
  • Jinja

Quickstart​

To download the project run :

git clone https://github.com/deepblue597/thesis.git

When you finished downloading go to the repository by running

cd thesis

Create a new virutal environment:

python -m venv <YOUR-VENV-NAME>

Activate the environment

Depending on your shell:

Bash:

source <YOUR-VENV-NAME>\bin\activate

PowerShell:

<YOUR-VENV-NAME>\Scripts\activate

To download all the necessary libraries run:

pip install -e .

Open a Text Editor of your choice and create a .bvr file. If you are unsure how to structure a .bvr you can check the docs or use one of the examples that are provided in the examples folder.

When you have added the necessary entities and constructed your pipeline:

  1. Open a terminal
  2. Run
python3 beaver_cli.py generate --input <PATH-TO-YOUR-METAMODEL> --output <PATH-TO-THE-GENERATED-FILE>

Note: Instead of python3 you may need to run python depending on the installment of python in your machine

  1. If there are no errors, your python file will have been generated at <PATH-TO-THE-GENERATED-FILE>

  2. Run your python file python3 <PATH-TO-THE-GENERATED-FILE>

Kafka setup​

If you don't have a kafka setup, Beaver provides one with 3 brokers, 3 controllers and a kafka UI provided by provectuslabs To set it up

  1. Go to kafka_proj folder cd kafka_proj

  2. Run the docker compose file docker compose up -d

  3. Your UI will be on localhost:8080 the brokers at which you can connect are on localhost:49092 , localhost:39092 and localhost:29092.