A Telegram bot to monitor and control deep learning experiments
Deep learning training processes can run for many hours/days, and you are not always near your station to see how it's progressing or to make adjustments. Using this Telegram bot you can seamlessly get constant updates and even control your training process, all through your phone.
Works with TensorFlow & Keras (with Keras, all you need is to include a dedicated callback). Should also be good for Pytorch though I did not try. PRs are welcome!
To start interacting with the bot, send
/start. At any time you can send
/helpto see all available commands.
Once you send
/startfrom your Telegram app, the bot will send you updates every epoch:
You can stop getting these automatic updates by sending
/quiet:
To turn updates back on, send
/startagain.
/statusto get the update of the latest epoch:
If your model's convergence plateaus, and you want to change the learning rate of your optimizer, simply send
/setlr:
You can also query the current learning rate any time by sending
\getlr:
To get a convergence plot of the loss, send
/plotfrom the app:
If you want, you can stop your training process from the app. Just send
/stoptrainingand click on the Yes button. With the Keras callback, training is stopped safely. Other operations that needed to happen after training will still take place:
Tested in the following environment: - Python 3.5 - Tensorflow 1.11 - Keras 2.2.4 - Windows OS
$ pip install python-telegram-bot --upgrade
Clone this repository
Add
dl_bot.pyto your project
Add
telegram_bot_callback.pyto your project (optional, only if you use Keras)
First, create a Telegram bot using the Telegram app. It is very simple, just follow the steps in the dedicated section below. Once you have created your bot, search for it and start a conversation with it on the Telegram app.
You can supply a
user_idto restrict interaction with your bot only to a specific user. This is highly recommended. (Instructions on how to find your user id are provided below)
You can either use the Keras callback to automatically interact with the bot, or to customize the interactions yourself.
Note that the bot will start sending messages only after you send it the
/startmessage from the app.
The following block is all you need in order to use the Keras Telegram bot callback: ```python
from dlbot import DLBot from telegrambot_callback import TelegramBotCallback
telegram_token = "TOKEN" # replace TOKEN with your bot's token
telegramuserid = None # replace None with your telegram user id (integer):
bot = DLBot(token=telegramtoken, userid=telegramuserid)
telegramcallback = TelegramBotCallback(bot) ``
Just addtelegramcallback
to the list of callbacks passed to model.fit:
python model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_data=(x_test, y_test), callbacks=[telegram_callback])` That's all, you are good to go!
An example usage is included in
keras_mnist_example.py
If you are using TensorFlow (or using Keras and want to customize interactions with the bot). Start by including the following code in your script: ```python
from dl_bot import DLBot
telegram_token = "TOKEN" # replace TOKEN with your bot's token
telegramuserid = None # replace None with your telegram user id (integer):
bot = DLBot(token=telegramtoken, userid=telegramuserid)
bot.activatebot() ``
Then you will need to implement responses for the/setlr
,/getlr
,/status
,/stoptraining
,/quiet
messages. Also, you will need to send the bot the loss values each epoch in order to use the/plot
command. It is fairly easy to include these responses, a full example is included intfmnist_example.py`
Implementation examples are included for both Keras and TensorFlow in
keras_mnist_example.pyand
tf_mnist_example.py. Both examples include all bot functions over the official keras/tf MNIST examples.
To create a Telegram bot using the Telegram app, follow these steps:
1. Open the Telegram app
2. Search for the BotFather user (@botfather):
3. Start a conversation with BotFather and click on
start