Tensorflow on multiple GPUs with FastAPI

Tensorflow on multiple GPUs with FastAPI

tab of content

• Tensorflow on multiple GPUs with FastAPI
• 1. Train a model on multiple GPUs
• 2. Get the data batch from Generator
• 3. use early stopping and save the best session
• 4. Deploy the saved model with FastAPI library
• • 1). add routers for training/evaluating/predicting
  • 2). apply BaseModel on client input to define how is needed for the each api call:
  • 3). call APIs to train/evaluate/predict
• 5. deploy your model with tensorflow serving
• • 1). save your model with desired format
  • 2). serve your saved model
• Conclusion

---

The article will be walking through how to train a tf model on multiple GPU environment and how to deploy it with a asynchronous library FastAPI and tensorflow serving. The details can be found on github: https://github.com/victorbai2/TFpackageText
Another my article link on medium:
https://medium.com/@what20051125/tensorflow-on-multiple-gpus-with-fastapi-6306b7b1e0ba

---

1. Train a model on multiple GPUs

To train a model on multi GPUs, please check below on multi_GPU_training function:

2. Get the data batch from Generator

You can define how much batches you would like to feed into model:

3. use early stopping and save the best session

4. Deploy the saved model with FastAPI library 1). add routers for training/evaluating/predicting

2). apply BaseModel on client input to define how is needed for the each api call:

3). call APIs to train/evaluate/predict

fastAPI provides the swagger UI where you can review and call apis

FastAPI works quite friendly with async / await which most of library does not support very well.
The sample response for calling api:

curl -X 'POST' \
'http://localhost:5000/api/pred' \
-H 'accept: application/json' \
-H 'Content-Type: application/json' \
-d '{
"inquiry": [
"sentense1", "sentense2","sentense3",
]
}'

you can add a list of queries in field “inquiry”
response:

5. deploy your model with tensorflow serving 1). save your model with desired format

2). serve your saved model

---

Conclusion

• We demenstrated how to train model on multiple GPU env.
• We showcased how to generate batches of data for trainin.
• We implemented FastAPI service interface for training/evalutating/predicting.
• we briefly cover how to serve your model with tensorflow servering.
• The full code can be found in github: https://github.com/victorbai2/TFpackageText