Thanks!

Yeah, it would work just fine.

If you want to use the embeddings to a downstream feature based task you can add BERT as a Keras Layer and set the trainable parameters so that it is fine-tuned as part of training the task.

On the other hand, if you just want to extract embeddings you can train your estimator separately and then extract the embeddings by following the steps listed in this post.

I have a working example for using BERT embeddings as a Keras Layer so let me know if you find any issues.