I am creating an EMR cluster, and using jupyter notebook to run some spark tasks. My tasks die after approximately 1 hour of execution, and the error is:
An error was encountered:
Invalid status code '400' from https://xxx.xx.x.xxx:18888/sessions/0/statements/20 with error payload: "requirement failed: Session isn't active."
My understanding is that it is related to the Livy config livy.server.session.timeout
, but I don't know how I can set it in the bootstrap of the cluster (I need to do it in the bootstrap because the cluster is created with no ssh access)
Thanks a lot in advance
On EMR, livy-conf
is the classification for the properties for livy's livy.conf
file, so when creating an EMR cluster, choose advanced options
with Livy as an application chosen to install, please pass this EMR configuration in the Enter Configuration
field.
[{'classification': 'livy-conf','Properties': {'livy.server.session.timeout':'5h'}}]
/etc/livy/
, and so the config file is at /etc/livy/conf/livy.conf
To verify this,
ec2 key-pair
, Livy and above configkey-pair
, login to the EC2 Master node associated with the cluster ssh -i some-ec2-key-pair.pem [email protected]
/etc/livy/conf
, vim livy.conf
& see the updated value of livy.server.session.timeout