MySQL python connector IndexError: bytearray index out of range

Question

I'm inserting some data to a database and most of the queries are inserted correctly but I keep getting at least one random query error.

I'm using Python 3, MySQL 5.6.17 and MySQL python connector 2.1.3 (upgraded after having the same problem with 2.0.2).

The queries are run in a Multiprocessing Pool map_async().

multiprocessing.pool.RemoteTraceback: bytearray index out of range
Traceback (most recent call last):
  File "./../../../my-python-script.py", line 930, in insert_into_database
    mysql_query(mysql_script, values) # <-- My mysql wrapper function
  File "./../../../my-python-script.py", line 297, in mysql_query
    for row in results:
  File "./../../../mysql/connector/cursor.py", line 450, in _execute_iter
    result = next(query_iter)
  File "./../../../mysql/connector/connection.py", line 520, in cmd_query_iter
    yield self._handle_result(self._send_cmd(ServerCmd.QUERY, statements))
  File "./../../../mysql/connector/connection.py", line 405, in _handle_result
    self._socket.recv(), self.python_charset)
  File "./../../../mysql/connector/protocol.py", line 238, in parse_column
    (packet, _) = utils.read_lc_string(packet[4:])  # catalog
  File "./../../../mysql/connector/utils.py", line 199, in read_lc_string
    if buf[0] == 251:  # \xfb
IndexError: bytearray index out of range

The above exception was the direct cause of the following exception:

IndexError: bytearray index out of range

Or sometimes I get (from the "for row in results" line)

  File "./../../../mysql/connector/cursor.py", line 450, in _execute_iter
    result = next(query_iter)
  File "./../../../mysql/connector/connection.py", line 520, in cmd_query_iter
    yield self._handle_result(self._send_cmd(ServerCmd.QUERY, statements))
  File "./../../../mysql/connector/connection.py", line 384, in _handle_result
    elif packet[4] == 0:
IndexError: bytearray index out of range

The above exception was the direct cause of the following exception:

IndexError: bytearray index out of range

My setup is something like

class InsertData:

    def __init__(self):
        with(multiprocessing.Pool(2) as Pool:
            Pool.map_async(self.insert_into_database(),set(1,2,3.....))
            Pool.close()
            Pool.join()

    def insert_into_database(self,values):
        # use the values to do some calculations then insert to database
        mysql_query(mysql_script, values)

def mysql_query(script, values):
    cursor.execute(query, values, multi = True)

And the sql script

'INSERT INTO table1 ( column1 ) VALUES ( "x"  ); '
'SET @table1 = LAST_INSERT_ID(); '
'INSERT INTO table2 ( column1, column2 ) VALUES ( "y", @table1 ); '
'SET @table2 = LAST_INSERT_ID(); '
...

I'm currently looking at the connector.py and the utils code trying to figure what's happening. But this is too advanced for me.

https://github.com/mysql/mysql-connector-python/blob/master/lib/mysql/connector/connection.py#L368

https://github.com/mysql/mysql-connector-python/blob/master/lib/mysql/connector/utils.py#L167

In a desperate attempt I've tried setting the buffered to True https://dev.mysql.com/doc/connector-python/en/connector-python-api-mysqlcursorbuffered.html

I need to read up on bytearrays but I suspect that my query script is causing the problem because I didn't have (I think?) this problem when I ran the queries one connector at a time cursor.execute(query, values, multi = False)

Answer 1

When I send the database connection as described in Accessing a MySQL connection pool from Python multiprocessing the problem disappears.

Something like

mysql_conn = None

def db_conn():
  global mysql_conn
  mysql_conn = connector.connect(...)

class InsertData:

    def __init__(self):
        with(multiprocessing.Pool(2, initializer = db_conn) as Pool:
            Pool.map_async(self.insert_into_database(),set(1,2,3.....))
            Pool.close()
            Pool.join()

    def insert_into_database(self,values):
        # use the values to do some calculations then insert to database
        self.mysql_query(mysql_script, values)

    def mysql_query(script, values):
        cursor = mysql_conn.cursor()
        cursor.execute(query, values, multi = True)