yaml.dump adding unwanted newlines in multiline strings

Question

I have a multiline string:

>>> import credstash
>>> d = credstash.getSecret('alex_test_key', region='ap-southeast-2')

To see the raw data (first 162 characters):

>>> credstash.getSecret('alex_test_key', region='ap-southeast-2')[0:162]
u'-----BEGIN RSA PRIVATE KEY-----\nMIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx\nxk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45\n'

And:

>>> print d[0:162]                                                                                                                                                                                          
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45

I write to a YAML file:

>>> import yaml
>>> with open('foo.yaml', 'w') as f:                                                                                                                                                                        
...     yaml.dump(d, f, default_flow_style=False, explicit_start=True)
... 

Now it looks like this:

$ head -5 foo.yaml 
--- !!python/unicode '-----BEGIN RSA PRIVATE KEY-----

  MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx

  xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45

i.e. each line has two newlines.

Now if I read it back into a string I see that all is okay in the round-trip:

>>> with open('foo.yaml', 'r') as f:
...     d = yaml.load(f)
... 
>>> print d[0:162]
-----BEGIN RSA PRIVATE KEY-----
MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45

(I don't understand why however.)

My real problem is that if humans read this YAML file they will probably assume, as I did, that my program has broken the formatting of the private key file.

Is there a way to use yaml.dump so as to output something without the additional newline characters?

Answer 1

If that is the only thing going into your YAML file then you can dump with the option default_style='|' which gives you block style literal for all of your scalars (probably not what you want).

Your string, contains no special characters (that need \ escaping and double quotes), because of the newlines PyYAML decides to represented single quoted. In single quoted style a double newline is the way to represent a single newline that occurred in string that is represented. This gets "undone" on loading, but is indeed not very readable.

If you want to get the block style literals on an individual basis, you can do multiple things:

• adapt the Representer to output all strings with embedded newlines using the literal scalar block style (assuming they don't need \ escaping of special characters, which will force double quotes)

  import sys
  import yaml
  
  x = u"""\
  -----BEGIN RSA PRIVATE KEY-----
  MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
  xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45
  ...
  """
  
  yaml.SafeDumper.org_represent_str = yaml.SafeDumper.represent_str
  
  def repr_str(dumper, data):
      if '\n' in data:
          return dumper.represent_scalar(u'tag:yaml.org,2002:str', data, style='|')
      return dumper.org_represent_str(data)
  
  yaml.add_representer(str, repr_str, Dumper=yaml.SafeDumper)
  
  yaml.safe_dump(dict(a=1, b='hello world', c=x), sys.stdout)
  

• make a subclass of string, that has its special representer. You should be able to take the code for that from here, here and here:

  import sys
  import yaml
  
  class PSS(str):
      pass
  
  x = PSS("""\
  -----BEGIN RSA PRIVATE KEY-----
  MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
  xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45
  ...
  """)
  
  def pss_representer(dumper, data):
          style = '|'
          # if sys.versioninfo < (3,) and not isinstance(data, unicode):
          #     data = unicode(data, 'ascii')
          tag = u'tag:yaml.org,2002:str'
          return dumper.represent_scalar(tag, data, style=style)
  
  yaml.add_representer(PSS, pss_representer, Dumper=yaml.SafeDumper)
  
  yaml.safe_dump(dict(a=1, b='hello world', c=x), sys.stdout)
  

• use ruamel.yaml:

  import sys
  from ruamel.yaml import YAML
  from ruamel.yaml.scalarstring import PreservedScalarString as pss
  
  x = pss("""\
  -----BEGIN RSA PRIVATE KEY-----
  MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
  xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45
  ...
  """)
  
  yaml = YAML()
  
  yaml.dump(dict(a=1, b='hello world', c=x), sys.stdout)
  

All of these give:

a: 1
b: hello world
c: |
  -----BEGIN RSA PRIVATE KEY-----
  MIIEogIBAAKCAQEA6oySC+8/N9VNpk0gJS7Gk8vn9sYN7FhjpAQnoHRqTN/Oaiyx
  xk2AleP2vXpojA/DHldT1JO+o3j56AHD+yfNFFeYvgWKDY35g49HsZZhbyCEAB45
  ...

Please note that it is not necessary to specify default_flow_style=False as the literal scalars can only appear in block style.