Using restclient for streaming multipart uploads


Anthony Rowlands <draftomatic@...>
 

Hello RestClient list,

I am trying to use the RestClient library to do a multipart upload from a StringIO, not from a file. After scouring the code, it seems that this is not possible. It should be something like this (although this example is obviously wrong):

response = RestClient.post(url, :payload => {
        :multipart => true,
        :content_type => 'multipart/form-data',
        :source => myStringIO
      }, :headers => {
          :cookies => {
            'JSESSIONID' => @sessionid
          }
        }
      )

In payload.rb, in Multipart.build_stream, v is the StringIO I passed to RestClient.post, and it leads to a call to create_file_field (StringIO seems to respond to :path even though this isn't documented in the rubydocs), but StringIO.path is always nil for me, which is causing problems:

Error: #<TypeError: can't convert nil into String>
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `basename'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `create_file_field'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:179:in `build_stream'

Can anyone help me with this? Is it possible to do a streaming multipart post from a StringIO?

Thanks,
Anthony R


Archiloque <code@...>
 

Shortest way would be to add a path method to your StringIO and RestClient will use it as a File

Please tell me if it works, this way I could add something about it in the code as it's not the first time the questions is asked.

A.

Le 2 nov. 2011 à 21:42, Anthony Rowlands a écrit :

Hello RestClient list,

I am trying to use the RestClient library to do a multipart upload from a StringIO, not from a file. After scouring the code, it seems that this is not possible. It should be something like this (although this example is obviously wrong):

response = RestClient.post(url, :payload => {
        :multipart => true,
        :content_type => 'multipart/form-data',
        :source => myStringIO
      }, :headers => {
          :cookies => {
            'JSESSIONID' => @sessionid
          }
        }
      )

In payload.rb, in Multipart.build_stream, v is the StringIO I passed to RestClient.post, and it leads to a call to create_file_field (StringIO seems to respond to :path even though this isn't documented in the rubydocs), but StringIO.path is always nil for me, which is causing problems:

Error: #<TypeError: can't convert nil into String>
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `basename'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `create_file_field'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:179:in `build_stream'

Can anyone help me with this? Is it possible to do a streaming multipart post from a StringIO?

Thanks,
Anthony R


Anthony Rowlands <draftomatic@...>
 

Thanks for the reply!

Actually, #path is defined on StringIO already, but it always returns nil (see this doc). Can I override this method? Never done that before in Ruby...

Anthony


On Wed, Nov 2, 2011 at 2:53 PM, Archiloque <code@...> wrote:
Shortest way would be to add a path method to your StringIO and RestClient will use it as a File

Please tell me if it works, this way I could add something about it in the code as it's not the first time the questions is asked.

A.

Le 2 nov. 2011 à 21:42, Anthony Rowlands a écrit :

Hello RestClient list,

I am trying to use the RestClient library to do a multipart upload from a StringIO, not from a file. After scouring the code, it seems that this is not possible. It should be something like this (although this example is obviously wrong):

response = RestClient.post(url, :payload => {
        :multipart => true,
        :content_type => 'multipart/form-data',
        :source => myStringIO
      }, :headers => {
          :cookies => {
            'JSESSIONID' => @sessionid
          }
        }
      )

In payload.rb, in Multipart.build_stream, v is the StringIO I passed to RestClient.post, and it leads to a call to create_file_field (StringIO seems to respond to :path even though this isn't documented in the rubydocs), but StringIO.path is always nil for me, which is causing problems:

Error: #<TypeError: can't convert nil into String>
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `basename'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `create_file_field'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:179:in `build_stream'

Can anyone help me with this? Is it possible to do a streaming multipart post from a StringIO?

Thanks,
Anthony R



Anthony Rowlands <draftomatic@...>
 

Okay, I am trying this now:

def io.path
  return 'upload.lsz'
end

response = RestClient.post(url, {:payload => {
  :multipart => true,
  :content_type => 'multipart/form-data',
  :source => io
}}, {:cookies => {
    'JSESSIONID' => @sessionid
  }
})

This seems to work, although my server isn't able to read the uploaded file contents. It does recognize the content-type as "multipart/form-data; boundary=993865", however... Not sure what happened. Do you know of a way to see what was actually written to the stream?

Anthony

On Wed, Nov 2, 2011 at 3:08 PM, Anthony Rowlands <draftomatic@...> wrote:
Thanks for the reply!

Actually, #path is defined on StringIO already, but it always returns nil (see this doc). Can I override this method? Never done that before in Ruby...

Anthony


On Wed, Nov 2, 2011 at 2:53 PM, Archiloque <code@...> wrote:
Shortest way would be to add a path method to your StringIO and RestClient will use it as a File

Please tell me if it works, this way I could add something about it in the code as it's not the first time the questions is asked.

A.

Le 2 nov. 2011 à 21:42, Anthony Rowlands a écrit :

Hello RestClient list,

I am trying to use the RestClient library to do a multipart upload from a StringIO, not from a file. After scouring the code, it seems that this is not possible. It should be something like this (although this example is obviously wrong):

response = RestClient.post(url, :payload => {
        :multipart => true,
        :content_type => 'multipart/form-data',
        :source => myStringIO
      }, :headers => {
          :cookies => {
            'JSESSIONID' => @sessionid
          }
        }
      )

In payload.rb, in Multipart.build_stream, v is the StringIO I passed to RestClient.post, and it leads to a call to create_file_field (StringIO seems to respond to :path even though this isn't documented in the rubydocs), but StringIO.path is always nil for me, which is causing problems:

Error: #<TypeError: can't convert nil into String>
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `basename'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `create_file_field'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:179:in `build_stream'

Can anyone help me with this? Is it possible to do a streaming multipart post from a StringIO?

Thanks,
Anthony R




Archiloque <code@...>
 

you can enable some logging to see what is happening, like RestClient.log = 'STDOUT'

A.


Le 2 nov. 2011 à 22:15, Anthony Rowlands a écrit :

Okay, I am trying this now:

def io.path
  return 'upload.lsz'
end

response = RestClient.post(url, {:payload => {
  :multipart => true,
  :content_type => 'multipart/form-data',
  :source => io
}}, {:cookies => {
    'JSESSIONID' => @sessionid
  }
})

This seems to work, although my server isn't able to read the uploaded file contents. It does recognize the content-type as "multipart/form-data; boundary=993865", however... Not sure what happened. Do you know of a way to see what was actually written to the stream?

Anthony

On Wed, Nov 2, 2011 at 3:08 PM, Anthony Rowlands <draftomatic@...> wrote:
Thanks for the reply!

Actually, #path is defined on StringIO already, but it always returns nil (see this doc). Can I override this method? Never done that before in Ruby...

Anthony


On Wed, Nov 2, 2011 at 2:53 PM, Archiloque <code@...> wrote:
Shortest way would be to add a path method to your StringIO and RestClient will use it as a File

Please tell me if it works, this way I could add something about it in the code as it's not the first time the questions is asked.

A.

Le 2 nov. 2011 à 21:42, Anthony Rowlands a écrit :

Hello RestClient list,

I am trying to use the RestClient library to do a multipart upload from a StringIO, not from a file. After scouring the code, it seems that this is not possible. It should be something like this (although this example is obviously wrong):

response = RestClient.post(url, :payload => {
        :multipart => true,
        :content_type => 'multipart/form-data',
        :source => myStringIO
      }, :headers => {
          :cookies => {
            'JSESSIONID' => @sessionid
          }
        }
      )

In payload.rb, in Multipart.build_stream, v is the StringIO I passed to RestClient.post, and it leads to a call to create_file_field (StringIO seems to respond to :path even though this isn't documented in the rubydocs), but StringIO.path is always nil for me, which is causing problems:

Error: #<TypeError: can't convert nil into String>
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `basename'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `create_file_field'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:179:in `build_stream'

Can anyone help me with this? Is it possible to do a streaming multipart post from a StringIO?

Thanks,
Anthony R





Anthony Rowlands <draftomatic@...>
 

Logging is showing this:

RestClient.post "http://localhost:8080/LightStanza/lsz-upload?projectKey=20", 30929 byte(s) length, "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"30929", "Content-Type"=>"multipart/form-data; boundary=89514", "Cookie"=>"JSESSIONID=2DC341E5B9A96F8AE9DDB94267BA47F4"

Does this look okay to you? 30kb sounds about right for the data I'm testing with. It could be server-side problem at this point, but without seeing the actual stream it's hard to tell.

Also, do you see a problem with generating a random number for the file boundary? Seems like it's possible that this exact number would show up in the file content, which goes against the RFC.

Thanks again,
Anthony

On Wed, Nov 2, 2011 at 3:23 PM, Archiloque <code@...> wrote:
you can enable some logging to see what is happening, like RestClient.log = 'STDOUT'

A.


Le 2 nov. 2011 à 22:15, Anthony Rowlands a écrit :

Okay, I am trying this now:

def io.path
  return 'upload.lsz'
end

response = RestClient.post(url, {:payload => {
  :multipart => true,
  :content_type => 'multipart/form-data',
  :source => io
}}, {:cookies => {
    'JSESSIONID' => @sessionid
  }
})

This seems to work, although my server isn't able to read the uploaded file contents. It does recognize the content-type as "multipart/form-data; boundary=993865", however... Not sure what happened. Do you know of a way to see what was actually written to the stream?

Anthony

On Wed, Nov 2, 2011 at 3:08 PM, Anthony Rowlands <draftomatic@...> wrote:
Thanks for the reply!

Actually, #path is defined on StringIO already, but it always returns nil (see this doc). Can I override this method? Never done that before in Ruby...

Anthony


On Wed, Nov 2, 2011 at 2:53 PM, Archiloque <code@...> wrote:
Shortest way would be to add a path method to your StringIO and RestClient will use it as a File

Please tell me if it works, this way I could add something about it in the code as it's not the first time the questions is asked.

A.

Le 2 nov. 2011 à 21:42, Anthony Rowlands a écrit :

Hello RestClient list,

I am trying to use the RestClient library to do a multipart upload from a StringIO, not from a file. After scouring the code, it seems that this is not possible. It should be something like this (although this example is obviously wrong):

response = RestClient.post(url, :payload => {
        :multipart => true,
        :content_type => 'multipart/form-data',
        :source => myStringIO
      }, :headers => {
          :cookies => {
            'JSESSIONID' => @sessionid
          }
        }
      )

In payload.rb, in Multipart.build_stream, v is the StringIO I passed to RestClient.post, and it leads to a call to create_file_field (StringIO seems to respond to :path even though this isn't documented in the rubydocs), but StringIO.path is always nil for me, which is causing problems:

Error: #<TypeError: can't convert nil into String>
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `basename'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:202:in `create_file_field'
C:/PROGRA~2/Google/GOOGLE~2/Plugins/Light Stanza/lib/restclient/payload.rb:179:in `build_stream'

Can anyone help me with this? Is it possible to do a streaming multipart post from a StringIO?

Thanks,
Anthony R






Archiloque <code@...>
 


Le 2 nov. 2011 à 22:50, Anthony Rowlands a écrit :

Logging is showing this:

RestClient.post "http://localhost:8080/LightStanza/lsz-upload?projectKey=20", 30929 byte(s) length, "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"30929", "Content-Type"=>"multipart/form-data; boundary=89514", "Cookie"=>"JSESSIONID=2DC341E5B9A96F8AE9DDB94267BA47F4"

Does this look okay to you? 30kb sounds about right for the data I'm testing with. It could be server-side problem at this point, but without seeing the actual stream it's hard to tell.

Also, do you see a problem with generating a random number for the file boundary? Seems like it's possible that this exact number would show up in the file content, which goes against the RFC.

seems difficult as you seem to have tried several time, you could perhaps use a simple logging proxy en your machine to see what exactly happen ?

A.


Anthony Rowlands <draftomatic@...>
 

I just printed the raw request body on my server, and it is empty... The correct headers were sent, but there is nothing in the body.

Are you sure RestClient can handle this? You said it's not the first time someone has tried... Has anyone else gotten it to work? 

Anthony


On Wed, Nov 2, 2011 at 4:30 PM, Archiloque <code@...> wrote:

Le 2 nov. 2011 à 22:50, Anthony Rowlands a écrit :

Logging is showing this:

RestClient.post "http://localhost:8080/LightStanza/lsz-upload?projectKey=20", 30929 byte(s) length, "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"30929", "Content-Type"=>"multipart/form-data; boundary=89514", "Cookie"=>"JSESSIONID=2DC341E5B9A96F8AE9DDB94267BA47F4"

Does this look okay to you? 30kb sounds about right for the data I'm testing with. It could be server-side problem at this point, but without seeing the actual stream it's hard to tell.

Also, do you see a problem with generating a random number for the file boundary? Seems like it's possible that this exact number would show up in the file content, which goes against the RFC.

seems difficult as you seem to have tried several time, you could perhaps use a simple logging proxy en your machine to see what exactly happen ?

A.



Archiloque <code@...>
 

I think so but unfortunately I don't have tile now to retrieve it, for a quickfix you could write the StringIO to a TempFile and use it as a parameter

A.


Le 2 nov. 2011 à 23:44, Anthony Rowlands a écrit :

I just printed the raw request body on my server, and it is empty... The correct headers were sent, but there is nothing in the body.

Are you sure RestClient can handle this? You said it's not the first time someone has tried... Has anyone else gotten it to work? 

Anthony

On Wed, Nov 2, 2011 at 4:30 PM, Archiloque <code@...> wrote:

Le 2 nov. 2011 à 22:50, Anthony Rowlands a écrit :

Logging is showing this:

RestClient.post "http://localhost:8080/LightStanza/lsz-upload?projectKey=20", 30929 byte(s) length, "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"30929", "Content-Type"=>"multipart/form-data; boundary=89514", "Cookie"=>"JSESSIONID=2DC341E5B9A96F8AE9DDB94267BA47F4"

Does this look okay to you? 30kb sounds about right for the data I'm testing with. It could be server-side problem at this point, but without seeing the actual stream it's hard to tell.

Also, do you see a problem with generating a random number for the file boundary? Seems like it's possible that this exact number would show up in the file content, which goes against the RFC.

seems difficult as you seem to have tried several time, you could perhaps use a simple logging proxy en your machine to see what exactly happen ?

A.




Anthony Rowlands <draftomatic@...>
 

I ran out of time myself, so hacked something together that is good enough for now with a straight stream instead of a multipart stream.

If you get ever find time to use StringIO with multipart, let me know, I'd be happy to have the solution since I pretty much spent all day today trying to get it working. =D This will suffice for the time being...

Thank you very much for your help!

Anthony R


On Wed, Nov 2, 2011 at 4:56 PM, Archiloque <code@...> wrote:
I think so but unfortunately I don't have tile now to retrieve it, for a quickfix you could write the StringIO to a TempFile and use it as a parameter

A.


Le 2 nov. 2011 à 23:44, Anthony Rowlands a écrit :

I just printed the raw request body on my server, and it is empty... The correct headers were sent, but there is nothing in the body.

Are you sure RestClient can handle this? You said it's not the first time someone has tried... Has anyone else gotten it to work? 

Anthony

On Wed, Nov 2, 2011 at 4:30 PM, Archiloque <code@...> wrote:

Le 2 nov. 2011 à 22:50, Anthony Rowlands a écrit :

Logging is showing this:

RestClient.post "http://localhost:8080/LightStanza/lsz-upload?projectKey=20", 30929 byte(s) length, "Accept"=>"*/*; q=0.5, application/xml", "Accept-Encoding"=>"gzip, deflate", "Content-Length"=>"30929", "Content-Type"=>"multipart/form-data; boundary=89514", "Cookie"=>"JSESSIONID=2DC341E5B9A96F8AE9DDB94267BA47F4"

Does this look okay to you? 30kb sounds about right for the data I'm testing with. It could be server-side problem at this point, but without seeing the actual stream it's hard to tell.

Also, do you see a problem with generating a random number for the file boundary? Seems like it's possible that this exact number would show up in the file content, which goes against the RFC.

seems difficult as you seem to have tried several time, you could perhaps use a simple logging proxy en your machine to see what exactly happen ?

A.