WCF Streaming large data (500MB / 1GB) on a self hosted service

c# .net wcf wcf-data-services
user2003667 picture user2003667 · Jan 23, 2013 · Viewed 42.6k times · Source

I'm currently experiencing an issue trying to send large data using WCF self hosted service (no IIS). Transferring 500MB using streaming results with my service crashing with System.OutOfMemoryException. Is it possible at all to transfer such amount of data?

Here is my WCF configuration:

<system.serviceModel>
<services>
  <service  name="CIService" behaviorConfiguration="CIBehavior">        
    <host>
      <baseAddresses>
        <add baseAddress="net.tcp://localhost:6547/CIService/CIService.svc" />
      </baseAddresses>
    </host>
    <endpoint binding="netTcpBinding" 
        bindingConfiguration="netTcpBindingConfig" 
        behaviorConfiguration="CIBehavior.EndpointBehavior" 
        contract="CIService.ICreatable" />
    <endpoint address="mex" 
        binding="mexHttpBinding" 
        name="mexTcpBinding" 
        contract="IMetadataExchange" />
  </service>
</services>
<serviceHostingEnvironment multippleSiteBindingEnabled="True" />
<bindings>
  <netTcpBinding>
    <binding name="netTcpBindingConfig" closeTimeout="00:01:00" openTimeout="00:01:00" 
        receiveTimeout="01:00:00" sendTimeout="00:10:00" 
        hostNameComparisonMode="StrongWildcard" listenBacklog="10" maxConnections="10"
        maxBufferSize="2147483647" maxBufferPoolSize="2147483647" maxReceivedMessageSize="2147483647" 
        transferMode="Streamed">
      <readerQuotas maxDepth="2147483647" maxStringContentLength="2147483647" maxArrayLength="2147483647"
        maxBytesPerRead="2147483647" maxNameTableCharCount="2147483647" />
    </binding>
  </netTcpBinding>
</bindings>

<behaviors>
  <serviceBehaviors>
    <behavior name="CIBehavior">
      <serviceMetadata httpGetEnabled="False" />
      <serviceDebug includeExceptionDetailInFaults="true" />
      <serviceThrottling maxConcurrentCalls="200"  maxConcurrentInstances="2147483647" maxConcurrentSessions="100" />
      <dataContractSerializer maxItemsInObjectGraph="2147483647"/>
    </behavior>
  </serviceBehaviors>
  <endpointBehavior>
    <behavior name="CIBehavior.EndpointBehavior">
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
    </behavior>
  </endpointBehavior>
</behaviors>
</system.serviceModel>

My client configuration:

<system.serviceModel>
<bindings>
  <netTcpBinding>
    <binding name="NetTcpBinding_ICreatable" 
             closeTimeout="00:01:00" openTimeout="00:01:00" 
             receiveTimeout="01:00:00" sendTimeout="00:10:00" 
             transactionFlow="false" 
             transferMode="Streamed" 
             transactionProtocol="OleTransactions" 
             hostNameComparisonMode="StrongWildcard" 
             listenBacklog="10" 
             maxBufferPoolSize="2147483647" 
             maxBufferSize="2147483647" 
             maxConnections="10"
             maxReceivedMessageSize ="2147483647">
      <readerQuotas
        maxDepth="2147483647" 
        maxStringContentLength="2147483647" 
        maxArrayLength="2147483647" 
        maxBytesPerRead="2147483647" 
        maxNameTableCharCount="2147483647" />
      <reliableSession ordered="true" inactivityTimeout="00:10:00" anabled="false" />
    </binding>
  </netTcpBinding>
</bindings>
<client>
  <endpoint name="NetTcpBinding_ICreatable" 
      address="net.tcp://localhost:6547/CIService/CIService.svc" 
      binding="netTcpBinding" 
      bindingConfiguration="NetTcpBinding_ICreatable" 
      behaviorConfiguration="CIBehavior.EndpointBehavior" 
      contract="ICreatable" />
</client>
<behaviors>
  <endpointBehavior>
    <behavior name="CIBehavior.EndpointBehavior">
      <dataContractSerializer maxItemsInObjectGraph="2147483647" />
    </behavior>
  </endpointBehavior>
</behaviors>
</system.serviceModel>

Answer

DerekGn picture DerekGn · Feb 4, 2013

You dont need maxBufferSize or maxBufferPoolSize set so high these are possibly causing your out of memory exception. The defaults should be fine.

Check out Large Data and Streaming on MSDN, specifically the section Special Security Considerations for Large Data this piece of the text is important

The MaxBufferSize property is required to constrain the memory that WCF buffers. It is important to set this to a safe value (or keep it at the default value) when streaming. For example, suppose your service must receive files up to 4 GB in size and store them on the local disk. Suppose also that your memory is constrained in such a way that you can only buffer 64 KB of data at a time. Then you would set the MaxReceivedMessageSize to 4 GB and MaxBufferSize to 64 KB. Also, in your service implementation, you must ensure that you read only from the incoming stream in 64-KB chunks and do not read the next chunk before the previous one has been written to disk and discarded from memory.

I put together a very simple example of streaming data from a self hosted service to a console client. To keep the post short I only added the server code and part of the client.

The service contract

using System.IO;
using System.ServiceModel;

namespace Service
{
    [ServiceContract]
    public interface IStream
    {
        [OperationContract]
        Stream GetLargeObject();
    }
}

The service implementation

using System;
using System.IO;
using System.ServiceModel;

namespace Service
{
   [ServiceBehavior]
   public class StreamService : IStream
   {
       public Stream GetLargeObject()
       {
           // Add path to a big file, this one is 2.5 gb
           string filePath = Path.Combine(Environment.CurrentDirectory, "C:\\Temp\\BFBC2_PC_Client_R11_795745_Patch.exe");

        try
        {
            FileStream imageFile = File.OpenRead(filePath);
            return imageFile;
        }
        catch (IOException ex)
        {
            Console.WriteLine(String.Format("An exception was thrown while trying to open file {0}", filePath));
            Console.WriteLine("Exception is: ");
            Console.WriteLine(ex.ToString());
            throw;
        }
    }
 }
}

The service main

using System;
using System.ServiceModel;

namespace Service
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                using (var serviceHost = new ServiceHost(typeof(StreamService)))
                {
                    serviceHost.Open();

                    Console.WriteLine("Press Any Key to end");
                    Console.ReadKey();
                }
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex.ToString());
            }
        }
    }
}

The service app.config

<?xml version="1.0"?>
<configuration>
  <startup>
    <supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0"/>
  </startup>

  <system.serviceModel>
    <behaviors>
      <serviceBehaviors>
        <behavior name="StreamServiceBehavior">
          <serviceMetadata httpGetEnabled="True" />
        </behavior>
      </serviceBehaviors>
    </behaviors>
    <bindings>
      <netTcpBinding>
        <binding name="NewBinding0" transferMode="Streamed"/>
      </netTcpBinding>
    </bindings>
    <services>
      <service behaviorConfiguration="StreamServiceBehavior" name="Service.StreamService">
        <endpoint address="net.tcp://localhost:9000/streamserver" binding="netTcpBinding"
          bindingConfiguration="NewBinding0" bindingName="" contract="Service.IStream" />
        <endpoint address="mex" binding="mexHttpBinding"
          contract="IMetadataExchange" />
        <host>
          <baseAddresses>
            <add baseAddress="http://localhost:8080/StreamService" />
          </baseAddresses>
        </host>
      </service>
    </services>
  </system.serviceModel>
</configuration>

Launch the service, may need to run under admin account to open the socket. Create a client console application and add a service reference using the url http:// localhost:8080 / StreamService, using Service as the namespace for the generated client.

The client main

using System;
using System.IO;
using Client.Service;

namespace Client
{
    class Program
    {
        static void Main(string[] args)
        {
            try
            {
                using (StreamClient streamClient = new StreamClient())
                {
                    streamClient.Open();

                    using (FileStream fileStream = new FileStream("c:\\temp\\bigfile.exe",FileMode.Create))
                    {
                        streamClient.GetLargeObject().CopyTo(fileStream);    
                    }
                }

                Console.WriteLine("Press any key to end");
                Console.ReadKey();
            }
            catch (Exception ex)
            {
                Console.WriteLine(ex);
            }
        }
    }
}

The generated client config file will need to be modified slightly, increase receiveTimeout and set maxReceivedMessageSize="4294967295"

<system.serviceModel>
    <bindings>
        <netTcpBinding>
            <binding name="NetTcpBinding_IStream" closeTimeout="00:01:00"
                openTimeout="00:01:00" receiveTimeout="00:30:00" sendTimeout="00:01:00"
                transactionFlow="false" transferMode="Streamed" transactionProtocol="OleTransactions"
                hostNameComparisonMode="StrongWildcard" listenBacklog="10"
                maxBufferPoolSize="524288" maxBufferSize="65536" maxConnections="10"
                maxReceivedMessageSize="4294967295">
                <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384"
                    maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                <reliableSession ordered="true" inactivityTimeout="00:10:00"
                    enabled="false" />
                <security mode="Transport">
                    <transport clientCredentialType="Windows" protectionLevel="EncryptAndSign" />
                    <message clientCredentialType="Windows" />
                </security>
            </binding>
        </netTcpBinding>
    </bindings>
    <client>
        <endpoint address="net.tcp://localhost:9000/streamserver" binding="netTcpBinding"
            bindingConfiguration="NetTcpBinding_IStream" contract="Service.IStream"
            name="NetTcpBinding_IStream">

        </endpoint>
    </client>
</system.serviceModel>

Launch the service then the client. It will stream a large file without issue.

Related questions

WCF Service Client: The content type text/html; charset=utf-8 of the response message does not match the content type of the binding

I've got a WCF Service running on my local IIS server. I've added it as a service reference to a C# Website Project and it adds fine and generates the proxy classes automatically. However, when I try and call any …

Read More
Error 5 : Access Denied when starting windows service

I'm getting this error when I try to start a windows service I've created in C#: My Code so far: private ServiceHost host = null; public RightAccessHost() { InitializeComponent(); } protected override void OnStart(string[] args) { host = new ServiceHost(typeof(RightAccessWcf)); host.Open(); } …

Read More
How can I ignore a property when serializing using the DataContractSerializer?

I am using .NET 3.5SP1 and DataContractSerializer to serialize a class. In SP1, they changed the behavior so that you don't have to include DataContract/DataMember attributes on the class and it will just serialize the entire thing. This is …

Read More