I have a program that writes records to a .TXT file.
The code seems to fail when the file reaches the 5mb mark.
Is this a limitation in ASP, or is it related to how much memory I have in the machine?
Thank you.how are you writing to the file?
if you are appending, it should not be a problem.
Well, here's a snippet...
fp = File.CreateText("c:\AccountFinal.txt")...gather data from 50,000 accounts & other sources, etc here...
fp.WriteLine(BranchNo & AcctType & AcctNo & AcctTitle & totAddr & Phone & SSN & OwnType & SPACE(21) & ActTotals2 & SPACE(41))
fp.Close()
Is there an easier way to do this?
Thanks.
Does it throw an error? If so, what's the error.
Maybe your script times out because it takes to long?
No errors.
Just stops.
It goes through the first 12,000 records (~5mb), then stops.
I change the code to go through the next roughly 12,000 records (~5mb), and it stops again.
... and so on, until I get to the last record.
If I only go through any ~10,000 records, the program finishes normally. (has totals at the end)
Just a wild guess, but could this be related to the database access instead of the writing to file?
I remember from VB6 (a long time ago) that records got read into in blocks, and that you had to do .MoveLast .MoveFirst to get the total amount of records. Maybe there's still something like that, and that it goes wrong when it wants to read the next 'block'
Instead of writing to a file, try incrementing a variable once. If it goes above 12000 the problem is in the file writing, if it stops at 12000 I'm guessing database issue
You might be on to something there. (database issue)
I first changed the code to Append the records to the text file like this...
'Get Records from DBs, and do stuff to them...
fp = File.AppendText(FILENAME)
fp.WriteLine(BranchNo & AcctType & AcctNo & AcctTitle & totAddr & Phone & SSN & OwnType & SPACE(21) & ActTotals2 & SPACE(41))
fp.Close()
It still bombed out, around the same time the previous code did.
I tried your suggestion, and got the same results.
Any thoughts on where I should look next?
Thanks.
How are you getting your results? What database are you using?
I remember something vaguely from MySQL, there there was an option to determine the max size of the request sent back.
The maximum size of a BLOB or TEXT object is determined by its type, but the largest value you can actually transmit between the client and server is determined by the amount of available memory and the size of the communications buffers. You can change the message buffer size (max_allowed_packet), but you must do so on both the server and client ends. See section 7.5.2 Tuning Server Parameters.
http://www.mysql.com/doc/en/Server_parameters.html
max_allowed_packet current value: 1048576
I think this is the one. Don't know if your database has something similar, set to 5MB.
You will have to change the settings in web.config or machine.config file for the file size limit. 5 mb is the default limit set by web.config
Try adding the following lines in the <system.web
<httpRuntime executionTimeout="500" maxRequestLength="72000" /
72000 here refers to approximately 72 mb. change it to the limit you want to set and see if that works.
Access 2k is where all of the account info resides.
Basically, I open the table with the 50k records in a DS, and do a for...Next through the entire file. Do you think this can be part of the problem?
Here is the main routine... I didn't include all of the functions, they really just chain to other tables & grab phone#'s & such.
<%@. Import Namespace="System.IO" %>
<%@. Import Namespace="System.Data" %>
<%@. Import Namespace="System.Data.OLEDB" %>
<%@. Page Language="VB" Debug="true" %
<script language="VB" Debug="True" runat="server"
Public strConn As String = "PROVIDER=Microsoft.Jet.OLEDB.4.0;DATA SOURCE=" & server.mappath("db2.mdb") &";"
Dim totAddr, AddressNumber, strPers, strOrg, lblStatus, fName, ActTotals2 as String
Dim myNewRow, totalCount, CKTot, JODTot, IRATot, INDTot, COATot, CSATot, SAVTot As Single
Dim x, fLen as IntegerSub Page_Load(Src As Object, E As EventArgs)
Dim strSQL AS String = "Select * From Accounts2 Order BY AcctNbr"
Dim GetAcctsCommand As OleDbCommand = New OleDbCommand
Dim Connect as New OLEDBConnection(strConn)
GetAcctsCommand.Connection = Connect
Dim Adapter As New OleDBDataAdapter(strSQL, Connect)
Dim AccountsDS As DataSet = New DataSet
Dim RcdCount As Integer
Adapter.Fill(AccountsDS,"Accounts2")
RcdCount = AccountsDS.Tables("Accounts2").Rows.Count
Dim fp As StreamWriter
Dim AdrNbr, AcctNo, Phone, SSN, AcctType, BranchNo, OwnType, AcctTotal, AcctTitle, tmpField As String
Dim FILENAME as String = Server.MapPath("AccountFinal.txt")
Try
For x= 0 to RcdCount - 1
myNewRow = CType(AccountsDS.Tables("Accounts2").Rows(X).Item("BALAMT"), Single)
If myNewRow > 0 then
totalCount=totalcount + myNewRow
tmpField = AccountsDS.Tables("Accounts2").Rows(X).Item("BranchOrgNbr")
BranchNo = zFill(tmpField, 3)
tmpField = AccountsDS.Tables("Accounts2").Rows(X).Item("AcctNbr")
AcctNo = zFill(tmpField, 13)
AcctType = AccountsDS.Tables("Accounts2").Rows(X).Item("MJAcctTypCd") & SPACE(3-LEN(AccountsDS.Tables("Accounts2").Rows(X).Item("MJAcctTypCd")))
If AcctType = "CK " Then
AcctType = "40"
CKTot = CKTot + myNewRow
Else
AcctType = "10"
SAVTot = SAVTot + myNewRow
End If
OwnType = AccountsDS.Tables("Accounts2").Rows(X).Item("OwnCd") & SPACE(3-LEN(AccountsDS.Tables("Accounts2").Rows(X).Item("OwnCd")))
Select CASE OwnType
Case "JO ", "JA "
OwnType = "JOD"
JODTot = JODTot + myNewRow
Case "S "
If Left(AccountsDS.Tables("Accounts2").Rows(X).Item("CurrMiAcctTypCD"),1) = "R" Then
OwnType = "IRA"
IRATot = IRATot + myNewRow
Else
OwnType = "IND"
INDTot = INDTot + myNewRow
End If
Case "PUB"
OwnType = "COA"
COATot = COATot + myNewRow
End Select
If InStr(AcctTitle, "(Min") > 0 then
OwnType = "CSA"
CSATot = CSATot + myNewRow
End If
AcctTitle = GetTitle(AccountsDS.Tables("Accounts2").Rows(X).Item("AcctNbr"))
SSN = GetSSN(AccountsDS.Tables("Accounts2").Rows(X).Item("PERS_TAXID"), AccountsDS.Tables("Accounts2").Rows(X).Item("ORG_TAXID"))
Phone = GetPhone(AccountsDS.Tables("Accounts2").Rows(X).Item("TAXRPTFORPERSNBR"), AccountsDS.Tables("Accounts2").Rows(X).Item("TAXRPTFORORGNBR"))
If Phone = "" then Phone = Space(14)
AdrNbr = GetAddrNbr(AccountsDS.Tables("Accounts2").Rows(X).Item("TAXRPTFORPERSNBR"), AccountsDS.Tables("Accounts2").Rows(X).Item("TAXRPTFORORGNBR"))
totAddr = GetAddress(AdrNbr)
ActTotals2 = CType(Format(myNewRow,"0000000000.00"), String)
fp = File.AppendText(FILENAME)
fp.WriteLine(BranchNo & AcctType & AcctNo & AcctTitle & totAddr & Phone & SSN & OwnType & SPACE(21) & ActTotals2 & SPACE(41))
fp.Flush()
fp.Close()
lblStatus = "File Succesfully created!"
End If
Next
Catch err As Exception
lblStatus = "File Creation failed. Reason is as follows " & err.ToString()
Finally
End Try
End Sub
I tried all of the other suggestions, now I get this error:
File Creation failed. Reason is as follows System.IO.IOException: The process cannot access the file "c:\inetpub\wwwroot\AccountFinal.txt" because it is being used by another process. at System.IO.__Error.WinIOError(Int32 errorCode, String str) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync, String msgPath, Boolean bFromProxy) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize) at System.IO.StreamWriter.CreateFile(String path, Boolean append) at System.IO.StreamWriter..ctor(String path, Boolean append, Encoding encoding, Int32 bufferSize) at System.IO.StreamWriter..ctor(String path, Boolean append) at System.IO.File.AppendText(String path) at ASP.Accountlist6_aspx.Page_Load(Object Src, EventArgs E) in c:\inetpub\wwwroot\accountlist6.aspx:line 76 DONE!
going to try this next...
"You will have to change the settings in web.config or machine.config file for the file size limit. 5 mb is the default limit set by web.config
Try adding the following lines in the <system.web
<httpRuntime executionTimeout="500" maxRequestLength="72000" /
72000 here refers to approximately 72 mb. change it to the limit you want to set and see if that works. "
I'll let you know what happens.
Thanks.
BTW, if I make the following change, will I need to re-start the web server?
<httpRuntime executionTimeout="500" maxRequestLength="72000" /
Thanks.
That's just a web.config change. You don't even have to recompile for that to take effect, just save it to the server.
Ok, made the last change, now I get this dreaded message:
The page cannot be displayed...
But it looks like the ASPNET_WS.exe process is still going?
Any thoughts on how to keep the page from timing out?
Thanks.
It has to go in the system.web like so
<configuration>
<appSettings>
<add key="connString" value="sss" />
</appSettings>
<system.web>
<httpRuntime maxRequestLength="100000" />
<compilation defaultLanguage="vb" debug="true"></compilation>
spot18/[/url]